Seolytics
Free SEO tools · no sign-up

Robots.txt checker for Google and AI crawlers

Enter a site and a page path to see whether Googlebot and the crawlers of ChatGPT, Perplexity and Claude are allowed to read it — and which rule decides.

What it shows

  • Google SearchGooglebot also feeds AI Overviews and AI Mode: blocking it removes the page from both.
  • AI search crawlersOAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Claude-User fetch pages for AI answers. Blocked pages can't be cited.
  • AI training crawlersGPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot collect training data. Blocking them does not affect AI search.

Questions

How are the rules read?

Like Google does: the most specific user-agent group wins, then the longest matching Allow or Disallow rule; on a tie, Allow wins.

Should I block AI crawlers?

Block training crawlers if you don't want your content used to train models. Keep AI search crawlers allowed if you want to be cited in ChatGPT, Perplexity and Claude answers.

Is robots.txt enough to hide a page?

No. robots.txt stops crawling, not indexing. Use noindex to keep a page out of search.

More free tools

Unlimitedllms.txt Validatorllms.txt is a proposed file that gives AI tools a short map of a website. Check whether a site has one and whether it follows the format. Free lookupsWebsite Traffic CheckerSee how many visits a site gets from Google search, which keywords bring them and which pages earn the most. 2 sites a dayWebsite auditCheck up to 25 pages of a site for SEO and AI-search problems, with a fix for each issue.