AI crawler checker
Enter a page and see which of 14 AI crawlers your robots.txt lets in, the exact rule that decides it, and the lines to paste if one you want is blocked.
What this checker does
It fetches your /robots.txt and the page you entered, then works out, for each AI crawler, which robots.txt group applies and which rule wins, the same way the crawlers do under the robots.txt standard (RFC 9309). It also reads the page's meta robots tags and X-Robots-Tag header, which can keep a page out of search indexes even when robots.txt allows the crawl.
- A crawler obeys the group that names it, and only that group. If GPTBot has its own group, the rules under User-agent: * no longer apply to it.
- If no group names it, it falls back to the * group. If there is none, everything is allowed.
- Within a group the longest matching rule wins, and when an Allow and a Disallow are the same length, Allow wins.
- Patterns can use * (any characters) and $ (end of the URL), so Disallow: /*.pdf$ blocks PDFs only.
- A robots.txt that returns a server error is read as "crawl nothing" until it recovers. A missing one (404) means everything is allowed.
Why AI crawlers matter for AI search
AI assistants can only cite pages they are allowed to read. The vendors split their crawlers by job. OpenAI says OAI-SearchBot surfaces websites in ChatGPT search while GPTBot collects training data, and that the two are independent settings. Anthropic documents ClaudeBot (training), Claude-SearchBot (search) and Claude-User (fetches for a user) separately, and Perplexity says PerplexityBot is for search results, not model training.
| Kind | Crawlers | Block it and… |
|---|---|---|
| Search | OAI-SearchBot, Claude-SearchBot, PerplexityBot, bingbot | AI search answers can't list or cite your pages |
| Fetch for a user | ChatGPT-User, Claude-User, Perplexity-User | The assistant can't open your page when someone asks about it |
| Training | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent | Future models learn less about you; current search answers are not affected |
Google-Extended and Applebot-Extended are tokens, not separate crawlers. Google says Google-Extended doesn't affect inclusion in Google Search; it controls Gemini training and grounding. AI Overviews use regular Googlebot.
How to fix a blocked crawler
- Decide which crawlers you want. For AI visibility, allow at least the search and user-fetch crawlers. Training crawlers are a business choice.
- Edit /robots.txt: delete the group that blocks the crawler, or paste the fix this tool gives you. Remember a crawler only reads its own group, so copy any Disallow lines for private paths into it.
- Check your CDN or firewall. Bot protection (for example an "AI bots" switch) can block crawlers even when robots.txt allows them. If this tool got a 403, that is a likely sign.
- Remove noindex from pages you want cited, then re-run this check. OpenAI says robots.txt changes can take about 24 hours to take effect.
Questions about the ai crawler checker
If I block GPTBot, will ChatGPT stop recommending me?
Not in search answers. OpenAI documents GPTBot (training) and OAI-SearchBot (search) as independent settings. Blocking GPTBot keeps your content out of future training; blocking OAI-SearchBot keeps you out of ChatGPT search results.
Does ChatGPT-User obey robots.txt?
Not necessarily. OpenAI says that because ChatGPT-User acts on a user's request, robots.txt rules may not apply. Perplexity says the same of Perplexity-User. The checker still shows what your file says for them.
What does "partly blocked" mean?
The page you entered and the homepage get different answers: either this page is blocked while the site is open, or the site is blocked and only some paths are allowed. Allowed bots with a few blocked private paths, such as /admin, show as allowed with those paths listed.
Why does the checker say the site returned 403?
Your server or CDN refused our scanner, which usually means bot protection is on. It may block AI crawlers the same way, regardless of robots.txt. Check your firewall's bot settings.
Is this tool free? Does it use AI?
Yes, free, with no signup. It only fetches and parses your robots.txt and page; no AI model is involved. Results are cached for an hour.