AI Crawler Tester
Enter a URL and test 19 crawlers, from GPTBot and ChatGPT-User to ClaudeBot, PerplexityBot and Googlebot. See what robots.txt says about each one, whether a live request as that crawler gets through or hits a firewall, and copy the robots.txt that fixes it.
- 19 AI and search crawlers
- robots.txt plus a live fetch
- Copy-ready fix
Score the whole page with the AI Search Visibility Checker, or build the file with the Robots.txt Generator.
Four kinds of crawler, four different decisions
Not every AI crawler does the same job, so blocking them all is rarely the right answer.
Search engines
Googlebot and Bingbot feed classic results and AI Overviews and Copilot answers. Block them and you disappear.
AI search
OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes AI answers cite from.
On-request fetchers
ChatGPT-User, Claude-User and Perplexity-User open a page when a person asks about it.
Training
GPTBot, ClaudeBot, CCBot and others collect data to train models. Allowing them is a separate choice from appearing in answers.
What each result means
A crawler can be stopped in two different places.
| Result | Meaning | Where to fix it |
|---|---|---|
| Can read the page | robots.txt allows it and the live request loaded the page. | Nothing to do. |
| Blocked by robots.txt | A Disallow rule applies to this crawler. | Edit robots.txt. A named group overrides the * group. |
| Blocked by firewall | robots.txt allows it, but the server or CDN refused the request. | Allow the user agent in your firewall, CDN or security plugin. |
| Gets much less text | The crawler receives a stripped or placeholder page. | Look for rules that serve bots a lighter page. |
How to test AI crawler access
Three steps from a URL to a fixed robots.txt.
Enter a page
Use a page you want AI tools to read. The tool reads its robots.txt and tests every crawler against that exact path.
Read the table
Each crawler shows the robots.txt rule that applies, the live response, and one verdict. Open a row for the reason.
Copy the fix
Pick a policy, copy the generated groups into robots.txt, deploy, and test again.
How far the live test can be trusted
The live fetch is useful, and it has limits.
A simulated request
The tool sends each crawler's published user agent from Cloudflare's network. Real crawlers come from verified IP ranges, so a firewall may treat them differently.
Token-only names
Google-Extended and Applebot-Extended are robots.txt control tokens with no crawler of their own, so only robots.txt can be tested for them.
robots.txt is a request
Well-behaved crawlers follow it. Some, like Bytespider, are known to ignore it, so a firewall rule is the only real block.
Group rules
A crawler uses its own group if one names it, and the * group otherwise. Adding a named group is the safest way to change one crawler.
Content signals
Some hosts add a Content-Signal line to robots.txt. The tool reads it and shows what you have said about search, AI input and AI training.
Test after changes
CDNs cache robots.txt. After editing it, wait for the cache to clear and test again.
What an AI crawler tester shows and why access comes first
AI products reach your site through crawlers, each with its own name and purpose. Some collect content that may be used to train models. Some build a search index. Some fetch a page only when a person asks about it. Whether any of them can read your pages depends on two layers: the rules in robots.txt, and whatever your server, CDN or security plugin does with the request itself.
An AI crawler tester checks both. It reads robots.txt, applies the rules for each named crawler to the page you enter, then requests the page with that crawler's published user agent to see whether it loads. The result separates a block you wrote on purpose from one you did not know you had.
The names matter because the effects differ. OpenAI's crawler documentation says blocking OAI-SearchBot keeps a site out of ChatGPT search answers, while blocking GPTBot signals that content should not be used for training. Anthropic and Perplexity document separate bots for similar jobs. Allowing or blocking all of them together is rarely the right call.
When to test AI crawler access
Test whenever you wonder whether AI tools can read a page, or after any change to your bot rules.
You are never mentioned in AI answers
Before looking at content, rule out the simple cause. If search crawlers are blocked by robots.txt or a firewall, nothing else you do will help.
After enabling bot protection
Cloudflare-style bot settings, hosting firewalls and WordPress security plugins can refuse unfamiliar user agents. The live test shows whether robots.txt says yes while the server says no.
Deciding a training policy
Pick between allowing training crawlers or only search and user-triggered ones. The tester shows the current position and generates the groups for the policy you choose.
After a launch or migration
Check a few key pages on the new site. A blanket staging rule or new CDN setting can block every AI crawler without anyone noticing.
Reviewing a client's or competitor's stance
See which AI crawlers a site welcomes or blocks. Some sites block training crawlers but allow search ones, and the table shows it per crawler.
As part of an AI readiness audit
Use it first, then move on to content structure, structured data and trust signals. Crawl access is the base the rest depends on.
AI crawler mistakes and how to fix them
The usual problems come from lumping different bots together or blocking at the wrong layer.
Treating all AI bots as one thing
What goes wrong: Blocking every AI user agent to stop training also blocks the bots that build search indexes and fetch pages for users, so you can disappear from AI answers.
The fix: Decide per purpose. Allow search and user-fetch bots if you want to be cited, and set training crawlers such as GPTBot and ClaudeBot separately.
Blocking OAI-SearchBot to opt out of training
What goes wrong: OAI-SearchBot surfaces sites in ChatGPT search. GPTBot is the one OpenAI documents for training. Mixing them up removes you from the wrong place.
The fix: Use GPTBot to opt out of training and leave OAI-SearchBot allowed. OpenAI says the settings are independent.
Fixing robots.txt when the firewall is the cause
What goes wrong: robots.txt allows a crawler, yet the page still fails to load for it. The block is in a CDN rule, security plugin or hosting firewall.
The fix: Allow the user agent in the firewall layer. Real crawlers come from published IP ranges, so check your logs and each company's IP list rather than the user agent alone.
Assuming robots.txt binds every fetch
What goes wrong: OpenAI says robots.txt rules may not apply to ChatGPT-User, which acts on a user's request, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason.
The fix: Do not rely on robots.txt to control user-initiated fetches. If you need a hard block, use your server or firewall.
Blocking Google-Extended to leave AI Overviews
What goes wrong: Google-Extended is a control token for Gemini training and grounding. It does not affect inclusion in Google Search.
The fix: To change what shows in Search, use Googlebot rules and snippet controls such as nosnippet or max-snippet. Use Google-Extended only for the Gemini choice.
Expecting immediate effect
What goes wrong: Crawlers cache robots.txt, and so does your CDN. Perplexity and OpenAI both say changes can take around a day to apply.
The fix: Make the change, wait for caches to clear, then test again. Do not judge a new policy minutes after deploying it.
AI crawler terms explained
- Training crawler
- A bot that collects content that may be used to train AI models, such as
GPTBot,ClaudeBotandCCBot. Blocking it does not stop search or on-request fetching. - Search crawler
- A bot that builds the index an AI search feature answers from, such as
OAI-SearchBot,Claude-SearchBotandPerplexityBot. Blocking it can keep you out of those answers. - User-triggered fetcher
- A bot that opens a page because a person asked, such as
ChatGPT-UserorPerplexity-User. Both companies say robots.txt may not apply to these. - Google-Extended
- A robots.txt token that controls whether Google may use your content for Gemini training and some grounding. It has no separate user agent and does not affect Search.
- CCBot
- The crawler of Common Crawl, a non-profit that publishes an open archive of web data. Many people use that archive for research and AI training.
- User agent string
- The name a client sends with each request. A firewall may use it to allow or refuse a bot, and it can be faked, so IP checks are more reliable.
- Published IP ranges
- Lists of addresses that companies such as OpenAI, Anthropic and Perplexity publish for their bots, so site owners can verify genuine traffic.
- Firewall block
- A refusal by a server, CDN or security plugin before the page is served, even though robots.txt allows the crawler.
Related tools, guides and services
Crawl access is the first layer of AI readiness. These cover the rest.
Robots.txt Generator
Create a file with the AI crawler groups you want, then test it before publishing.
ToolRobots.txt Tester
Test any URL against Googlebot, Bingbot and AI crawlers and see the deciding rule.
ToolAI Search Visibility Checker
Score the page's content, structured data and trust signals once crawlers can reach it.
ToolHTTP Header Viewer
Inspect the response headers your server sends to see where a block might come from.
GuideFuture of SEO in Nepal
Covers crawler controls, Google-Extended and AI search in a practical plan.
ServiceAI SEO service
Get AI search readiness reviewed and improved for your brand.
Built by
Madhav Kushwaha
Whether AI tools can quote a business starts with a plain question: can their crawlers read the page? Plenty of sites block them by accident, through a broad robots.txt rule or a firewall setting nobody remembers. I built this tester to show, crawler by crawler, where the block is and to give the exact fix.
AI Crawler Tester: Frequently Asked Questions
More free SEO tools
Crawl control is one part of AI readiness. These tools cover the rest.