Robots.txt Tester
Fetch any website's robots.txt or paste your own, then test whether Googlebot, Bingbot and AI crawlers can fetch a URL. See the exact rule that allows or blocks it, and catch syntax mistakes before they cost you traffic.
- Follows Google's matching rules
- Compares 20 search, AI and social crawlers
- Pasted text never leaves your browser
Need to write the file first? Use the Robots.txt Generator.
Four rules decide whether a crawler can fetch a URL
Robots.txt looks simple, but the order of lines doesn't matter and the most specific rule wins. This is what the tester does with your file.
1. Pick the group
A crawler follows the group that names it. If there is none, it follows the User-agent: * group. Groups naming the same crawler are merged.
2. Longest match wins
Every Allow and Disallow rule that matches the URL competes. The rule with the longest path is the most specific, so it wins.
3. Allow wins a tie
If an Allow and a Disallow match with the same length, the less restrictive rule applies, so the URL stays crawlable.
4. Wildcards and $
* matches any run of characters and a trailing $ means the URL must end there. Matching is case-sensitive and includes the query string.
What Google does with each type of robots.txt response
The HTTP status of /robots.txt matters as much as its content. The tester reads the status when it fetches a live file and tells you which case applies.
| Response | What Google does | What to do |
|---|---|---|
| 200 OK | Reads the file and follows its rules. Content past 500 KiB is ignored. | Keep the file small and valid. |
| 301 or 302 redirect | Follows up to 5 redirects, then treats the file as missing. | Serve robots.txt directly, from the exact host that is being crawled. |
| 404 or other 4xx | Treats the file as missing and crawls everything. | Fine if you want no restrictions. Add a file if you do. |
| 429 or 5xx | Assumes the whole site is off limits and pauses crawling. After 30 days of errors it uses the last good copy or crawls freely. | Fix server errors on /robots.txt quickly: they can stop all crawling. |
| 200 with an HTML page | Tries to read the HTML as robots.txt and finds nothing useful. | Return a real plain-text file instead of a soft 404 page. |
Robots.txt problems this tester catches
These are the issues I find most often when auditing real sites.
Disallow: / left from staging
A site-wide block copied to production removes every page from crawling. The tester flags it and shows which URLs it blocks.
Blocking CSS and JavaScript
Google needs your CSS and JS to render pages and judge mobile-friendliness. Rules like Disallow: /wp-includes/ are flagged.
Full URLs instead of paths
Disallow: https://example.com/private/ never matches anything. Rules must be paths that start with /.
Rules before User-agent
An Allow or Disallow that comes before any User-agent line applies to nobody and is silently ignored.
Noindex in robots.txt
Google stopped supporting Noindex: in robots.txt in 2019. Use a robots meta tag or an X-Robots-Tag header instead.
Typos and wrong directives
Misspelled directives, missing colons and unsupported lines such as Crawl-delay (ignored by Google) are listed with the line number.
What robots.txt, and any tester, can't do
Robots.txt controls crawling. It does not control everything people expect it to.
It isn't security
Anyone can read robots.txt, and bad crawlers ignore it. Never use it to hide private data.
It doesn't remove pages
A blocked URL can still appear in results, without a description, if other sites link to it. Use noindex or remove it in Search Console.
AI crawlers choose to obey
The big AI companies say they follow robots.txt, but it is a request, not enforcement.
Google caches the file
Google may use a cached copy for up to 24 hours, so a change isn't instant.
What a robots.txt tester does and how crawlers read your rules
A robots.txt tester answers one question: given this file, can this crawler fetch this URL? It picks the group that applies to the crawler, compares every Allow and Disallow rule against the URL's path, and shows which rule decided. That last part is what makes a tester useful. Reading a file by eye is unreliable, because line order does not matter and the longest matching rule wins.
Google describes the logic in its robots.txt specification: paths are case-sensitive, * and $ are supported wildcards, the most specific rule applies, and the less restrictive rule wins when two conflict. The wider standard is RFC 9309.
The file's HTTP response matters as much as its text. A missing file, a redirect and a server error each lead to different crawler behaviour. So this tester reads the status of the live file as well as its rules, and it can also take pasted text if you want to try a change before it goes live.
When to test a robots.txt file
Test whenever a rule could have changed, and before any change goes live.
A page is reported as blocked
Search Console says a URL is blocked by robots.txt. Paste the URL, choose Googlebot and see the exact line responsible, then decide whether the rule or the page should change.
Before you deploy a change
Paste the new version of the file and test the URLs that must stay crawlable, such as top products and key articles. Fix any surprise before the file replaces the old one.
Right after a launch or migration
Fetch the live file and confirm no staging Disallow: / came across. Then test a handful of important URLs for Googlebot and Bingbot.
Checking AI crawler access
Run the same URL as several crawlers to see which AI user agents your file allows. A named group for one crawler overrides the * group for that crawler.
Auditing a client or competitor
Fetch any public site's file to see what it blocks, which sitemaps it lists and whether it carries unsupported lines such as Crawl-delay or Noindex.
Investigating sudden crawl drops
If crawling stops or slows, check the file's HTTP status. Repeated server errors on /robots.txt can pause crawling of the whole site.
Mistakes people make when testing robots.txt
A tester is only as good as the question you put to it. These are the usual ways to get a misleading answer.
Testing only Googlebot
What goes wrong: A file can allow Googlebot and still block another crawler through a named group. Bing, AI crawlers and social preview bots may all get different results.
The fix: Test each crawler that matters to you. Remember that a crawler follows only the most specific group that names it, so a named group does not inherit rules from the * group.
Assuming line order decides
What goes wrong: People expect the first matching rule to win. In Google's matching the order of lines does not matter. The longest matching path wins.
The fix: Read results by specificity. To open one file inside a blocked folder, add a longer Allow such as Allow: /private/press-kit/ under Disallow: /private/.
Forgetting case and query strings
What goes wrong: Paths are case-sensitive, and the query string is part of the URL being matched. A rule for /Shop/ does not cover /shop/, and a rule can miss or catch URLs because of a parameter.
The fix: Test the exact URL as it appears in your sitemap or logs, with its capitalisation and parameters.
Ignoring the HTTP status of robots.txt
What goes wrong: A 404 means no rules, so everything may be crawled. A 5xx stops Google from crawling for a while. A soft 404 returns an HTML page with a 200 status, which contains no usable rules.
The fix: Make sure /robots.txt returns 200 with plain text, or a clean 404 if you want no rules. Fix server errors on it quickly.
Trusting a pasted copy over the live file
What goes wrong: The text in your editor may differ from what the server returns, for example after a CDN, plugin or host rewrites it. Google also caches the file, so a change is not seen at once.
The fix: After deploying, fetch the live file and test again. Expect Google to take up to about a day to refresh its cached copy, then confirm in Search Console.
Reading 'allowed' as 'indexed'
What goes wrong: A URL that is allowed can still be left out of the index. A URL that is blocked can still be indexed without a description if others link to it.
The fix: Use the tester for crawl access only. For indexing questions, check for noindex, canonicals and the Page indexing report in Search Console.
Robots.txt testing terms explained
- Matching group
- The group of rules a crawler obeys. It is the group whose
User-agentline names the crawler most specifically, or the*group if none does. - Longest match
- Among all rules that match a URL, the one with the longest path is the most specific and applies. Google uses this instead of reading the file top to bottom.
- Least restrictive rule
- When an
Allowand aDisallowmatch with equal specificity, Google applies the one that permits crawling, so the URL stays crawlable. - Product token
- The short name a crawler uses in robots.txt, such as
GooglebotorGPTBot. Matching of this token is case-insensitive, while URL paths are case-sensitive. - Soft 404
- A page that says it is missing but returns a 200 status. If this happens at
/robots.txt, crawlers receive an HTML page and find no usable rules. - Cached robots.txt
- A copy a crawler keeps instead of fetching the file on every request. Google generally caches it for up to 24 hours, so edits are not seen immediately.
- Crawl versus index
- Crawling is fetching a URL. Indexing is storing and serving it in results. Robots.txt controls the first only, which is why a blocked URL can still appear.
- Unsupported directive
- A line Google does not act on, such as
Crawl-delayorNoindex. The line is harmless to include but has no effect on Google.
Related tools, guides and services
Once you know what the file does, use these to fix it, extend it or see how it fits the rest of your indexing.
Robots.txt Generator
Draft a clean file from a template, with AI crawler groups and a sitemap line.
ToolAI Crawler Tester
Go beyond the rules and test live requests as each AI crawler to catch firewall blocks.
ToolHTTP Header Viewer
Inspect the status code, content type and cache headers your server sends for robots.txt.
GuidePages not indexed? Causes and fixes
Separates a robots.txt block from noindex, canonical and quality problems.
GuideSEO audit errors that matter
See where robots.txt blocks rank among the issues that affect crawling and indexing.
ServiceTechnical SEO service
Have crawl rules, indexing and site architecture reviewed by a specialist.
Built by
Madhav Kushwaha
SEO, AEO, GEO & Digital Marketing Consultant in Kathmandu, Nepal, and trainer at ODcrU Academy and TechAxis Institute. I built this tester because a single wrong line in robots.txt can hide a whole section of a site from Google, and most testers don't show which rule made the decision. The matching here follows Google's published specification, and the logic is exactly what the code does.
Robots.txt Tester: Frequently Asked Questions
More free SEO tools
Crawling and indexing are connected: use these tools together to keep search engines pointed at the right pages.