Robots.txt Generator
Build a clean robots.txt in minutes. Start from a template for your platform, add or remove rules, block AI crawlers with a click, list your sitemaps and test the result before you upload it.
- Templates for WordPress, Shopify and more
- Block AI crawlers in one click
- Test every rule before you publish
Already have a file? Check it with the Robots.txt Tester.
How to create and publish a robots.txt file
Robots.txt is a plain text file that sits at the root of your site. These four steps get a correct one live.
Pick a starting point
Choose the template closest to your site, or start from scratch. You can change everything afterwards.
Adjust the rules
Add the folders and URL patterns you don't want crawled, and list your sitemap. Use the Test tab to check real URLs.
Upload it to the root
Save it as robots.txt and put it at https://yoursite.com/robots.txt. It must be at the root of each host.
Re-test the live file
Open the Robots.txt Tester on your live site to confirm the right URLs are crawlable, then watch Search Console.
AI crawlers: what each one does and what blocking it means
Not all AI user agents do the same job. Blocking the wrong one can remove you from AI search results while doing nothing about training.
| Crawler | Owner | Purpose | Blocking it |
|---|---|---|---|
| GPTBot | OpenAI | Collects content for training AI models | Keeps your content out of future OpenAI training data |
| OAI-SearchBot | OpenAI | Builds the index for ChatGPT search | Can stop your pages appearing in ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches a page when a user asks ChatGPT to | Stops ChatGPT opening your pages on request |
| ClaudeBot | Anthropic | Collects content for training AI models | Keeps your content out of future Anthropic training data |
| Google-Extended | Control token for Gemini and Vertex AI use of your content | Opts out of Gemini training and grounding. Doesn't affect Google Search ranking | |
| PerplexityBot | Perplexity | Builds the Perplexity search index | Can stop your pages appearing in Perplexity answers |
| CCBot | Common Crawl | Builds an open dataset used to train many AI models | Keeps you out of Common Crawl snapshots |
Crawler names change over time. Check each company's documentation before relying on this list, and remember that obeying robots.txt is voluntary.
Robots.txt rules that help, and rules that hurt
A good robots.txt is short. Block what wastes crawl budget, and leave everything Google needs.
Block
Admin and login areas, cart and checkout pages, internal search results, endless filter and sort parameters, and staging sites.
Never block
CSS, JavaScript and images that your pages need to render, and any page you want in search results.
Don't use it to hide pages
Blocked pages can still be indexed. Use a noindex tag to keep a page out of results, and a login to keep it private.
Always list your sitemap
A Sitemap line is the easiest way to tell every crawler where your sitemap is, without a separate submission.
One file per host
Each subdomain and protocol needs its own robots.txt. The file on www doesn't cover blog.example.com.
Test after every change
One typo can block a whole section. Test the URLs that matter before and after you publish.
What a robots.txt generator does and why the file matters
A robots.txt file is a plain text file at the root of a website that tells crawlers which paths they may fetch. It is made of groups. Each group starts with a User-agent line that names a crawler (or * for all of them), followed by Allow and Disallow rules. You can also add Sitemap lines. The format is standardised in RFC 9309, and Google explains the parts it supports in its guide to creating a robots.txt file.
A generator helps because the syntax is simple to write and easy to get slightly wrong. One misplaced slash can block a whole section, and a file copied from another site may block folders your own site needs. This tool starts from a template for your platform, lets you add or remove rules, includes a switch for AI crawlers, and tests the result against real URLs before you upload anything.
Keep the limits in mind. A robots.txt file controls crawling, not indexing, and it is not security: the RFC says these rules are not a form of access authorization. Most sites need only a handful of lines, so a short and tested file is usually better than a long one.
When to build a new robots.txt file
Write a fresh file at the moments when crawl rules are most likely to be missing or wrong.
Launching a new site
Draft a simple first file that allows crawling, lists your sitemap and blocks only admin or utility areas. Then check that no staging rule survived the move to production.
Replacing the WordPress default
WordPress builds a basic robots.txt on the fly. If you want to add a sitemap line, AI crawler rules or extra blocks, draft the full file here and publish it by the route your host supports.
Customising a Shopify store
Shopify generates a default file and lets you change it through a robots.txt.liquid template. Use the generator to work out which extra rules you want before you edit the template.
Choosing a position on AI crawlers
Decide whether to allow training crawlers, search crawlers, or both, and generate the right groups for each one instead of writing the names from memory.
Keeping crawlers out of low-value URLs
Block internal search results, cart and checkout pages, or endless filter and sort parameters so crawlers spend their requests on pages you want found.
Setting up a new subdomain or host
A robots.txt file only applies to the host, protocol and port it is served from. A new blog, shop or app subdomain needs its own file, not a copy of the main one.
Robots.txt mistakes and how to avoid them
Most damaging robots.txt problems come from a rule that was right for one purpose and wrong for another.
Using it to hide pages from search
What goes wrong: A Disallow rule stops crawling, but a blocked URL can still be indexed if other pages link to it. And a blocked page cannot show Google a noindex tag, because Google never reads the page.
The fix: To keep a page out of results, leave it crawlable and add a noindex robots meta tag or X-Robots-Tag header. For private content, use a login.
Blocking CSS, JavaScript or images
What goes wrong: Rules such as Disallow: /wp-includes/ or a block on a theme or assets folder can stop Google from rendering your pages the way visitors see them.
The fix: Allow the files a page needs to render. Block only areas that are of no use to a searcher, such as admin screens, carts and internal search.
Carrying a staging block into production
What goes wrong: A Disallow: / copied from a staging server tells every crawler to stay away from the whole live site.
The fix: Read the file you are about to publish line by line, test your key pages against it, and check the live file again right after launch.
Putting the file in the wrong place
What goes wrong: A file in a folder, or only on www while the site also answers on another host, is ignored for the hosts it does not cover.
The fix: Serve it at the root of every host, for example https://example.com/robots.txt and https://blog.example.com/robots.txt, as plain UTF-8 text.
Blocking the wrong AI crawler
What goes wrong: Blocking OAI-SearchBot removes your site from ChatGPT search answers, even if your aim was only to opt out of AI training. Training and search use different crawlers, so one rule does not cover both goals.
The fix: Read each company's crawler documentation. OpenAI, for example, lists GPTBot for training and OAI-SearchBot for search, and says the two settings are independent.
Expecting Crawl-delay or Noindex to work in Google
What goes wrong: Google does not support the Crawl-delay directive, and it does not support noindex inside robots.txt. Lines like these look like controls but change nothing for Google.
The fix: Use a robots meta tag or header for noindex. Add Crawl-delay only for a specific crawler that documents support for it, for example Bing or Anthropic's crawlers, and check that crawler's documentation first.
Robots.txt terms explained
- User-agent
- The line that starts a group and names the crawler the rules apply to.
*means every crawler that has no group of its own. - Disallow
- A rule that asks a crawler not to fetch URLs whose path starts with the given value. An empty
Disallowblocks nothing. - Allow
- A rule that permits a path, usually to make an exception inside a blocked folder. If rules conflict, Google applies the most specific one, and the less restrictive one on a tie.
- Group
- A
User-agentline plus the rules under it. A crawler follows only the group that matches it most specifically, not every group in the file. - Wildcard (
*) and$ *matches any run of characters and$marks the end of the URL. For example,Disallow: /*.pdf$matches URLs that end in.pdf. Paths are case-sensitive.- Sitemap directive
- A
Sitemap:line with a full URL that tells crawlers where your sitemap is. It sits outside the groups and applies to every crawler. - Crawl-delay
- A non-standard directive that asks a crawler to wait between requests. Google ignores it. Some other crawlers honour it.
- X-Robots-Tag
- An HTTP response header that carries indexing rules such as
noindex. It works for files like PDFs, where a meta tag is not possible, and the page must be crawlable for it to be read.
Related tools, guides and services
Writing the file is the first half of the job. These help you test it and connect it to the rest of your crawl setup.
Robots.txt Tester
Test any live or pasted file against Googlebot, Bingbot and AI crawlers, and see the exact rule that decides.
ToolAI Crawler Tester
See whether GPTBot, ClaudeBot, PerplexityBot and others can reach a page, including firewall blocks.
ToolXML Sitemap Generator
Build the sitemap that your robots.txt Sitemap line should point to.
GuidePages not indexed? Causes and fixes
Learn how a robots.txt block differs from noindex and other indexing problems.
GuideRedesign SEO migration checklist
Includes removing staging blocks before launch, the costliest robots.txt mistake.
ServiceTechnical SEO service
Get crawling, indexing and robots.txt rules reviewed and fixed for your site.
Built by
Madhav Kushwaha
SEO, AEO, GEO & Digital Marketing Consultant in Kathmandu, Nepal, and trainer at ODcrU Academy and TechAxis Institute. I built this generator because most robots.txt files on the web are copied from somewhere else and never tested. Every template here is a starting point I would be comfortable handing to a client, and the built-in tester uses Google's matching rules.
Robots.txt Generator: Frequently Asked Questions
More free SEO tools
Crawling and indexing are connected: use these tools together to keep search engines pointed at the right pages.