Robots.txt Doesn't Remove Pages From Google (Here's What Does)

The short answer

I see this mistake often enough that I check for it early in any indexing review. Someone wants a page gone from Google, so they add Disallow: /that-page/ to robots.txt. A week later the page is still in results, and now it looks worse, with no snippet. The file did exactly what it is designed to do. It just was never designed to do this.

Google's own documentation is direct about it: robots.txt is used mainly to manage crawler traffic, and it is not a mechanism for keeping a web page out of Google. To keep a page out of search results, Google lists three options: block indexing with noindex, password-protect the page, or remove the page.

Crawling and indexing are two different decisions

Crawling is Google fetching a URL. Indexing is Google analysing what it fetched and deciding whether to store it and show it. Robots.txt only speaks to the first. When you disallow a URL you are telling compliant crawlers not to request it. You are saying nothing about whether the URL should appear in search.

That has a consequence people rarely expect. Because Google cannot fetch a blocked page, it cannot read anything on it: not the title, not the content, and not a noindex tag or a canonical tag. If other pages link to the URL, Google may still index the bare URL, using the link text and other external signals, but with no description. In the Page indexing report this appears as "Indexed, though blocked by robots.txt".

How blocked pages still end up in results

• The page was already indexed. Blocking it afterwards stops fresh crawls, so Google keeps the version it already stored and cannot learn that anything changed.
• The page is linked from elsewhere. Internal links, other websites and sitemaps can all point at it. Google may index the URL without ever fetching it.
• The block hides the real removal signal. If you have already added noindex, or the page now returns a 404, the Disallow rule stops Google seeing it. Google's crawl budget guidance notes that a 404 is a strong signal not to crawl a URL again, while blocked URLs stay in the crawl queue much longer.

So blocking can leave a removed page in search for longer than if you had done nothing. The next sections cover what to use instead.

What actually removes a page from Google

The right method depends on whether the page should still exist and who should be able to see it.

noindex

This is the method I reach for most when a page should stay available but out of search, such as an internal search results page, a thank-you page or a thin filtered view. Google supports it as a meta tag in the page or as an HTTP response header. The header is the way to handle files that cannot hold a tag, such as PDFs.

<meta name="robots" content="noindex">

X-Robots-Tag: noindex

Two practical points. The rule only takes effect when Google next crawls the page, so it is not instant. And it only affects Google's results (and other engines that honour it). The page is still reachable by anyone with the link. You can check what a page actually serves with the Meta Tag Checker for the tag and the HTTP Header Viewer for the header.

404 or 410

If a page is gone for good and nothing replaces it, say so with the status code. Google's crawl guidance says to return a 404 or 410 for permanently removed pages, and its crawler documentation says it treats all 4xx errors except 429 the same way: the content is treated as nonexistent, previously indexed URLs are removed from the index, and crawling of that URL gradually slows. So I do not agonise over 404 versus 410. What matters is that the response is a real error status and not a 200 page that says "not found", which Google may report as a soft 404. A server error (5xx) is not a removal instruction either. Google keeps already-indexed URLs for a while and drops them only if the errors persist, which is a symptom of a broken site and not a method. You can test a list of URLs in bulk with the HTTP Status Code Checker.

Authentication

For private material, put it behind a login. Google lists password protection as a way to keep a page out of search and, unlike a robots rule, it also keeps the content from anyone who is not meant to see it. Never list sensitive paths in robots.txt to "protect" them. The file is public, so it works as a map of exactly what you wanted to hide.

The Removals tool

Search Console's Removals tool can hide a URL from results quickly, but Google describes the effect as temporary: a successful request lasts about six months, and the page can come back unless you also make a permanent change. It blocks the URL from showing in results. It does not stop Google crawling it. I treat it as emergency cover while the permanent fix, noindex, a 404 or 410, or a login, takes effect. If you are not the site owner and want outdated content updated, Google has a separate route for that.

If you have already blocked a page you wanted removed

This is the order I follow, because the fix has to let Google see the new signal.

1. Decide what the page should be. Live but hidden from search (noindex), gone (404 or 410), moved (301) or private (login).
2. Remove the Disallow rule for that URL so Google is allowed to fetch it. Confirm with the Robots.txt Tester and the robots.txt report in Search Console.
3. Apply the chosen signal. Add the noindex tag or header, or return the correct status code, or set up the redirect or login.
4. Verify it on the live URL. Use the URL Inspection tool's live test to see what Google receives, including whether indexing is allowed.
5. Use the Removals tool only if the page must disappear quickly, knowing it is temporary.
6. Wait for recrawling. Google has to fetch the page again to process the change, which takes anywhere from days to months depending on the page. Watch the Page indexing report rather than checking by hand.
7. Re-add a Disallow only if you also want to stop crawling, and only after the page has dropped out. For a 404 or 410 you usually do not need to.

When robots.txt is the right tool

None of this makes robots.txt useless. It is the correct tool for managing crawler traffic. Typical good uses:

• Keeping crawlers away from endless URL combinations created by filters, sorting and internal search, where you do not need those URLs crawled at all.
• Blocking crawling of areas that add server load without adding search value, such as cart or account paths.
• Pointing crawlers to your sitemap.
• Setting different rules for specific crawlers, including AI crawlers. Compliance is voluntary and not every crawler follows the file, as Google itself notes, so it is a request, not enforcement. The AI Crawler Tester shows which crawlers your file allows.

Mistakes I check for

• Blocking pages that are already indexed and expecting them to vanish.
• Blocking scripts or styles a page needs in order to render. Google may then see a different page from your visitors. The URL Inspection tool lists page resources it could not load, and the JavaScript SEO guide shows how I check the rendered result.
• A blanket Disallow: / carried over from staging. This is one of the ways a template change hurts a live site, covered in the deploy guide.
• Different rules for mobile and desktop versions of a site. Google generally wants the same robots rules on both.
• Treating robots.txt as security. It is public and advisory.

What I did on my own site

In October 2026 I cleaned up URLs reported as not found in Search Console after I removed several old sections from this website. Where an equivalent page existed, I pointed the old URL to it with a single 301. Where nothing equivalent existed, I returned 410 Gone. I deliberately did not add Disallow lines for any of them: a blocked URL stays in the crawl queue longer, and Google would never have seen the status code that tells it the page is finished. I also did not send the removed sections to the homepage, because redirecting unrelated pages to one irrelevant page can be treated as a soft 404.

That is the same logic I apply on client sites during migrations: say clearly what happened to each URL, in the signal Google actually reads. More on how I handle the diagnosis is in the indexing guide, and the four-stage view that puts this in context is in technical SEO in four questions. If you need hands-on help, see my technical SEO service.

Frequently asked questions

Does blocking a page in robots.txt remove it from Google?

No. It stops compliant crawlers fetching the page. Google says it might still index a disallowed URL if it is linked from other places, without a description. To remove a page, use noindex on a crawlable page, a 404 or 410, or authentication.

Can I use noindex and robots.txt together?

Not on the same URL if you want the noindex to work. Google must be able to crawl the page to see the tag. If robots.txt blocks it, the rule is never read. Google also does not support a noindex directive inside robots.txt.

How long does it take for a page to disappear after I add noindex?

Until Google recrawls the page, which depends on how often it visits that URL. It can be days or much longer. The Removals tool can hide a URL faster, but only temporarily.

What is the difference between a 404 and a 410?

A 404 says the page was not found and a 410 says it is gone for good. Google's crawler documentation says it treats all 4xx errors except 429 the same way, so I do not count on a difference in speed. What matters is returning a real error status, not a 200 page that says "not found".

Should I redirect removed pages to the homepage?

No. Redirect to the closest equivalent page if one exists. If none exists, return a 404 or 410. Redirecting many unrelated URLs to one page can be treated as a soft 404.

How do I hide a staging site from Google?

Restrict access with a login or an IP allow-list. A robots.txt rule alone does not keep URLs out, and a copied Disallow or noindex can later cause real damage on production. Some site builders have their own staging switch, and the platform guide explains how Webflow's works and what it does not do.

Is the Removals tool permanent?

No. Google says a successful request lasts about six months, and it does not remove the page from the index or stop crawling. Pair it with a permanent method.

Sources and further reading

Statements about Google in this guide are based on the pages below, checked on 24 October 2026. How long removals take varies by site and page.

1. Introduction to robots.txt: robots.txt is not a mechanism for keeping a page out of Google; blocked URLs can still be indexed
2. Block Search indexing with noindex: noindex needs a crawlable page; noindex in robots.txt is not supported
3. Robots meta tag, data-nosnippet, and X-Robots-Tag specifications: noindex as a meta tag or header, including non-HTML files
4. Remove information from Google: update or remove content, password protection, noindex and the Removals tool
5. Removals tool: temporary removals last about six months and need a permanent fix
6. Managing crawl budget for large sites: 404 and 410 for removed pages; blocked URLs stay in the crawl queue longer
7. How HTTP status codes affect Google's crawlers: 4xx treated alike, soft 404s and persistent 5xx errors
8. Redirects and Google Search: permanent redirects and canonicalisation
9. Page indexing report: including "Indexed, though blocked by robots.txt"
10. robots.txt report: fetch status of your robots.txt file
11. URL Inspection tool: live test, indexing allowed and page resources