Why Your Important Pages Aren't Getting Indexed (and How an Audit Finds the Cause)

What indexing success looks like

Success is not "every page indexed". It is that the pages you want to rank are indexed, and the pages you do not want (internal search, filters, thank-you pages) are not. I start every indexing investigation by writing down the list of pages that must be indexed: the homepage, key service or product pages, important categories and the content that supports them. Everything else is secondary.

Indexing is also never guaranteed. Google states that submitting a sitemap is merely a hint, and that requesting indexing in Search Console does not guarantee a page will appear in the index. A page can be fine and still wait. What I look for is a pattern: a whole template, section or type of page that stays out, because that points to a cause I can fix.

Where is the page stuck?

Google finds a URL, crawls it, renders it and decides whether to index it. The Page indexing report in Search Console tells you which stage a URL reached. These are Google's own definitions, with the first thing I check for each.

If you are staring at thousands of rows across these statuses, group them first; the 12 audit errors that actually matter shows how.

Noindex and robots directives

A noindex rule tells Google not to show a page in results. It can sit in a robots meta tag in the HTML or in an X-Robots-Tag HTTP header, which is how PDFs and images are controlled. The classic failure is a single template or setting that applies noindex to a whole section.

• Template-level rules affect every page that uses the template. Check one affected URL and one unaffected one and compare the output.
• Leftovers from staging are the cause I see most often after a launch. A test site is protected with noindex or a robots.txt block, and the setting travels to the live site with everything else. Google lists removing these blocks as part of any site move. To stop a block from shipping in the first place, see How a Single Deploy Can Wipe Out Your Organic Traffic.
• Plugins and CMS settings can add or remove directives after an update, so a page that was fine last month can change without anyone editing it.

I test any suspect rule with the Robots.txt Tester and check headers with the HTTP Header Viewer.

Canonical signals and duplicate content

When several URLs show the same content, Google picks one as canonical and indexes that one. You can guide the choice with redirects (a strong signal), the rel="canonical" tag (also strong) and the sitemap (a weaker one), and these can stack. Google may still choose differently from the one you declare. Trouble starts when the signals disagree: a canonical pointing to a redirected or noindex URL, a sitemap listing the non-canonical version, or internal links pointing to a different variant than the canonical.

• Use one canonical method and be consistent. Use absolute URLs, and put the tag in the head of the page.
• Give the canonical page a self-referencing canonical too.
• "Duplicate without user-selected canonical" is not necessarily an error. Inspect the URL, see which canonical Google chose, and decide whether that is what you want.
• If a page is not really a duplicate, make its content substantially different, rather than relying on a tag.

The Canonical URL Checker tests batches of pages for missing, conflicting or chained canonicals. For online stores, where filters and variants create thousands of near-duplicate URLs, see the faceted navigation section of SEO for Nepali Ecommerce.

Rendering and JavaScript

Google crawls the HTML, queues the page for rendering, then runs the JavaScript in a headless browser and indexes the rendered result. The page may wait in that queue for a few seconds or longer. Three problems come up repeatedly:

• Content that appears only after a click or scroll event may never be seen by Google.
• Links that are not real anchors. Google can only discover links that are anchor (<a>) elements with an href attribute.
• Blocked resources. Google does not render JavaScript from files or pages that robots.txt blocks.

To see what Google actually gets, use the URL Inspection tool's live test. The rendered HTML and screenshot are only available in a live test. If the main content or navigation is missing from the rendered version, the fix is usually to render it on the server rather than rely on the browser. Single-page apps should also make sure error views return a real 404 status, or add a noindex tag to them, to avoid soft 404s.

Crawl budget and site structure

Crawl budget is the most over-diagnosed cause I see. Google's guidance says it matters for very large sites, roughly a million or more unique pages that change weekly, medium or larger sites of 10,000 or more pages with rapidly changing content, or sites with a large share of URLs sitting as "Discovered, currently not indexed". Google itself says these numbers are rough estimates and that a site whose pages are crawled the same day they are published does not need to worry. For most business sites in Nepal, the real causes are elsewhere.

Where crawling efficiency is a real issue, the advice from Google is practical: remove duplicate URLs, return a real 404 or 410 for removed pages (a 404 is a strong signal not to crawl the URL again, whereas blocked URLs stay in the queue longer), avoid long redirect chains, and keep the sitemap current with accurate lastmod dates.

Structure matters for a simpler reason: Google finds most pages through links. A page that nothing links to relies on the sitemap alone. I check that every important page is reachable in a few clicks, with descriptive anchor text, and I use the Internal Link Analyzer to find orphan pages and click depth. A tidy sitemap helps too; the XML Sitemap Validator shows whether it lists blocked, redirected or dead URLs.

Technical blockers that hide pages

• Server errors and overload. Persistent 5xx or 429 responses make Google slow down, and URLs that keep failing are eventually dropped. The Crawl stats report shows host status and the response code mix.
• Firewalls, CDNs and bot protection that block or challenge Googlebot. The page works in your browser but not for Google. Test with the live URL test, and verify genuine Googlebot traffic using Google's reverse DNS method or its published IP ranges rather than trusting the user agent name.
• DNS and TLS faults, including an expired certificate, make a site unreachable intermittently. The SSL Certificate Checker catches the second.
• Redirect chains and loops that never reach a final, indexable URL. Trace them with the Redirect Checker.

When Google crawled it but chose not to index it

"Crawled, currently not indexed" is the status that frustrates people most, because Google does not state a cause. In my audits I treat it as a question about the page itself. I compare the page with similar pages Google did index and ask:

• Is it substantially different from other pages on the site, or a variation of one?
• Does it say something useful that is not already said better elsewhere?
• Do important pages link to it, so the site itself treats it as important?
• Does it show who is responsible for it and where its claims come from? Google's guidance on helpful content asks about the creator, how the content was made and why it exists, and says trust is the most important element.

I do not resubmit these pages repeatedly. Google says there is no need. I improve them, link to them from relevant pages, and give Google time. If the pages come from a template and a data feed, the real question is whether each one should exist at all, and Programmatic SEO Without the Penalty Risk gives the decision test I use. This matters beyond classic results too: Google says that to appear in AI Overviews and AI Mode a page needs to be indexed and eligible to appear in Search with a snippet, with no other special requirements; the wider picture is in The Future of SEO in Nepal.

An audit workflow to diagnose and fix

1. List the pages that must be indexed, ranked by revenue or leads, and group the rest by template.
2. Inspect each in Search Console. Read the indexed status, the last crawl date, the referring sitemaps, and the declared and Google-selected canonicals.
3. Match the status to its stage using the table above, and find the pattern across a whole template or section.
4. Run a live test on a sample to see what Google gets now: the rendered HTML, the headers and any blocked resources.
5. Check the rules: robots.txt, meta robots, X-Robots-Tag and the canonical on the page.
6. Check discovery: internal links to the page and its presence in an accurate sitemap.
7. Check access: server logs and Crawl stats for errors, slowness and blocks.
8. Fix the cause, not the symptom, at template level where possible, and record the change.
9. Validate. Retest with the live URL test, use the validation option in the Page indexing report, and request indexing for the most important URLs only. Google notes there is a daily limit and that indexing can take up to a week or two.

If the problem started right after a redesign, the first suspect is the move itself, which the SEO migration audit checklist walks through. And if you are not sure whether you need a narrow technical audit or a complete one, this comparison will help you choose.

Frequently asked questions

Why is a live page not indexed?

Common causes are a noindex rule, a robots.txt block, a canonical pointing to another URL, content or links that only exist after JavaScript, server errors, no internal links, or a page too similar to another. The Page indexing report shows which stage the page reached.

How long does indexing take?

Google says it can take up to a week or two after a request, and it offers no guarantee. A new page on a site Google already crawls often appears sooner, but there is no fixed timetable.

Does a sitemap guarantee indexing?

No. A sitemap is a hint that helps discovery. Google may still decide not to index a page, particularly if it is duplicate or adds little.

Should I use robots.txt to remove a page from Google?

No. robots.txt blocks crawling and can leave a bare URL in the index. Allow crawling and use a noindex directive, or remove the page and return a 404 or 410.

Is crawl budget why my pages are not indexed?

Rarely. Google reserves crawl budget guidance for very large sites. For most sites the cause is a directive, a canonical, rendering, server access, internal links or page quality.

Can I fix this myself?

Often, yes. Search Console, the free tools linked above and Google's documentation are enough to find most causes. If you want to learn the process, see How to Learn SEO in Nepal. For a site with several overlapping problems, an expert-led SEO audit saves time.

Sources and further reading

Technical statements about Google in this guide are based on the pages below, checked on 17 October 2026.

1. Page indexing report: status definitions and suggested fixes
2. URL Inspection tool: indexed version, live test and request indexing
3. Robots meta tag, data-nosnippet and X-Robots-Tag: noindex and robots.txt interaction
4. How to specify a canonical URL: canonical signals
5. JavaScript SEO basics: rendering, links and soft 404s in single-page apps
6. Crawl budget management for large sites: who needs it and what helps
7. HTTP status codes, and network and DNS errors: 5xx, 429 and 404 handling
8. Build and submit a sitemap: a sitemap is a hint
9. Crawl Stats report: host status and response codes
10. Verifying Googlebot and other Google crawlers: reverse DNS and IP ranges
11. Site moves with URL changes: removing migration-only blocks
12. Creating helpful, reliable, people-first content: Who, How, Why and E-E-A-T
13. AI features and your website: requirements for AI Overviews and AI Mode