Do not change a site-wide template until you have isolated the affected URL group and failure layer. Compare indexed and non-indexed examples from the same template across access, status, rendered content, robots, canonical, discovery, duplication and server logs, then test the smallest plausible change.
This is a technical SEO and web development incident. A broad redesign can erase the evidence, create new indexability problems and make the original cause impossible to identify.
Google's minimum technical requirements are that Googlebot is not blocked, the page returns a successful 200 status and the page has indexable content, while indexing is still not guaranteed Source: Google Search Central.
Define the symptom precisely
Record whether pages are undiscovered, discovered but not crawled, crawled but not indexed, excluded by directive, treated as duplicates or dropping after previously being indexed.
State the affected count, template, date range and business importance. Avoid using one manual site search as the diagnosis.
Build a representative sample
Select indexed and non-indexed URLs from the same template, including new, old, internally linked, orphaned, high-value and low-value examples. Add a known-good template for comparison.
Record content length only as context, not a quality verdict. Include query demand, duplication and whether the page has a distinct customer purpose.
Averages can hide one subcategory or release version that fails consistently.
Verify crawl access
Check robots.txt, authentication, firewall, CDN, bot protection, rate limiting, DNS and server logs. Confirm Googlebot can reach the URL and required resources.
Do not use robots.txt to remove an already known URL from the index. Google advises using noindex while allowing crawling when exclusion is intended Source: Google Search Central.
Review source and live configuration because deployment settings may differ.
Check the HTTP response
Inspect status, headers, redirect chain, content type and body from the public edge. A page displaying normally in your logged-in browser may return a challenge, empty cache or error to crawlers.
Find soft 404s that return 200 for missing or empty content. Check intermittent 5xx and timeout patterns in logs.
Compare raw and rendered content
Inspect initial HTML and rendered DOM. Verify the main heading, primary copy, links, canonical, robots and structured data exist and remain consistent.
Test slow or failed APIs, client exceptions and blocked resources. If important content appears only after a user gesture, the template may not expose it reliably.
Google recommends using URL Inspection to render a live page and verify what Google sees Source: Google Search Central.
Audit robots and canonical signals
Look for noindex in HTML and HTTP headers, including values inserted by middleware. Confirm the canonical points to the intended indexable URL and does not change after hydration.
Compare sitemap, internal links, redirects and canonicals. Conflicting signals make the preferred version harder to understand.
Check pagination, parameters, variants and localisation separately.
Verify discovery and internal links
Determine how crawlers find the URL. Use real anchors with href values from navigation, category pages, breadcrumbs and related content.
Check crawl depth, orphan pages, sitemap accuracy and last-modified values. A sitemap can aid discovery but does not replace useful site structure.
Check duplication and page purpose
Compare pages within the template for unique intent, product facts, location evidence or customer value. Parameter or programmatic pages may be technically indexable but substantially interchangeable.
Decide whether to improve, consolidate, canonicalise or exclude variants. Do not change markup to force every generated URL into the index.
Index coverage should follow a useful canonical page set.
Review template-wide quality signals
Inspect titles, headings, main content prominence, intrusive overlays, broken navigation, mobile usability and performance. Confirm advertisements or widgets do not displace the answer.
Look for stale placeholders, empty fields and repeated boilerplate caused by missing source data. Fix data completeness as well as presentation.
Sample actual customer tasks on the page.
Correlate with releases and logs
Overlay indexing changes with template deployments, routing, CDN, CMS, feed and internal-link releases. Inspect Googlebot requests, response codes, render resources and crawl frequency.
Do not assume correlation proves cause. Use it to narrow the test.
Preserve deploy hashes and feature-flag states for representative URLs.
Decide whether a template change is justified
Write the observed evidence, suspected mechanism and expected measurable change. If only one data source or content group fails, repair that layer rather than rebuilding every template component.
Define affected routes, rollback, SEO signals and visual non-goals. Keep the public UI stable unless a functional issue requires change.
Estimate recrawl time without promising an indexing date.
Run a controlled release
Apply the change to a representative group using a feature flag or bounded template variation. Keep a comparable group unchanged where practical.
Validate status, content, metadata, links, schema, performance and analytics before release. Submit only a small sample for inspection rather than repeatedly requesting every URL.
Monitor errors and crawler behaviour immediately.
Measure the right outcome
Track successful crawls, canonical selection, index status, impressions, relevant queries, qualified traffic and conversions. Allow time for recrawl and reassessment.
An increase in indexed URLs is not a win if they earn no useful visibility and create duplication. Measure the important canonical set.
Document pages that remain excluded and the next hypothesis.
My take: templates magnify both fixes and mistakes
My take is that a template change has attractive leverage, which is exactly why diagnosis must come first. A single corrected directive can repair thousands of pages; a mistaken canonical can damage the same number.
Use that leverage only after representative evidence points to the shared layer.
Prevent recurrence
Add automated checks for status, title, canonical, robots, heading and key content on every public template. Crawl preview deployments and alert on unexpected URL or directive changes.
Assign SEO review to routing, rendering, CMS and middleware changes. Keep an indexability runbook.
Compare production variants
Check whether the affected template changes by device, language, location, login state, experiment, CDN region or data-source version. A clean test from the office can miss an edge response served to unauthenticated crawlers elsewhere.
Repeat representative requests with an empty cache and production headers. Compare HTML hashes and critical fields across variants, then remove experiments from the diagnosis or analyse each variant separately. Preserve the evidence so a later cache purge does not erase the failure pattern.
Frequently asked questions
Does “crawled, currently not indexed” mean the template is broken?
Not necessarily. Review duplication, usefulness, canonical selection and site context as well as technical rendering.
Should I request indexing for every page?
No. Fix the shared cause and let normal discovery work. Use inspection for diagnosis and important samples.
Can a new template guarantee indexing?
No. It can remove technical barriers and improve usefulness, but search systems decide whether to index a URL.
How long should I wait after a fix?
It depends on crawl frequency, site size and change type. Monitor logs and inspection evidence rather than promising a universal period.
Change the template only when evidence reaches it
If your business needs an indexing diagnosis and controlled repair, get in touch for technical SEO.
Related guides
For further background, read our seo audit guide and search intent. The javascript seo glossary explains the terminology used in those guides.