A site ships, weeks pass, and the pages are still not in the index. The usual advice jumps straight to content quality. Before that, there is a list of mechanical causes that are far more common and much easier to verify.
1. Can it actually be fetched
Use the URL inspection tool and fetch the live page. Not your browser — your browser has your cookies, your session, and possibly a cached copy.
Things that show up here regularly:
- The page returns 200 to you and 403 to the crawler, usually via a bot-protection rule.
- The HTML is nearly empty because content renders client-side after a fetch the crawler did not wait for.
- A
noindextag left over from staging. This is embarrassingly common and takes ten seconds to check.
Check robots.txt too. A Disallow written for a staging path can match more than intended.
2. Canonical tags pointing elsewhere
If page A declares page B as canonical, you have asked search engines not to index A. That is working as designed.
The ways this happens by accident:
- A template with a hardcoded canonical, so every page claims the homepage.
- Cross-posting to a platform without setting the canonical there, so the copy on the bigger domain wins.
- Protocol or host mismatch — canonical says
https://www.example.com/page, the site serveshttps://example.com/page. Those are different URLs.
Every URL should declare itself canonical unless you genuinely mean otherwise.
3. Redirect chains from trailing slashes
/page redirects to /page/, which redirects to https://example.com/page/. Each hop costs a request and dilutes the signal, and if your sitemap lists the pre-redirect form, every crawl starts by bouncing.
Pick a form. Make internal links, the sitemap, and the canonical all use it. Then check what your host does automatically, because it may be adding a hop you did not configure.
4. Sitemap says one thing, the site says another
A sitemap generated from your content config lists what should exist. If the build dropped a page, the sitemap still lists it and you are pointing crawlers at 404s. Enough of those and the whole file is trusted less.
Generate the sitemap from the built output on disk, not from the source config. Then it cannot describe pages that were not shipped.
While you are there: lastmod should reflect real changes. Setting it to today's date on every build teaches crawlers that your dates mean nothing.
5. Slugs that changed without redirects
If URLs changed during a redesign and the old ones return 404, you did not migrate — you deleted a site and published a new one. Everything the old URLs had earned is gone.
Old URL to new URL, 301, one hop each. Keep them indefinitely. Links on other people's sites do not update themselves.
6. Near-duplicate pages competing
Twelve pages built from one template with a city name swapped in are, functionally, one page. Search engines pick one and ignore the rest, and "crawled, currently not indexed" is exactly what that looks like.
The fix is not more pages. It is fewer pages that each say something the others do not.
7. It has genuinely not been long enough
New domains get crawled slowly. Weeks is normal, not a symptom. Submitting the same URL repeatedly does not speed it up.
Do the useful things instead: internal links from pages that are already indexed, a clean sitemap referenced in robots.txt, and links from somewhere real.
A note on tooling
Check more than one search engine's webmaster tools. They surface different diagnostics, and a crawl problem that shows up in one is often the same problem in both — sometimes reported more clearly on the side you were not looking at.
The pattern I keep seeing: the cause is almost always in items 1 through 5, and almost always something mechanical that a person could have checked on day one. Content is worth improving. It is just rarely the reason a page is missing from the index entirely.
Top comments (0)