Our site has an indexing gap: 2,478 known pages, 208 indexed. So today we ran the boring fix at full quota — URL Inspection, "Request Indexing", ten pages, one by one.
Nine went through: tested, queued for priority crawling. The tenth — our brand-new signup page — came back rejected, and the reason is worth more than the nine successes:
noindex in an HTTP X-Robots-Tag header.
Not a meta tag. Not robots.txt. The page fetched fine, was allowed to crawl, allowed to index — and the header quietly overruled all of it. Classic pattern: auth-adjacent pages (signup, login, account) get a blanket noindex from middleware, and one day the page you actually want ranking inherits it.
Three checks that catch it in a minute:
-
curl -sI https://yoursite.com/page | grep -i robots— see the header directly - URL Inspection in Search Console — the page-indexing section names the exact blocker, in our case the X-Robots-Tag header
- Test the specific route, not the template — middleware rules apply per-route and bite where you least expect
The meta lesson for us: we spent four days thinking the Request Indexing button was broken (it wasn't opening in our setup — that's another story). The tool worked; our process for using it didn't. Indexing tools fail quietly, in both directions.
(For the record: of the nine queued pages — core /app/ routes — one had been crawled but never indexed; the other eight were discovered and never crawled. Requesting indexing doesn't guarantee inclusion; it puts them in a priority queue. The scoreboard is the indexed-page count, and we'll report it honestly whatever it does.)
Check your signup page's headers before you assume Google is ignoring you. Ours was — because we told it to.
Top comments (1)
Good catch on testing the route and not the template. To list every page that inherited it we would run the header check over the whole sitemap:
curl -sL example.com/sitemap.xml | grep -o "<loc>[^<]*" | cut -c6- | while read u; do curl -sI "$u" | grep -qi "x-robots-tag:.*noindex" && echo "$u"; doneEvery URL it prints tells Google not to index that page. If the sitemap is an index file run it once per child sitemap.