Two emails from Search Console arrived the same morning. The first was the one that worried me:
New reasons prevent pages in a sitemap from being...
For further actions, you may consider blocking this person and/or reporting abuse
This is the right kind of Search Console response: verify the actual page state before treating the report as a mystery. I would check source HTML, rendered DOM, headers, canonical signals, sitemap inclusion, and whether templates are applying noindex conditionally.
Your list is the right one, and two items on it did the work here.
Headers as well as HTML: an X-Robots-Tag does the same job invisibly, and checking only the markup is how people end up deciding the report is wrong.
And sitemap inclusion, which turned out to be the entire answer. All 101 URL shapes in my sitemaps came back 200 with no noindex anywhere, header or markup. The 51 were pages Google had reached by other routes — /api/, /cron/, /dl/*, account pages, plus the untranslated legal pages that are deliberately noindex outside English. Every one correctly excluded.
The conditional-template case you mention is the one I was braced for, and it wasn't that. Worth saying out loud, because "the report is right and your site is fine" is a real outcome and nobody writes that one up.
That distinction between sitemap URLs and discovered URLs is important. Search Console can look wrong when it is really reporting a crawl path you did not mean to expose. Checking headers, markup, status, and sitemap membership separately gives you a much cleaner answer.
"A crawl path you did not mean to expose" is a better description of it than the one I used in the post, and it points at the part I still cannot answer.
The sitemap was clean — I checked all 101 URL shapes and none carried a noindex. So Google reached /api/, /cron/ and /dl/ some other way, and I do not actually know which. They are not linked from any public page. Not from a sitemap, not from navigation.
My guesses in order: an old crawl from before those routes were locked down, a stray link in something I have forgotten about, or plain URL discovery from somewhere outside my control.
None of those are things I can rule in or out from my side, which is the mildly uncomfortable part — the report was right, my pages were right, and the interesting question turned out to be one Search Console does not answer at all: not "why is this excluded" but "how did you find it".
Have you found a reliable way to work out the discovery path? That is the one piece of this I would still like a method for.
That unknown crawl path is the interesting part. I would log it like a local SEO crawl incident: URL shape, first-seen date, referrer if available, sitemap presence, internal-link proof, robots state, and server logs around Googlebot. The fix is not only noindex cleanup; it is understanding which discovery surface made Google think those routes existed.
That list is the answer, and the item I would have skipped is the one that would have settled it: server logs around Googlebot. Everything else describes the page's current state, which is what I checked and why I got stuck — none of it records how the URL was first reached.
The referrer and first-seen date are the only two fields that point backwards, and they are exactly the two I do not have, because I went looking after the report rather than logging before it. Which is the honest shape of the problem: it was answerable, and it stopped being answerable at the moment Google crawled it.
Turning it into a standing record rather than an investigation is the part I am taking from this. A crawl-incident row written when the URL first appears costs nothing and is worth more than any amount of checking after the fact.
ngl i always struggle with finding which exact pages are triggering that noindex warning in GSC. thanks for showing the workflow.
The thing that made it tractable was not sampling randomly. A noindex almost always comes from a template or a route, so it hits a whole class of URLs at once — 50 random URLs out of 6,000 can miss a class of 51 completely.
Collapsing 5,881 URLs into 101 shapes (strip the language prefix, keep the first path segment, wildcard the rest) meant one fetch per shape covered everything. 101 requests instead of 5,881, and it structurally can't miss a class.
The other half is checking both places the tag can live — X-Robots-Tag in the response headers as well as the meta tag. Grep only the HTML and you'll come away convinced the report is lying to you.