Google said 51 pages were excluded by noindex. Here's how I checked in ten minutes.

Two emails from Search Console arrived the same morning. The first was the one that worried me:

New reasons prevent pages in a sitemap from being indexed on site utilorax.com
Excluded by 'noindex' tag

Read that carefully. Pages in a sitemap — pages I am explicitly asking Google to index — are being skipped because they carry a noindex. That is a contradiction, and it is the kind that quietly costs you: the sitemap says "index this", the page says "don't", and Google believes the page.

Fifty-one of them. My site has around 6,000 URLs across five languages, so 51 was small enough to be a real bug and large enough to matter.

Here is how I found out whether it was one, without opening 51 URLs by hand.

The method: sample by URL shape, not at random

Random sampling is the wrong tool here. A noindex bug is almost never random — it comes from a template, a route, or a language variant, so it affects a whole class of URLs. If you sample 50 URLs out of 6,000 at random you will probably miss a class of 51. If you sample one URL from every distinct shape, you cannot.

So:

1. Collect every URL the sitemaps actually claim. Not what you think you publish — what the XML says today.

curl -s https://example.com/sitemap.xml \
  | grep -o '<loc>[^<]*</loc>' | sed 's|</\?loc>||g' > maps.txt

while read m; do
  curl -s --retry 2 "$m" | grep -o '<loc>[^<]*</loc>' | sed 's|</\?loc>||g'
done < maps.txt | sort -u > allurls.txt

That gave me 5,881 URLs across 54 sitemap files.

2. Reduce them to shapes. Strip the language prefix, keep the first path segment, mark anything deeper with a wildcard. /es/convert/miles-to-km and /es/convert/kg-to-lb are the same shape; /es/blog/some-post is a different one.

5,881 URLs collapsed into 101 distinct shapes — en /convert/*, es /convert/*, fr /tools/*, en /about, and so on.

3. Fetch one of each and check both places a noindex can hide. This is the step people get wrong: they check the HTML meta tag and forget the header, or the reverse.

while read u; do
  H=$(curl -s -m 30 -D - -o body.html "$u")
  XR=$(echo "$H" | grep -i "^x-robots-tag")
  MR=$(grep -o '<meta name="robots" content="[^"]*"' body.html | head -1)
  if echo "$XR $MR" | grep -qi "noindex"; then echo "HIT: $u"; fi
done < sample.txt

Result: 101 out of 101 returned 200, and not one carried a noindex — in the header or in the markup. No contradiction. The sitemap and the robots directives agreed everywhere.

Ten minutes, and the scary email was answered.

So what were the 51?

Once the sitemap was cleared, the answer had to be pages Google found some other way — through internal links, old crawls, or the wider web. Grepping my own routing for every place that sets a noindex gave the list immediately:

• /api/* — JSON endpoints
• /cron/* — scheduled jobs
• /dl/* — file downloads
• /developers/* — logged-in account pages
• and eight pages I had to work out: the es, fr, pt and zh versions of /refund and /terms, which have no translation and are deliberately noindex outside English until someone writes one

Every one of them is noindex on purpose. Google was not reporting a bug. It was reporting a fact, in a message worded strongly enough to look like a bug.

That is the part worth keeping. "Excluded by 'noindex' tag" is not an error. It is Search Console telling you what it found, and on any site with an admin area, an API or logged-in pages, a non-zero number there is correct and healthy. The number to react to is not the count — it is whether any of those URLs are ones you wanted indexed.

The other number I had been reading wrong for a week

While I was in there, the Page Indexing report finished processing for the first time since launch. It said:

Indexed        3,040
Not indexed    3,237
Total known    6,277        ->  48.4% indexed

My notes said 12%.

That figure had come from the HTTPS report, which showed 728 URLs at a time when the indexing report was stuck on "processing". I had divided 728 by my sitemap total and written down "only 12% is indexed, so indexing is the bottleneck" — and then let that sentence drive my priorities for a week.

The HTTPS report counts URLs whose HTTPS status Google evaluated. It is a sample. It was never the index size, and it never claimed to be. I had made it answer a question it was not asked.

Half of a twelve-day-old site being indexed is not a bottleneck. It is fine. The actual bottleneck was somewhere else entirely, and I had spent a week not looking at it.

The one that was genuinely worth opening

The bigger bucket was "Crawled — currently not indexed": 215 pages. That one means Google fetched the page, read it, and decided against it — which is a judgement, not a mechanic, and worth understanding.

I pulled the full list and counted it rather than eyeballing the examples:

Non-English	97% (es 38%, fr 35%, pt 23%)
/convert/* pages	71%

So: non-English converter pages, almost exclusively.

The obvious theory is thin or machine-translated content. I tested it by comparing a Spanish page against its English counterpart word by word — they share only 17–21% of their vocabulary and each runs about 1,000 words. The translations are real.

What I did find was worse in a subtler way: three places built an English sentence and dropped translated nouns into the gaps. The breadcrumb read "Caballos de Fuerza to Vatios". Related cards read "Convert Caballos de Fuerza to Kilovatios instantly." Eleven English fragments per page, on every converter, in four languages.

That is a genuine quality bug and I fixed it. But I want to be honest about what fixing it does, because the temptation is to declare victory: it is almost certainly not why those pages were skipped. A twelve-day-old domain with zero external links does not get unlimited index budget, and non-English pages of a site nobody has ever linked to are the first thing dropped. That is an authority problem. No amount of copy editing fixes it — links and time do.

What I would tell myself a week ago

1. Sample by URL shape, not at random. Template bugs affect classes, so test classes.
2. Check the header and the meta tag. A noindex can live in either, and the report does not tell you which.
3. "Excluded by noindex" is usually correct. Verify it is only hitting things you meant.
4. Never infer one report's number from another's. The HTTPS report is not the index count. A number that is roughly the right shape is the easiest kind of wrong to keep believing.
5. When you find a real bug during an investigation, do not let it become the answer. It felt like the cause. It was not.

---

I build Utilorax, a set of free browser-based tools. Everything runs client-side, which produces a steady supply of problems like this one.

Comments

[Alex Shev]

This is the right kind of Search Console response: verify the actual page state before treating the report as a mystery. I would check source HTML, rendered DOM, headers, canonical signals, sitemap inclusion, and whether templates are applying noindex conditionally.