Of My Unindexed Pages, 5,511 Trace Back to My Own Site

Search Console's page indexing report has two boxes. The green one says "Indexed: 2.41K", the grey one says "Not indexed: 8.41K". The report was last updated on 12 October 2026. Of every four pages Google knows on this blog, three are not in the index.

On a first read this looks like a report card and the grade is bad. The grey column is three and a half times the green one, and on the chart the grey part has been growing since July.

But under the chart there is a column nobody reads at first glance: Source. Next to every row it says either "Website" or "Google systems". Add that column up and the table turns into something else. Most of the grey column is not Google's verdict about me. It is an itemised list of what my own site has been telling Google.

This post tries to open that list one row at a time. Along the way it also shows two rules, each set up as the other's justification, that cancel each other out.

The table and the Source column

The "Why pages aren't indexed" table lists 13 reasons. The first 10 are visible on screen:

Reason	Source	Pages
Page with redirect	Website	2,741
Blocked by robots.txt	Website	1,652
Alternate page with proper canonical tag	Website	463
Not found (404)	Website	320
Excluded by "noindex" tag	Website	305
Redirect error	Website	18
Duplicate without user-selected canonical	Website	10
Server error (5xx)	Website	1
Blocked due to unauthorized request (401)	Website	1
Crawled - currently not indexed	Google systems	2,629

The visible "Website" rows add up to 5,511. The single "Google systems" row is 2,629. The ten visible rows come to 8,140; since the 8.41K in the box is rounded, I can only derive that the three rows I did not see hold roughly 270 pages. I have not read those rows.

Google's help page explains the column in one sentence: the source value shows whether the source of the issue is Google or the website. Right after that it adds that, in general, you can fix only issues where the source is listed as "Website".

So, going by the visible rows, roughly two thirds of the grey column is in my hands. Even if all three rows I didn't see turn out to be Google's, Google's share stays under 34.5 percent; if all of them are the site's, my share rises from 5,511 to about 5,780. The number in the title is therefore a lower bound.

Every row also shows a validation status of "Not started". It is tempting to read that as "Google hasn't looked yet". It isn't. According to the help page, it means no validation request has ever been sent for that issue. Sending one is my job.

My sitemap gives 2,735 URLs, Google knows 10,820

First I pulled the number on my side. The live sitemap carries 2,735 URLs: 1,354 under /blog/, 1,354 under /en/blog/, six series pages in each language and a handful of static pages. No tag pages.

The total Google knows is the sum of the two boxes, roughly 10,820. That leaves close to eight thousand URLs in between. None of them is in today's sitemap, but it wasn't always so. A commit dated 4 June 2026, which removed tag pages from the sitemap, recorded the situation that day: 1,164 unique tags, 2,257 thin tag pages, 60 percent of the sitemap. The sitemap went from 3,788 URLs to 1,436 that day. The duplicate posts deleted on 15 June were also in the sitemap until then.

Google's sitemap documentation is clear about this: include the URLs you want to see in Google's search results; if several URLs lead to the same content, choose the one you prefer rather than all of them. And a sitemap is merely a hint, it does not guarantee crawling.

So Google learned some of these addresses from my own sitemap at one point; the rest from links inside my pages, old addresses, links from other sites. For most of them, the code base shows where they come from.

2,741 redirects: every post has four twins

I took one post and tried the variants of its address:

https://mustafaerbay.com.tr/blog/life/<post>     → 301 → .../blog/life/<post>/
http://mustafaerbay.com.tr/blog/life/<post>/     → 301 → https://...
https://www.mustafaerbay.com.tr/blog/life/<post>/ → 301 → https://mustafaerbay.com.tr/...
https://mustafaerbay.com.tr/blog/<post>/         → 301 → .../blog/life/<post>/
https://mustafaerbay.com.tr/blog/life/<post>/    → 200

One post, five addresses. One of them is the page itself, four redirect to it: the version without the trailing slash, http, www, and the old format without a category.

Each of these is a deliberate decision, and the code records it with a date. A comment in middleware.ts says the trailing-slash redirect was added on 15 May 2026, and why: Search Console was indexing both /blog/X and /blog/X/ as separate pages, about 922 duplicates by the comment's count. The 301 that rescues category-less addresses was moved from page code into middleware on 26 June.

There is one more item, and it has to do with the blog's own history. The file src/lib/redirects.ts was created on 15 June 2026 with this commit: "near-duplicate content merge — 38 dup → 7 canonical". The blog's AI pipeline had produced the same narrow topic again and again between April and June; a single PostgreSQL WAL topic has eight duplicates in that map. Today the map holds 72 deleted slugs pointing at 13 canonical posts. Each entry covers both the Turkish and the English address, so 144 URLs redirect from a post that no longer exists to one that does.

Google's definition of "Page with redirect" is short: a non-canonical URL that redirects to another page. The definition suggests no fix; I don't think there is anything to fix here either, because these redirects do exactly the job I want. I cannot know which URLs make up the 2,741 without exporting Google's example list. With four twins per canonical URL the ceiling is above ten thousand; what Google counted is less than a quarter of that. That this row looks like the biggest problem at first glance is the most misleading thing about the table.

1,652 robots.txt blocks, and two rules that cancel each other out

The only path in robots.txt that concerns Googlebot is this:

User-agent: *
Allow: /
# /tags/ sayfaları noindex + düşük değerli — crawl bütçesini boşa harcamasın.
Disallow: /tags/
Disallow: /en/tags/

(The comment reads: "/tags/ pages are noindex and low value — don't waste crawl budget.")

That is why I infer that the 1,652 blocked URLs are tag pages. I have not seen the example list; it is an inference. Published posts carry 1,261 distinct Turkish tags and 1,264 distinct English ones; Until 4 June the sitemap held 2,257 tag pages; Google didn't even need to follow links to know these addresses. Today the /tags/... links under every post keep pointing at them.

The interesting part is the comment itself: "/tags/ pages are noindex". True, the page template says so too:

src/pages/tags/[tag].astro:26:  robots="noindex,follow"

And the live site agrees; a tag page comes back with a noindex,follow meta tag.

Here is the problem. According to Google's documentation on blocking indexing, for noindex to be effective the page must not be blocked by robots.txt. If the crawler cannot get in, it never sees the noindex, and the page can still appear in search results if other pages link to it. The robots.txt introduction says the same thing from the other side: robots.txt is not a mechanism for keeping a web page out of Google, and a blocked URL can still show up in results without a description.

The history of both rules is in git. The noindex came on 4 June. The Disallow came two days later, on 6 June, for a different reason: according to the commit message the site was running in a single Node process at the time, bot bursts (especially the long-tail crawl under /tags/) were choking that process and producing around 640 504 errors a day. The same commit made the server multi-process and added the Disallow to robots.txt; its justification line also says "noindex + low value".

So the two rules were set side by side, one as the reason for the other. One says "do not index this", the other stops Google from reading that sentence. According to the documentation, the second one silences the first.

Whether this costs anything, I don't know for certain. The Disallow was meant to reduce crawl load; I have no measurement of how large that load is today. On the cost side, what the documentation says is that tag addresses can show up as results without a description. Rather than guess whether that is happening, checking the warnings on the report's "Indexed" side is a cheaper next step.

There are two clean paths. Either remove the Disallow: /tags/ line and let Google see the noindex, in which case tag pages get crawled and stay out of the index. Or treat the noindex as symbolic and take the crawl savings. The current state sits between the two.

The first path isn't free, though. The Disallow line was added because of 640 504s a day. Since the server went multi-process the same day, that load may not be a problem today; "may not be" is not the same as "measured".

noindex 305: a page in my sitemap says "don't take me"

The source of the 305 noindex row is also visible in the code. Pages carrying noindex: blog pagination (/blog/page/N/), category pages, search, bookmarks, subscription result, 404, and /hakkimda.

That last one is small but instructive. /hakkimda ("about me" in Turkish) is an alias: a page that immediately redirects Turkish searchers to /about, says noindex, and points its canonical at /about. Sensible so far. But the same address is also in my live sitemap.

So my sitemap tells Google "I want to see this page in search results", and the page itself says "don't index me". A one-URL contradiction, and the interesting part is that a safeguard for exactly this already exists. A 6 July comment on the sitemap filter in astro.config.mjs says that telling Google both "crawl this" and "don't index this" is a contradiction, and drops noindex pages from the sitemap: analytics, search, subscribe-result, secrets, bookmarks, admin, ops. hakkimda is not on the list. The safeguard is right, the list is incomplete. The noindex decision lives in the page's own file, the sitemap exclusion list is kept by hand in another file, and nothing ties the two together.

Small rows, small stories

463 alternate pages. By Google's definition these pages correctly point to their canonical page, which is indexed; nothing to do. When I request a post with a parameter such as ?utm_source=x, the canonical tag points to the address without the parameter; part of this row is probably addresses like that, but I can't say for sure without seeing the examples.

320 not-found pages. A 4 June comment in middleware.ts recorded the table that day as "about 194 historical 404s" and closed old address patterns with 301s. Today the number is 320. The 72 duplicate slugs deleted since June redirect, they don't return 404; I can't see where the increase comes from on this screen.

18 redirect errors. In Google's definition this category covers redirect chains that are too long and redirect loops. On 25 July 2026 a commit fixed a loop where 32 English posts were 301-ing to themselves: 7f7563e4, "32 EN post 301 self-loop". Whether the 18 are leftovers of that loop, again, I can't say without looking at the examples. No validation has been requested for this row either.

Google's own call: 2,629

This is the only row in the table whose source is "Google systems" and, to me, the most important number in this post.

"Crawled - currently not indexed." The definition is two sentences: the page was crawled by Google but not indexed; it may or may not be indexed in the future. No redirect, no block, no noindex. Google read the page and, for now, did not take it.

This row is not about my signals; it is a decision Google made after reading the page, and it gives no reason. Its size is on the same order as the indexed total: 2,629 against 2,410.

There is one more point of comparison. The 4 June commit also recorded that day's report: 2,854 pages indexed, 3,570 not indexed. That day the sitemap held 1,436 URLs including posts, series and static pages; today it holds 2,735. While the number of URLs in the sitemap nearly doubled, the number of indexed pages fell by 444. The Source column doesn't explain that.

I don't know which pages are in this row. I won't say anything definite without knowing. But I see two candidates, and both are in this blog's own records. One is the near-duplicate period described above: from an archive that tells the same topic eight times, Google cannot be expected to want all of them. The other is what came out of the post where I counted my blog's recommendation engine: 476 posts received a link neither from a sentence I wrote nor from a recommendation card; only list and archive pages point at them. The pages in this row have already been crawled, so discovery is not the problem; but it would not be surprising if Google doesn't find important a post that no other post on my own site mentions. Both are hypotheses. The next measurement is to export this row's example list and intersect it with those 476 posts and with the deleted duplicate clusters.

In the 13,975 views post in this series, I counted only six views from search in thirty days. How many of the 2,410 indexed pages actually answer a query is a separate question this report does not answer.

Reading your own report

This report sits in the same place in everyone's Search Console. In the order I took from this measurement:

1. Source column first. Split the number in the grey box into "Website" and "Google systems". The first is what you told Google, the second is Google's decision.
2. Compare the number of URLs in your sitemap with the report's total. The gap is the URLs Google found through paths you didn't show it. If it is large, ask where they come from.
3. Compare every path you close in robots.txt with the pages you mark noindex. If they meet on the same address, the noindex is not being read.
4. Make sure none of the URLs in your sitemap carry noindex. One turned up in mine, despite a filter written for exactly this. If you have such a filter, compare its list with your noindex decisions.
5. "Not started" means it's your turn. If you fixed a row and don't start validation yourself, nobody does.
6. Export the example list of "Crawled - currently not indexed". The real question is there.

The size of the grey box says nothing on its own. Google's own documentation says that not being indexed is not necessarily bad; and the 2,741 redirect row comes from rules I wrote.

Before looking at this table I had one number: 8.41K. After, I have three. 5,511, what my site tells Google; inside it sat two rules, each set up as the other's justification, silencing each other. 2,629, what Google tells me. And 444: the pages that dropped out of the index while my sitemap doubled. The Source column separated the first two. For the third, that column is not enough.

Official Sources

• Google Search Console Help — Page indexing report — the Source column, fixability of "Website"-sourced issues, the definition of each status and the "Not started" validation state.
• Google Search Central — Block search indexing with noindex — that for noindex to be effective, the page must not be blocked by robots.txt.
• Google Search Central — Introduction to robots.txt — that robots.txt is not a mechanism for keeping a page out of Google, and a blocked URL can appear in results without a description.
• Google Search Central — Build and submit a sitemap — that sitemaps should list the canonical URLs you want in search results, and that a sitemap is merely a hint.