My site has one page in Google's index and 79 in the sitemap. I assumed the content was being crawled and judged. The access log says it was never crawled at all.
Two days of raw log, every Googlebot request:
66.249.72.1 - [04/Sep/2026:19:57:09] "GET /robots.txt HTTP/1.1" 200 4951
66.249.72.8 - [04/Sep/2026:19:57:10] "GET /post-sitemap.xml HTTP/1.1" 200 2793
66.249.72.2 - [04/Sep/2026:23:04:52] "GET /category-sitemap.xml HTTP/1.1" 200 5267
Three requests. It read robots.txt, read both sitemaps, and fetched zero articles.
Same window, same server, other crawlers:
YandexBot 154 requests
bingbot 139 requests (64 distinct article URLs)
ClaudeBot 7 requests
Googlebot 3 requests (0 article URLs)
PerplexityBot 1 request
GPTBot 0 requests
Bingbot pulled 64 different articles plus cover images and the WP REST endpoints. Same URLs, same robots.txt, same response times.
That rules out most of what I would otherwise have spent a week on. Not robots.txt, Googlebot fetched it and it disallows nothing relevant. Not the sitemap, it fetched that too and got a 200 with 79 URLs in it. Not the content, it has never seen the content. Not server errors, everything returned 200. It knows exactly which URLs exist and is choosing not to spend requests on them.
If you have shell access, this is the whole check:
grep -i googlebot access.log | grep -oE '"GET [^ ]+' | sort | uniq -c | sort -rn
If the only lines that come back are robots.txt and your sitemaps, no amount of on-page work will move anything, because the crawler never reaches the pages you are editing. Search Console will not tell you this either — it reports "URL is unknown to Google", which reads like a submission problem rather than a budget decision.
Worth knowing before you rewrite titles for the fourth time.
Top comments (0)