DEV Community

toolfreebie
toolfreebie

Posted on

Googlebot read my sitemap and fetched zero articles. Bingbot fetched 64.

My site has one page in Google's index and 79 in the sitemap. I assumed the content was being crawled and judged. The access log says it was never crawled at all.

Two days of raw log, every Googlebot request:

66.249.72.1 - [04/Sep/2026:19:57:09] "GET /robots.txt HTTP/1.1" 200 4951
66.249.72.8 - [04/Sep/2026:19:57:10] "GET /post-sitemap.xml HTTP/1.1" 200 2793
66.249.72.2 - [04/Sep/2026:23:04:52] "GET /category-sitemap.xml HTTP/1.1" 200 5267
Enter fullscreen mode Exit fullscreen mode

Three requests. It read robots.txt, read both sitemaps, and fetched zero articles.

Same window, same server, other crawlers:

YandexBot      154 requests
bingbot        139 requests   (64 distinct article URLs)
ClaudeBot        7 requests
Googlebot        3 requests   (0 article URLs)
PerplexityBot    1 request
GPTBot           0 requests
Enter fullscreen mode Exit fullscreen mode

Bingbot pulled 64 different articles plus cover images and the WP REST endpoints. Same URLs, same robots.txt, same response times.

That rules out most of what I would otherwise have spent a week on. Not robots.txt, Googlebot fetched it and it disallows nothing relevant. Not the sitemap, it fetched that too and got a 200 with 79 URLs in it. Not the content, it has never seen the content. Not server errors, everything returned 200. It knows exactly which URLs exist and is choosing not to spend requests on them.

If you have shell access, this is the whole check:

grep -i googlebot access.log   | grep -oE '"GET [^ ]+' | sort | uniq -c | sort -rn
Enter fullscreen mode Exit fullscreen mode

If the only lines that come back are robots.txt and your sitemaps, no amount of on-page work will move anything, because the crawler never reaches the pages you are editing. Search Console will not tell you this either — it reports "URL is unknown to Google", which reads like a submission problem rather than a budget decision.

Worth knowing before you rewrite titles for the fourth time.

Top comments (0)