I run two small Next.js sites on the same host. Google indexes neither of them well, and this week I read the crawl stats for both instead of guessing. They fail in opposite ways.
Site one: a calculator site, robots.txt allows everything
Search Console, crawl stats, last 90 days:
total crawl requests 988
average response time 173 ms
by response 200: 97 % 404: 2 %
by purpose refresh: 100 %
by file type JavaScript 50 % HTML 37 % CSS 7 % image 4 %
by Googlebot type page resource load 57 % smartphone 34 %
And the Pages report:
indexed 32
discovered - currently not indexed 176
crawled - currently not indexed 40
Read together: Google made 988 requests in three months, and more than half of them were not pages. They were the scripts and stylesheets it needs to render pages it already knows. About 37 % were HTML, which is roughly 365 page fetches in 90 days, four a day, for a sitemap of 263 URLs. Every request was a refresh of something known. None went to discovering anything new, and 176 URLs sit in "discovered" with no crawl date at all.
Nothing is broken. The server answers in 173 ms, nothing is blocked, the sitemap is read. Google gives this site a very small allowance and spends more than half of it on JavaScript.
Site two: a resume builder, robots.txt blocked /_next/
Same host, same framework, and one line that had been in robots.txt since May: Disallow: /_next/. It was there to keep "infrastructure" out of search. /_next/static is where every stylesheet and script lives.
total crawl requests about 1,680
by file type HTML 80 % JavaScript 2 % image 4 %
by Googlebot type smartphone 57 % page resource load 4 %
crawled - currently not indexed 411
Here Google fetched far more HTML, because it was not allowed to fetch anything else. It read 411 pages and declined to index them. I cannot prove the blocked resources are the reason: this site also had a week in September when a redirect loop hit every crawler, and it has few links. But a page whose CSS and JS cannot be loaded is not the page a visitor sees, and Google's own guidance is not to block them. I removed the line today.
What I take from the two together
- The same complaint, "Google does not index my site", had two different shapes: never crawled on one site, crawled and declined on the other. The Pages report tells you which one you have, and the fix is different.
- Blocking
/_next/does not save crawl budget in any useful way. It trades rendering for raw HTML fetches. - Allowing it has a cost on a site with little authority: the rendering requests come out of the same small allowance. My guess, and it is only a guess, is that a deploy makes it worse, because the chunk file names change and Google has to fetch them again.
- Crawl stats sit under Settings, not in the main reports, and I had never opened them. Ten minutes there replaced weeks of theories.
What is left for site one is the unglamorous part: links from outside, since the allowance seems to follow how much the rest of the web points at you.
The calculator site is AllSmartCalculators; a typical page is the PPF calculator. Written by an AI assistant from my own Search Console data.
Top comments (1)
Your first site reminds me of our unit converter, where every unit pair has its own page. Bing indexed nearly all of them, while Google left many uncrawled and others stuck in "Crawled, currently not indexed".
Interestingly, most of our Google impressions came from longer explainer articles rather than the individual converter pages.
We also had very few backlinks, and much of our traffic came from Bing and Yandex.
Have you checked how Bing Webmaster Tools sees your first site? I'd be curious whether Bing is picking up those calculator pages that Google hasn't crawled.