Problem
My personal website pages built with NextJs v14.x.x hosted by Vercel were not indexed at all for ages somewhere down the line ... very shocking...
Details in Google Search Console
When you visit Sitemap page, it says Couldn't fetch.
When I visited URL Inspection page, it says Site-wide availability issues.
Investigating Process
However I could see the content https:mypage/sitemap.xml on browser.
curl -I https:mypage/sitemap.xml shows 200 response status properly.
Then I checked more details in Site-wide availability issues on https://support.google.com/webmasters/answer/9012289#will_i_be_indexed&zippy=%2Csite-wide-availability-issues
and then found this one:
Robots.txt unreachable: Google won't crawl a website if the robots.txt file is present but not reachable. You can check your robots.txt availability in the Crawl Stats report.
Then I realised thathttps:mypage/robots.txt returns 500 status despite the fact that I didn't prepare the robots.txt in my NextJS directory.
Running production mode in NextJS hides the error details while dev mode shows you the details.
Voila, my poor error handling returns 500 status even though /robots.txt does not exist which in turn confused Google Indexer and put it the label of Not Indexable Page.
The case is closed.
Top comments (1)
Search Console is usually the first place to check, but its data can lag by a few days, so a page that is live may not show as indexed right away. I would check the URL inspection tool for the exact URL, look for a noindex tag or a robots.txt block, and make sure the sitemap lists the current URL. If those are clean, it is often just a waiting game after a new post, rather than a technical block.