DEV Community

Toshiya Matsumoto
Toshiya Matsumoto

Posted on

Page can't be indexed in Google Search Console

Problem

My personal website pages built with NextJs v14.x.x hosted by Vercel were not indexed at all for ages somewhere down the line ... very shocking...

Details in Google Search Console

When you visit Sitemap page, it says Couldn't fetch.
When I visited URL Inspection page, it says Site-wide availability issues.

Investigating Process

However I could see the content https:mypage/sitemap.xml on browser.

curl -I https:mypage/sitemap.xml shows 200 response status properly.

Then I checked more details in Site-wide availability issues on https://support.google.com/webmasters/answer/9012289#will_i_be_indexed&zippy=%2Csite-wide-availability-issues

and then found this one:

Robots.txt unreachable: Google won't crawl a website if the robots.txt file is present but not reachable. You can check your robots.txt availability in the Crawl Stats report.

Then I realised thathttps:mypage/robots.txt returns 500 status despite the fact that I didn't prepare the robots.txt in my NextJS directory.

Running production mode in NextJS hides the error details while dev mode shows you the details.

Voila, my poor error handling returns 500 status even though /robots.txt does not exist which in turn confused Google Indexer and put it the label of Not Indexable Page.

The case is closed.

Image of Timescale

Timescale – the developer's data platform for modern apps, built on PostgreSQL

Timescale Cloud is PostgreSQL optimized for speed, scale, and performance. Over 3 million IoT, AI, crypto, and dev tool apps are powered by Timescale. Try it free today! No credit card required.

Try free

Top comments (0)

Image of Docusign

🛠️ Bring your solution into Docusign. Reach over 1.6M customers.

Docusign is now extensible. Overcome challenges with disconnected products and inaccessible data by bringing your solutions into Docusign and publishing to 1.6M customers in the App Center.

Learn more

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay