DEV Community

Rahil Khan
Rahil Khan

Posted on

Is your website accidentally hidden from Google? Check noindex and robots.txt in 5 minutes

Your new website might be invisible to Google, and nothing on the page will tell you. The two most common reasons are a "noindex" tag left over from the build and a robots.txt that blocks the whole site. Both are often switched on while a site is being built and then forgotten at launch.

1. Check for a leftover noindex tag

Open your homepage, press Ctrl+U (View Source) and search for noindex. If you find this, Google is being told not to list the page:

<meta name="robots" content="noindex, nofollow">
Enter fullscreen mode Exit fullscreen mode

On WordPress, go to Settings → Reading and make sure "Discourage search engines from indexing this site" is unticked. Many builders and staging tools tick it by default.

2. Check robots.txt

Visit yourdomain.com/robots.txt. This blocks everything:

User-agent: *
Disallow: /
Enter fullscreen mode Exit fullscreen mode

A normal small-business site usually needs only this:

User-agent: *
Disallow:
Sitemap: https://yourdomain.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

3. Check the server header too

Some hosts add the noindex as an HTTP header instead of a tag:

curl -sI https://yourdomain.com | grep -i x-robots-tag
Enter fullscreen mode Exit fullscreen mode

If that prints noindex, fix it in the host or server config.

4. Confirm with Google

In Google Search Console, paste your homepage URL into URL Inspection. It tells you whether the page is indexed and, if not, why. After fixing, click "Request indexing".

Five minutes of checking can be the difference between a site that brings enquiries and one nobody can find.


NEXCOD builds and fixes fast, mobile-first websites for small businesses. nexcod.vercel.app · WhatsApp +1 289 902 5119

Top comments (0)