DEV Community

Kunal
Kunal

Posted on

Where is my noindex coming from? 3 places it hides (and how to check each with curl)

#debugging #html #seo #webdev

Your page loads fine. The HTML looks clean. But Search Console says "Excluded by 'noindex' tag", and you can't find the tag anywhere.

I ran into this so often while building a site monitor that I wrote down where the noindex actually lives. There are 3 places. Here's how to check each one from the terminal.

1. The meta robots tag (the one you can see)

This is the one everybody knows. It sits in the <head> of the page:

<meta name="robots" content="noindex">
Enter fullscreen mode Exit fullscreen mode

Check it:

curl -s https://quietwatch.app/signin | grep -io '<meta[^>]*robots[^>]*>'
Enter fullscreen mode Exit fullscreen mode 
<meta name="robots" content="noindex">
Enter fullscreen mode Exit fullscreen mode

That sign-in page is noindexed on purpose. If one of your real pages shows this, the usual causes are:

  • WordPress: Settings → Reading → "Discourage search engines from indexing this site" left ticked after launch. It adds the tag to every page.
  • An SEO plugin (Yoast, Rank Math) set to noindex for that post type.
  • A staging setting copied to production.

Watch out for <meta name="googlebot" content="noindex"> too. It only targets Google, so a quick search for "robots" can miss it.

2. The X-Robots-Tag header (the one you can't see)

This is the sneaky one. The noindex isn't in the HTML at all. The server, the host or a CDN sends it as an HTTP header:

X-Robots-Tag: noindex
Enter fullscreen mode Exit fullscreen mode

View source won't show it, and neither will your CMS. Check the headers instead:

curl -sI "https://httpbin.org/response-headers?X-Robots-Tag=noindex" | grep -i x-robots
Enter fullscreen mode Exit fullscreen mode 
X-Robots-Tag: noindex
Enter fullscreen mode Exit fullscreen mode

(httpbin is a public test service that returns whatever header you ask for, so it's a safe way to see what a real one looks like.)

Common sources: a rule in .htaccess or the nginx config, a hosting panel setting, a Cloudflare transform rule, or a staging environment that adds the header to everything.

3. A redirect hop (the one nobody checks)

If a URL redirects, every hop has its own headers. A noindex on any hop before the final page can still cause trouble, and most checkers only look at the last response.

Follow the whole chain and print the headers of every hop:

curl -sIL http://quietwatch.app | grep -iE "^(HTTP|location|x-robots)"
Enter fullscreen mode Exit fullscreen mode 
HTTP/1.1 301 Moved Permanently
Location: https://quietwatch.app/
HTTP/1.1 200 OK
Enter fullscreen mode Exit fullscreen mode

That chain is clean: one 301 to HTTPS, then a 200, with no X-Robots-Tag on either response. If you see an X-Robots-Tag line under the 301, that hop is sending a noindex.

Bonus: robots.txt can hide all of this

If robots.txt blocks a URL, Google never fetches the page, so it never sees the noindex either. You end up with a URL that's "Indexed, though blocked by robots.txt", and your noindex does nothing.

curl -s https://example.com/robots.txt
Enter fullscreen mode Exit fullscreen mode

So the order matters: allow crawling first, let Google see the noindex, then block it later if you really want to.

The quick checklist

  1. curl -s URL | grep -io '<meta[^>]*robots[^>]*>' for the meta tag (look for googlebot too)
  2. curl -sI URL | grep -i x-robots for the header
  3. curl -sIL URL to check every redirect hop
  4. Check robots.txt isn't blocking the URL

If you'd rather not use the terminal, I built a free checker that does all four for up to 10 URLs at once and tells you which one is responsible, no signup: quietwatch.app/tools/noindex-checker.

Have you ever found a noindex somewhere weirder than these three? I'd like to hear about it.

Top comments (0)