A page can be noindexed in three different places, and only one of them shows up in view-source. If you audit indexing by reading the HTML, you will miss the other two.
Google combines every directive it finds and applies the most restrictive one. So a page can look indexable in the browser, carry a hidden X-Robots-Tag: noindex from the server, and never get indexed. The reverse also happens: a page is noindexed in the HTML but blocked in robots.txt, so Google never fetches it and never sees the noindex.
This script checks all three layers for one URL, using only the Python standard library.
What it flags
- A URL blocked in robots.txt, and whether it is also noindexed (the noindex is then never read)
- noindex that exists only in the
X-Robots-Tagheader (common on PDFs, staging rules and CDN or server config) - Header and meta tag that disagree
- A meta robots tag placed in
<body> - Directives aimed at one crawler only, such as
bingbot: noindex - nofollow,
unavailable_afterand snippet restrictions that quietly change how the page appears in results
The script
#!/usr/bin/env python3
"""Check whether a URL's indexing directives agree across robots.txt,
the X-Robots-Tag HTTP header and the meta robots tag.
Usage: python3 robots_directives_check.py https://example.com/page
"""
import sys
import urllib.request
from html.parser import HTMLParser
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser
# Directives that take a value after a colon. Anything else before a colon
# in an X-Robots-Tag value is treated as a user-agent name.
VALUE_DIRECTIVES = {"unavailable_after", "max-snippet", "max-image-preview", "max-video-preview"}
BOT_NAMES = {"robots", "googlebot", "googlebot-news", "bingbot"}
class MetaRobotsParser(HTMLParser):
def __init__(self):
super().__init__()
self.metas = [] # (bot_name, directives, in_head)
self.in_head = True
def handle_starttag(self, tag, attrs):
if tag == "body":
self.in_head = False
if tag != "meta":
return
a = dict(attrs)
name = (a.get("name") or "").lower().strip()
if name in BOT_NAMES:
content = (a.get("content") or "").lower()
directives = {d.strip() for d in content.split(",") if d.strip()}
self.metas.append((name, directives, self.in_head))
def parse_x_robots(values):
"""Turn raw X-Robots-Tag header values into [(bot_name, directives)]."""
parsed = []
for raw in values:
raw = raw.lower().strip()
bot, rest = "robots", raw
head, sep, tail = raw.partition(":")
if sep and head.strip() not in VALUE_DIRECTIVES and " " not in head.strip():
bot, rest = head.strip(), tail
directives = {d.strip() for d in rest.split(",") if d.strip()}
parsed.append((bot, directives))
return parsed
def blocked_by_robots_txt(url, agent="Googlebot"):
parts = urlsplit(url)
rp = RobotFileParser(f"{parts.scheme}://{parts.netloc}/robots.txt")
try:
rp.read()
except Exception:
return None # robots.txt unreachable: unknown
return not rp.can_fetch(agent, url)
def analyse(url, final_url, status, header_values, html):
parser = MetaRobotsParser()
parser.feed(html)
header_rules = parse_x_robots(header_values)
meta_rules = [(n, d) for n, d, _ in parser.metas]
issues = []
def has(rules, word, bots=("robots", "googlebot")):
return any(word in d for n, d in rules if n in bots)
header_noindex = has(header_rules, "noindex") or has(header_rules, "none")
meta_noindex = has(meta_rules, "noindex") or has(meta_rules, "none")
if blocked_by_robots_txt(final_url):
if header_noindex or meta_noindex:
issues.append("Blocked in robots.txt AND noindexed: Google cannot crawl the page, so it never sees the noindex")
else:
issues.append("Blocked in robots.txt (the URL can still be indexed without content if it is linked)")
if header_noindex and not meta_noindex:
issues.append("noindex only in the X-Robots-Tag header: invisible in view-source and in most HTML crawls")
if meta_noindex and not header_noindex:
issues.append("noindex in meta robots")
if has(header_rules, "index") and meta_noindex:
issues.append("Header says index, meta says noindex: Google applies the most restrictive rule")
if any(not in_head for _, _, in_head in parser.metas):
issues.append("Meta robots found in <body>: move it to <head>")
for n, d in header_rules + meta_rules:
if n not in ("robots", "googlebot") and ("noindex" in d or "none" in d):
issues.append(f"Directive targets only '{n}': other crawlers are unaffected")
if has(header_rules + meta_rules, "nofollow") or has(header_rules + meta_rules, "none"):
issues.append("nofollow set: links on this page pass no signals and may not be crawled")
for _, d in header_rules + meta_rules:
for item in d:
if item.startswith("unavailable_after"):
issues.append(f"Page expires from the index: {item}")
if item.replace(" ", "") == "max-snippet:0" or item == "nosnippet":
issues.append("Snippets disabled: the page can rank without a description or AI answer excerpt")
if status >= 400 and (header_noindex or meta_noindex):
issues.append(f"HTTP {status} plus noindex: the status code already removes the page")
return {
"final_url": final_url,
"status": status,
"header_rules": header_rules,
"meta_rules": meta_rules,
"issues": issues,
}
def fetch(url):
req = urllib.request.Request(url, headers={"User-Agent": "robots-directives-check/1.0"})
try:
resp = urllib.request.urlopen(req, timeout=15)
except urllib.error.HTTPError as e:
resp = e
html = resp.read(500_000).decode("utf-8", errors="replace")
return resp.geturl(), resp.status, resp.headers.get_all("X-Robots-Tag") or [], html
if __name__ == "__main__":
if len(sys.argv) != 2:
sys.exit("Usage: python3 robots_directives_check.py <url>")
final_url, status, headers, html = fetch(sys.argv[1])
r = analyse(sys.argv[1], final_url, status, headers, html)
print("Final URL: ", r["final_url"], f"(HTTP {r['status']})")
print("X-Robots-Tag: ", r["header_rules"] or "none")
print("Meta robots: ", r["meta_rules"] or "none")
print("Issues:" if r["issues"] else "No issues found.")
for issue in r["issues"]:
print(" -", issue)
How to read the output
I tested it against a local server with eight cases: a header-only noindex, a header that says index while the meta says noindex, a noindex on a robots.txt-blocked URL, a meta tag in the body, a bingbot-only rule, an unavailable_after date, a 404, and a clean page. Each one produced the expected flag.
The two results that need action first:
- Blocked in robots.txt AND noindexed. Remove the robots.txt block so Google can crawl the page and read the noindex. If the page is already indexed, the block keeps it in the index.
- noindex only in the X-Robots-Tag header. Nobody editing the page template will find it. Look at server config, CDN rules and plugins, and remove it where it is set.
Limits worth knowing
- It reads the raw HTML, so a meta robots tag added or changed by JavaScript will not show up. Compare with the rendered DOM before deciding it is missing.
- It checks one URL at a time. Loop it over a sitemap export for a site-wide view, and keep the request rate polite.
- It requests the page with a generic user agent. Some servers send different headers to Googlebot, so confirm important URLs with the URL Inspection tool in Search Console.
More checks in this series
This is part 9 of my Python checks for technical SEO. Related posts:
- Do your canonical signals agree?, because a noindexed page with a canonical pointing elsewhere sends Google mixed instructions
- Is your sitemap lastmod telling the truth?, since noindexed URLs should not sit in your sitemap
- Which AI crawlers does your robots.txt actually block?, for the robots.txt side of the same question
Where have you found a noindex hiding that nobody on the team knew about?
Top comments (0)