What Is a Technical SEO Audit? Reproduce a noindex Finding Before Opening a Ticket

An audit line that says "47 pages have noindex" is a claim, not a bug report. Ask what is a technical SEO audit and most people describe the report. For the developer who gets the tickets, it is the part you can reproduce in five minutes, on pages you picked, with output you can paste into the issue.

This post builds that check. It samples URLs by template, reads the response header and the HTML for noindex, compares the canonical, and flags URLs that Googlebot is not allowed to crawl, so it cannot read their noindex, even though this script can still request them directly.

Treat every finding as a hypothesis

Say a plumbing company receives an audit export with 4,000 rows and no assigned owner. Somewhere in there is the real problem: the shared service template prints noindex on every service page. Also in there: account pages that carry noindex on purpose.

"Remove all noindex tags" would fix the first and break the second. So before anyone opens a ticket, you want three things: which templates are affected, which samples are supposed to stay out of search, and a command that proves the state before and after.

Sample by template, not by error count

Pick two or three URLs per template and write down what each one should do. That intent column is the part an export cannot give you.

template,expect,url
service,index,http://localhost:8001/services/drain-repair
service,index,http://localhost:8001/services/water-heaters
blog,index,http://localhost:8001/blog/winter-pipes
account,noindex,http://localhost:8001/account/settings
account,noindex,http://localhost:8001/account/orders

On a real site you swap in your own URLs. Here they point at a local fixture so you can reproduce every line below.

The fixture

Three templates. The service template has the bug. The account routes send X-Robots-Tag: noindex on purpose, and robots.txt blocks one of them. They are test routes, not authenticated pages. On a real site, protect account data with a login, not with noindex.

# site.py: local fixture with three templates.
# FIXED=1 serves the corrected service template, ROBOTS_FIXED=1 drops the Disallow line.
import os
from http.server import BaseHTTPRequestHandler, HTTPServer

FIXED = os.environ.get("FIXED") == "1"
ROBOTS_FIXED = os.environ.get("ROBOTS_FIXED") == "1"
BASE = "http://localhost:8001"

def page(path, robots=None):
    meta = f'<meta name="robots" content="{robots}">' if robots else ""
    return (f'<html><head><title>{path}</title>{meta}'
            f'<link rel="canonical" href="{BASE}{path}"></head><body><h1>{path}</h1></body></html>')

def route(path):
    if path == "/robots.txt":
        rules = "" if ROBOTS_FIXED else "Disallow: /account/orders\n"
        return 200, {"Content-Type": "text/plain"}, "User-agent: *\n" + rules
    if path.startswith("/services/"):   # shared service template
        return 200, {}, page(path, None if FIXED else "noindex, follow")
    if path.startswith("/blog/"):       # blog template, never had the bug
        return 200, {}, page(path)
    if path.startswith("/account/"):    # non-indexable account routes (no auth in this fixture)
        return 200, {"X-Robots-Tag": "noindex"}, page(path)
    return 404, {}, "<h1>Not found</h1>"

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        status, headers, body = route(self.path)
        self.send_response(status)
        self.send_header("Content-Type", headers.pop("Content-Type", "text/html; charset=utf-8"))
        for k, v in headers.items():
            self.send_header(k, v)
        self.end_headers()
        self.wfile.write(body.encode())
    def log_message(self, *args):
        pass

HTTPServer(("127.0.0.1", 8001), Handler).serve_forever()

The check

Standard library only. It does a real GET (not HEAD), reads X-Robots-Tag, parses meta robots and rel="canonical" with html.parser instead of regex, and asks Python's robots.txt parser whether the URL may be crawled.

# repro.py samples.csv : reproduce noindex and canonical findings per template
import csv, sys, urllib.request, urllib.robotparser
from html.parser import HTMLParser
from urllib.parse import urljoin, urlparse

class Head(HTMLParser):
    def __init__(self):
        super().__init__()
        self.robots, self.canonical = [], []
    def handle_starttag(self, tag, attrs):
        a = {k.lower(): (v or "") for k, v in attrs}
        if tag == "meta" and a.get("name", "").lower() == "robots":
            self.robots.append(a.get("content", "").lower())
        if tag == "link" and "canonical" in a.get("rel", "").lower().split():
            self.canonical.append(a.get("href", ""))

def check(url):
    rp = urllib.robotparser.RobotFileParser(urljoin(url, "/robots.txt"))
    rp.read()
    try:
        resp = urllib.request.urlopen(url)          # a real GET, not HEAD
    except urllib.error.HTTPError as e:
        resp = e
    body = resp.read().decode("utf-8", "replace")
    p = Head(); p.feed(body)
    header = (resp.headers.get("X-Robots-Tag") or "").lower()
    noindex = "noindex" in header or any("noindex" in r for r in p.robots)
    return {
        "status": resp.status,
        "crawlable": rp.can_fetch("*", url),
        "noindex": noindex,
        "where": "header" if "noindex" in header else ("meta" if noindex else "-"),
        "canonical_ok": p.canonical == [url],
    }

failures = 0
with open(sys.argv[1]) as f:
    for row in csv.DictReader(f):
        r = check(row["url"])
        problems = []
        if r["status"] != 200:
            problems.append(f"status {r['status']}")
        if row["expect"] == "index" and r["noindex"]:
            problems.append(f"noindex in {r['where']}")
        if row["expect"] == "noindex" and not r["noindex"]:
            problems.append("noindex missing")
        if r["noindex"] and not r["crawlable"]:
            problems.append("robots.txt blocks crawling; Googlebot cannot read the noindex")
        if not r["canonical_ok"]:
            problems.append("canonical is not a single self reference")
        failures += bool(problems)
        print(f"{row['template']:<8} {row['expect']:<8} {urlparse(row['url']).path:<26} {'FAIL ' + '; '.join(problems) if problems else 'ok'}")
print(f"\n{failures} failing sample(s)")
sys.exit(1 if failures else 0)

Environment: Linux, Python 3.13.17, standard library only. Start the fixture in one terminal with python3 site.py, then run python3 repro.py samples.csv from a second terminal. The output below is copied from my runs. The script exits with code 1 when any sample fails and 0 when all pass, so CI can gate on it.

Run 1: reproduce the finding

service  index    /services/drain-repair     FAIL noindex in meta
service  index    /services/water-heaters    FAIL noindex in meta
blog     index    /blog/winter-pipes         ok
account  noindex  /account/settings          ok
account  noindex  /account/orders            FAIL robots.txt blocks crawling; Googlebot cannot read the noindex

3 failing sample(s)

Read it by template. Both service samples fail the same way, the blog sample passes, and the account settings page does exactly what it should. That pattern points at the service template, which is a much better ticket than "47 pages have noindex." It is still a hypothesis about root cause until someone finds the line in the template that prints the tag.

The last row is a separate finding. Google's robots meta tag documentation says that if a page is disallowed in robots.txt, its indexing and serving rules will not be found and are ignored. Google's canonicalization guide adds that Google may still index a disallowed URL without its content, and that robots.txt should not be used for canonicalization. So a page that relies on noindex has to stay crawlable for that directive to work. Truly private pages belong behind a login.

Run 2: fix the template only

Stop the fixture with Ctrl+C (the port is still taken otherwise), start it again with FIXED=1 python3 site.py, and rerun the check:

service  index    /services/drain-repair     ok
service  index    /services/water-heaters    ok
blog     index    /blog/winter-pipes         ok
account  noindex  /account/settings          ok
account  noindex  /account/orders            FAIL robots.txt blocks crawling; Googlebot cannot read the noindex

1 failing sample(s)

The template ticket is done. The robots.txt finding is still open, and it belongs to whoever owns robots.txt, which is often not the template developer. Two findings, two tickets, two owners.

Run 3: fix robots.txt too

Ctrl+C again, then FIXED=1 ROBOTS_FIXED=1 python3 site.py and one more run:

service  index    /services/drain-repair     ok
service  index    /services/water-heaters    ok
blog     index    /blog/winter-pipes         ok
account  noindex  /account/settings          ok
account  noindex  /account/orders            ok

0 failing sample(s)

Write the acceptance test into the ticket

• Symptom: service pages carry noindex, follow in meta robots
• Representative URLs: /services/drain-repair, /services/water-heaters
• Affected template: service page
• Must not change: /account/* keeps X-Robots-Tag: noindex
• Owner: template developer
• Acceptance: repro.py samples.csv reports no service failures and both account rows still show noindex

Keep samples.csv in the repo and run the script in CI after each deploy. It becomes a regression check for exactly the thing the audit flagged.

Limits

• urllib.robotparser is a simple parser. It does not match every rule Google supports, such as wildcards, so confirm edge cases with Search Console's robots.txt report. The check also asks about the * group; if your robots.txt has a separate Googlebot group, pass "Googlebot" to can_fetch instead.
• urlopen follows redirects, so the status you see is the final one. Add a redirect check if the finding involves moved URLs.
• The parser reads meta tags anywhere in the document, and the canonical check wants one exact, absolute self reference.
• It only sees initial HTML. Tags added by JavaScript need a rendered check.
• A clean run says the sampled pages send what you intended. It does not show they are indexed, and a scan of any size only flags candidates. It does not establish root cause or business impact.

Disclosure: I work with GuaranteedSEOServices.com, which published the original guide this post is based on.

The original guide on what is a technical SEO audit covers scope, datasets and what a useful deliverable looks like. What would you add to samples.csv to catch a template edge case: a paginated page, a filtered URL, or something stranger?