DEV Community

seotto.com
seotto.com

Posted on

How to catch SEO regressions before they ship: diff titles, canonicals and robots tags between production and a preview deploy (Python, for CI)

Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com

The SEO incidents that hurt most are rarely strategy mistakes. They're one-line template changes:

  • A noindex that was added on staging and merged with everything else.
  • A canonical URL built from an environment variable that's wrong in production.
  • A layout refactor that drops the <h1> or the meta description from every product page.
  • Structured data that disappears because a component stopped rendering server-side.

Each of these is easy to spot if you compare the page before and after the change. So let's compare them automatically, on every pull request, against the preview deploy most hosting platforms already give you.

What the script compares

For each path, it fetches production and preview and extracts:

  • HTTP status, and the final URL after redirects
  • <title> and meta description
  • rel="canonical" (normalised so the preview host doesn't count as a difference)
  • meta robots and the X-Robots-Tag header
  • the first <h1>
  • the number of hreflang alternates
  • the @type values in JSON-LD blocks

Then it reports every difference, and exits non-zero for the dangerous ones.

The script

#!/usr/bin/env python3
"""Compare SEO-relevant tags between production and a preview deploy."""
import json, re, sys
from html.parser import HTMLParser
from urllib.parse import urlsplit
import requests

PROD, PREVIEW = sys.argv[1].rstrip("/"), sys.argv[2].rstrip("/")
PATHS = [l.strip() for l in open(sys.argv[3]) if l.strip()]
FATAL = {"status", "robots", "x_robots", "canonical"}

class Head(HTMLParser):
    def __init__(self):
        super().__init__(); self.d = {"hreflang": 0, "jsonld": []}
        self._in = None; self._buf = ""
    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "meta" and a.get("name", "").lower() == "description":
            self.d["description"] = a.get("content", "").strip()
        if tag == "meta" and a.get("name", "").lower() == "robots":
            self.d["robots"] = a.get("content", "").replace(" ", "").lower()
        if tag == "link" and "canonical" in a.get("rel", "").lower().split():
            self.d["canonical"] = urlsplit(a.get("href", "")).path or "/"
        if tag == "link" and a.get("hreflang"):
            self.d["hreflang"] += 1
        if tag in ("title", "h1") and tag not in self.d or (tag == "script" and a.get("type") == "application/ld+json"):
            self._in, self._buf = tag, ""
    def handle_data(self, data):
        if self._in: self._buf += data
    def handle_endtag(self, tag):
        if tag == self._in:
            if tag == "script":
                try:
                    j = json.loads(self._buf); items = j if isinstance(j, list) else [j]
                    self.d["jsonld"] += sorted(str(i.get("@type")) for i in items if isinstance(i, dict))
                except ValueError:
                    self.d["jsonld"].append("INVALID")
            else:
                self.d[tag] = re.sub(r"\s+", " ", self._buf).strip()
            self._in = None

def snapshot(base, path):
    r = requests.get(base + path, timeout=30, headers={"User-Agent": "seo-diff/1.0"})
    p = Head(); p.feed(r.text)
    d = p.d
    d["status"] = r.status_code
    d["final_path"] = urlsplit(r.url).path
    d["x_robots"] = r.headers.get("X-Robots-Tag", "").replace(" ", "").lower()
    d["jsonld"] = sorted(d["jsonld"])
    return d

fatal = 0
for path in PATHS:
    a, b = snapshot(PROD, path), snapshot(PREVIEW, path)
    for key in sorted(set(a) | set(b)):
        if a.get(key) != b.get(key):
            level = "FATAL" if key in FATAL else "warn "
            fatal += level == "FATAL"
            print(f"{level} {path}  {key}: {a.get(key)!r} -> {b.get(key)!r}")
print(f"{len(PATHS)} paths compared, {fatal} fatal difference(s)")
sys.exit(1 if fatal else 0)
Enter fullscreen mode Exit fullscreen mode

Usage, with a file of representative paths:

/
/pricing/
/blog/
/blog/some-popular-post/
/docs/getting-started/
/products/example-product/
Enter fullscreen mode Exit fullscreen mode
python3 seo_diff.py https://www.example.com https://pr-142.preview.example.com paths.txt
Enter fullscreen mode Exit fullscreen mode
FATAL /products/example-product/  robots: None -> 'noindex,nofollow'
warn  /blog/some-popular-post/  description: 'How we cut build times...' -> None
2 paths compared, 1 fatal difference(s)
Enter fullscreen mode Exit fullscreen mode

Why it's built this way

Compare two live environments, not a snapshot. A stored baseline goes stale every time content changes. Comparing production with the preview for the same commit isolates what this change did.

Canonicals compared by path. The preview's canonical will (and should) differ in host if it's built from the request. Comparing the path catches the real bug, a canonical pointing at a different page, without flagging every preview. Add a separate check that production canonicals use the production host.

Fatal vs warning. A changed title might be intentional. A new noindex, a status code change or a different canonical almost never is on a pull request that wasn't about SEO. Make the dangerous ones fail the build and the others visible.

A short list of representative paths. One URL per template type (home, listing, article, product, docs page, landing page) catches most template regressions in seconds. You don't need to crawl the whole site on every PR.

A CI diff catches what a pull request changes; production can still drift after a CMS edit or a CDN rule. seotto.com keeps checking the live site for indexing blocks and SEO regressions after deploy, so the changes that never went through a pull request get caught too.

Things to watch out for

  • Preview deploys are often noindexed on purpose, with an X-Robots-Tag: noindex header from the platform. You'll see this as a fatal difference on every path. Either strip that one header from the comparison when the host is a preview, or have the platform skip the header for requests with a specific user agent.
  • Personalised or A/B-tested pages can differ between two fetches of the same environment. Exclude them or pin the variant with a cookie.
  • Client-rendered tags won't be seen. The script reads server HTML, which is deliberate: it's what crawlers get first. If your titles only exist after JavaScript runs, that's a finding in itself.

Wiring it into CI

Run it as a job after the preview deploy finishes, passing the preview URL from your platform's deployment event. Post the output as a PR comment when there are warnings, and fail the check on fatal differences. It takes a few seconds, and it's the cheapest SEO insurance I know.

Related reading

What's the worst SEO regression that made it to production on your site? I'd bet a good share of them would have shown up in a diff like this.

Top comments (0)