Your redesign will 404 the pages Google already ranks. This script catches it before launch

A new site goes live, it looks better than the old one, and a few weeks later organic traffic starts sliding. Nobody touched the content. The old URLs just stopped working.

It's the most common way a redesign loses what the old site earned, and it's almost never a design problem. It's a URL that was never mapped, a redirect that chains through three hops, a 302 someone meant as a 301, or a noindex tag that came over from staging on launch day.

We care about this more than most, partly because three of the six criticisms in our own Clutch reviews are about details that went unchecked. So we stopped trusting a spreadsheet of redirects and started testing them with a script.

What can go wrong with an old URL

Every URL in your current sitemap has one of a few futures on the new site:

What happens	What it costs you
200 at the same path	Nothing. Best case.
One 301/308 to the new path	A little crawl time, rankings usually follow
A chain: 301 to 301 to 200	Slower crawling, and each hop is one more thing to break later
302/307 to the new path	You're telling search engines the move is temporary
404 or 410	The page and its links are gone
200, but noindex	Live for people, invisible to search
200, but the canonical points somewhere else	You asked Google to rank a different page

The first two are fine. Everything below them is something you want to find on staging, not in Search Console a month later.

The script

It reads the old site's sitemap (including sitemap indexes), requests every path on the new site, follows redirects by hand so it can see each hop, and checks the page it lands on.

# check_redirects.py
import re
import sys
import xml.etree.ElementTree as ET
from urllib.parse import urljoin, urlsplit

import requests

OLD_SITEMAP = sys.argv[1]   # e.g. https://example.com/sitemap.xml
NEW_BASE = sys.argv[2]      # e.g. https://staging.example.com
MAX_HOPS = 5
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

session = requests.Session()
session.headers["User-Agent"] = "redirect-check/1.0"


def old_paths(sitemap_url):
    root = ET.fromstring(session.get(sitemap_url, timeout=20).content)
    for loc in root.iterfind(".//sm:loc", NS):
        url = loc.text.strip()
        if url.endswith(".xml"):          # sitemap index: recurse
            yield from old_paths(url)
        else:
            parts = urlsplit(url)
            yield parts.path + (f"?{parts.query}" if parts.query else "")


def follow(url):
    hops = []
    for _ in range(MAX_HOPS):
        r = session.get(url, allow_redirects=False, timeout=20)
        hops.append((r.status_code, url))
        if r.status_code in (301, 302, 303, 307, 308):
            url = urljoin(url, r.headers["Location"])
        else:
            return hops, r
    return hops, None


def problems(hops, final):
    found = []
    redirects = hops[:-1]
    if final is None:
        return ["redirect loop or more than 5 hops"]
    if final.status_code != 200:
        found.append(f"ends in {final.status_code}")
    if len(redirects) > 1:
        found.append(f"chain of {len(redirects)} redirects")
    if any(code in (302, 307) for code, _ in redirects):
        found.append("temporary redirect (302/307), use 301/308")
    html = final.text if "html" in final.headers.get("Content-Type", "") else ""
    if re.search(r'<meta[^>]+name=["\']robots["\'][^>]+noindex', html, re.I) \
            or "noindex" in final.headers.get("X-Robots-Tag", ""):
        found.append("noindex")
    m = re.search(r'<link[^>]+rel=["\']canonical["\'][^>]+href=["\']([^"\']+)', html, re.I)
    # compare paths only, so a staging page canonicalised to production passes
    canonical_path = urlsplit(urljoin(final.url, m.group(1))).path if m else None
    if m and canonical_path.rstrip("/") != urlsplit(final.url).path.rstrip("/"):
        found.append(f"canonical points elsewhere: {m.group(1)}")
    return found


failures = 0
paths = sorted(set(old_paths(OLD_SITEMAP)))
for path in paths:
    hops, final = follow(NEW_BASE.rstrip("/") + path)
    issues = problems(hops, final)
    if issues:
        failures += 1
        print(f"FAIL {path}")
        for code, url in hops:
            print(f"     {code} {url}")
        for issue in issues:
            print(f"     -> {issue}")

print(f"\n{len(paths) - failures}/{len(paths)} old URLs land on a live, indexable page")
sys.exit(1 if failures else 0)

Run it against staging before launch:

pip install requests
python check_redirects.py https://example.com/sitemap.xml https://staging.example.com

What the output looks like

Here's a run against a small test site with one of each mistake in it:

FAIL /blog/a
     301 https://staging.example.com/blog/a
     301 https://staging.example.com/blog/a/
     200 https://staging.example.com/insights/a
     -> chain of 2 redirects
FAIL /contact
     200 https://staging.example.com/contact
     -> noindex
FAIL /loop
     301 https://staging.example.com/loop
     301 https://staging.example.com/loop2
     301 https://staging.example.com/loop
     301 https://staging.example.com/loop2
     301 https://staging.example.com/loop
     -> redirect loop or more than 5 hops
FAIL /pricing
     302 https://staging.example.com/pricing
     200 https://staging.example.com/plans
     -> temporary redirect (302/307), use 301/308
FAIL /team
     404 https://staging.example.com/team
     -> ends in 404

2/7 old URLs land on a live, indexable page

The /blog/a chain is the sneaky one. It's usually two rules that each look right on their own: one adds a trailing slash, the other moves /blog/ to /insights/. Fix it by pointing the old URL straight at the final one.

Three things that will trip you up

Staging is usually noindex on purpose. If your staging site blocks indexing, every page will report noindex. That's expected there. Run the script again against production the hour you launch: the same check then catches the tag if it survived the move, which happens more often than anyone admits.

Canonicals are compared by path, not domain. Staging pages often already point their canonical at the production domain. The script compares only the path, so staging.example.com/plans with a canonical of example.com/plans passes.

Your sitemap isn't every URL that matters. Pages with backlinks don't always appear in the sitemap. Before launch, export the top linked pages from your SEO tool, or the most visited ones from analytics, and run them through the same follow() function.

Put it in CI

The script exits with 1 if anything fails, so it works as a deploy gate as is:

# .github/workflows/redirects.yml
name: redirect check
on: [deployment_status]
jobs:
  check:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install requests
      - run: python check_redirects.py https://example.com/sitemap.xml ${{ github.event.deployment_status.environment_url }}

Before launch, save the old sitemap and publish it somewhere stable, for example /old-sitemap.xml on the new site, then point the workflow at that. Otherwise the "old" sitemap quietly becomes the new one after launch and the check passes forever.

What it doesn't check

This catches the URL side of a redesign. It won't tell you that the new page dropped the paragraph that answered the search query, that your analytics tag didn't come across, or that a form stopped recording where the lead came from. Those are the other ways redesigns lose enquiries. We put all twelve steps in order, from the URL map to the weeks after launch, in our website redesign checklist.

Over to you: run it against your own staging site before your next launch. What's the longest redirect chain you've found in the wild? Post it in the comments, along with whatever caused it.

From the team at Redlio Labs, where we design, build and grow websites and software for mid-size and large companies.