DEV Community

pdbear
pdbear

Posted on

Before you migrate a website, save a list of every old URL (free Wayback Machine script)

#seo #python #webdev #tutorial

Disclosure: I build AppGripes and a small free migration page mentioned at the end. Everything else here works with free tools. This article was drafted with AI assistance; the code was tested on Oct 30, 2026.

The most expensive mistake in a site migration is quiet: old URLs that used to get traffic and backlinks start returning 404 on the new site, and nobody notices for weeks. Search Console only shows you some of them, and only after Google has crawled the broken pages.

The fix is boring and cheap: before you switch, write down every URL the old site ever had, then check each one against the new site and add a 301 for anything that breaks.

The catch is getting that list. Your CMS knows today's pages, not the /blog/2019/old-slug from three redesigns ago that a partner still links to. The Internet Archive does know, and it has a free API for it.

Step 1: list every archived URL with the Wayback CDX API

The CDX server returns one row per capture. With collapse=urlkey you get one row per unique URL, and filters keep only pages that returned 200 HTML (no images, no 404s).

import csv, sys, time, urllib.parse, urllib.request, json

def old_urls(domain, limit=5000):
    """Return unique archived URLs (status 200, HTML) for a domain from the Wayback CDX API."""
    params = {
        "url": f"{domain}/*",
        "output": "json",
        "fl": "original,timestamp,statuscode,mimetype",
        "filter": ["statuscode:200", "mimetype:text/html"],
        "collapse": "urlkey",   # one row per unique URL
        "limit": str(limit),
    }
    q = urllib.parse.urlencode(params, doseq=True)
    req = urllib.request.Request(f"https://web.archive.org/cdx/search/cdx?{q}",
                                 headers={"User-Agent": "old-url-list/1.0"})
    for attempt in range(3):
        try:
            with urllib.request.urlopen(req, timeout=60) as r:
                rows = json.load(r)
            return rows[1:]  # first row is the header
        except Exception as e:
            print(f"attempt {attempt+1} failed: {e}", file=sys.stderr)
            time.sleep(10 * (attempt + 1))
    raise SystemExit("Wayback CDX did not answer; try again in a few minutes.")

def norm(u):
    p = urllib.parse.urlsplit(u)
    path = p.path.rstrip("/") or "/"
    return path + (("?" + p.query) if p.query else "")

if __name__ == "__main__":
    domain = sys.argv[1]
    rows = old_urls(domain)
    seen = {}
    for original, ts, status, mime in rows:
        key = norm(original)
        if key not in seen or ts > seen[key]:
            seen[key] = ts
    with open("old_urls.csv", "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["old_path", "last_archived", "new_path"])
        for path, ts in sorted(seen.items()):
            w.writerow([path, f"{ts[:4]}-{ts[4:6]}-{ts[6:8]}", ""])
    print(f"{len(seen)} unique old paths written to old_urls.csv")
Enter fullscreen mode Exit fullscreen mode

Run it with your domain:

python3 old_urls.py example.com
Enter fullscreen mode Exit fullscreen mode

You get old_urls.csv with old_path, last_archived and an empty new_path column for you to fill in. Trailing slashes are normalized so /about and /about/ count once.

A few notes from testing:

  • The CDX API is free and needs no key, but it can be slow or answer 429 when you hammer it. The script retries with a pause. Don't run it in a tight loop.
  • limit caps the rows. For a big site, raise it or split by path (example.com/blog/*).
  • last_archived tells you how stale a URL is. A page last seen in 2004 probably doesn't need a redirect; one seen last month does.

Step 2: check every old path on the new site

Point the second script at the new site (staging works too, as long as it's reachable):

import csv, sys, urllib.request, urllib.error

def status(url):
    req = urllib.request.Request(url, method="HEAD", headers={"User-Agent": "old-url-check/1.0"})
    try:
        with urllib.request.urlopen(req, timeout=20) as r:
            return r.status, r.geturl()
    except urllib.error.HTTPError as e:
        return e.code, url
    except Exception as e:
        return f"error: {e.__class__.__name__}", url

new_site = sys.argv[1].rstrip("/")   # e.g. https://www.example.com
rows = list(csv.DictReader(open("old_urls.csv")))
missing = 0
with open("redirect_map.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["old_path", "status_on_new_site", "final_url", "new_path"])
    for r in rows:
        code, final = status(new_site + r["old_path"])
        if code != 200:
            missing += 1
        w.writerow([r["old_path"], code, final, r["new_path"]])
print(f"{missing} of {len(rows)} old paths do not return 200 on {new_site}; see redirect_map.csv")
Enter fullscreen mode Exit fullscreen mode 
python3 check_new.py https://staging.example.com
Enter fullscreen mode Exit fullscreen mode

When I ran both scripts on my own small domain, the archive had 4 unique HTML paths going back to 2003, and 3 of them return 404 today:

old_path,status_on_new_site,final_url,new_path
/,200,https://pdbear.com/,
/forum,404,https://pdbear.com/forum,
/forum/index.html,404,https://pdbear.com/forum/index.html,
/forum/messages/1.html,404,https://pdbear.com/forum/messages/1.html,
Enter fullscreen mode Exit fullscreen mode

Those are 20-year-old forum pages from a previous owner, so I'm fine letting them 404. On a real migration the same table is your to-do list.

Step 3: turn the CSV into redirects

Fill new_path for every row that isn't 200 and still matters (recent last_archived, or you know it has backlinks). Then convert it to whatever your host uses. For example, for Netlify or Cloudflare Pages _redirects:

import csv
with open("_redirects", "w") as out:
    for r in csv.DictReader(open("redirect_map.csv")):
        if r["new_path"]:
            out.write(f"{r['old_path']} {r['new_path']} 301\n")
Enter fullscreen mode Exit fullscreen mode

Test a handful by hand after deploy (curl -I https://example.com/old-path should show 301 and the right location).

Limits

  • The archive only knows pages it crawled. Brand-new or blocked (robots.txt, login) pages won't be there; combine this list with your sitemap and analytics top pages.
  • Query strings are kept as separate paths. If your old site used ?page=2 style URLs, decide which ones deserve redirects.
  • HEAD requests are blocked by a few servers; switch to GET if you see 405.

A no-code version

If you'd rather not run Python, I put the same Wayback lookup on a free page: old URL list before migration. Enter a domain and download the CSV. It's a side project, no signup.

Checklist

  • [ ] Old URL list exported before DNS or CMS switch
  • [ ] Every old path checked against the new site
  • [ ] 301s added for paths that still matter
  • [ ] Spot-checked with curl -I after deploy
  • [ ] Sitemap submitted for the new site

Top comments (2)

Collapse
 
bonin_ferizi_7abbbdcf7650 profile image
B0n1 •

Pulling old URLs from Wayback is underrated, since sitemaps miss years of history. After mapping, test that each old URL takes one 301 hop, not a chain. DBLScanner (I built it) has a free redirect checker.

Collapse
 
andrewsmithzeng profile image
pdbear •

Good call, redirect chains are sneaky. One clean 301 per old URL is the way to go. Thanks for reading!