How to find and fix lost pages after a website migration

Disclosure: I build and sell two paid Apify Actors that automate parts of this workflow. They're mentioned in step 6 as one option. Everything else in this post works with free tools. This article was drafted with AI assistance; the code and examples were tested.

A migration is the easiest way to lose search traffic you already earned. You redesign the site, change the CMS, move /blog/2019/my-post to /insights/my-post, and forget about a few hundred old URLs. Other sites still link to those URLs, and Google still has them indexed. After launch they return 404, or every one of them redirects to the homepage, and the rankings they held disappear.

The fix is simple to describe and tedious to do:

1. Build a complete list of URLs the old site had.
2. Check what each one returns today.
3. Map every broken URL to its closest live equivalent with a 301.
4. Verify, deploy, and monitor.

This post walks through each step with commands you can run right now.

Step 1: Build the "old URL" inventory from several sources

No single source has every URL, so combine a few:

• Your old XML sitemap. If you still have a backup or a staging copy, export it. It's the cleanest list, but it only has what the CMS chose to include.
• Google Search Console. Indexing → Pages and the Performance report (export the "Pages" tab) show URLs Google actually knew about and that actually got clicks. Those are the ones to protect first.
• Analytics. Export landing pages from the last 12–16 months.
• Backlink data. Any backlink tool (or GSC's Links report) shows "top linked pages". A linked URL that now 404s is wasting link equity.
• The Internet Archive's Wayback Machine. This is the one people forget. It often has URLs that never made it into a sitemap: old campaign pages, docs from a previous product version, tag archives.

Pulling URLs from the Wayback Machine (free)

The Internet Archive has a public CDX API that lists every capture for a host. This query returns unique HTML URLs that were archived with status 200:

curl -s "https://web.archive.org/cdx/search/cdx?url=example.com&matchType=host&fl=original,timestamp&collapse=urlkey&filter=statuscode:200&filter=mimetype:text/html" > wayback.txt

Useful parameters:

• matchType=host means exactly that host. domain includes subdomains. prefix with url=example.com/blog/ limits it to one section.
• collapse=urlkey removes duplicate captures of the same URL.
• limit= and the showResumeKey/resumeKey parameters let you page through big sites.

Two things to know before you rely on it:

1. It's slow and flaky. Large hosts can take tens of seconds per page, and you will see 503/504 or "temporarily offline" responses. Retry with backoff instead of giving up.
2. It's noisy. You'll get ?utm_source= and ?fbclid= variants, wp-json endpoints, login links, and so on. Normalize before you check anything:

from urllib.parse import urlsplit, urlunsplit

def normalize(url: str) -> str:
    parts = urlsplit(url.strip())
    host = parts.netloc.lower().split(":")[0]   # drop :80
    path = parts.path or "/"
    return urlunsplit(("https", host, path, "", ""))  # drop query + fragment

SKIP = ("/wp-json", "/cdn-cgi", "/wp-login", ".css", ".js", ".png", ".jpg", ".woff")

with open("wayback.txt") as f:
    urls = {normalize(line.split()[0]) for line in f if line.strip()}
urls = sorted(u for u in urls if not any(s in u for s in SKIP))

Dropping all query strings is a blunt choice. If your site uses ?id=123 as real page identity, keep those parameters instead.

Merge this list with your sitemap, GSC, analytics, and backlink exports, and de-duplicate. That's your inventory.

Step 2: Check what every old URL returns today

For each URL, you want the status code, the final URL after redirects, and the number of hops. Be polite: low concurrency, a real User-Agent, and timeouts.

import asyncio, httpx, csv

async def check(client, url):
    try:
        r = await client.get(url, follow_redirects=True, timeout=20)
        return url, r.status_code, str(r.url), len(r.history)
    except httpx.HTTPError as e:
        return url, None, None, type(e).__name__

async def main(urls):
    limits = httpx.Limits(max_connections=4)
    headers = {"User-Agent": "migration-audit (contact: [e-mail])"}
    async with httpx.AsyncClient(limits=limits, headers=headers) as client:
        results = await asyncio.gather(*(check(client, u) for u in urls))
    with open("live_status.csv", "w", newline="") as f:
        csv.writer(f).writerows([("url", "status", "final_url", "hops"), *results])

asyncio.run(main(urls))

Then put each URL into one of these buckets:

Bucket	What it means	Action
200, same URL	Fine	Nothing
301 → relevant page	Fine	Collapse chains longer than one hop
404 / 410	Lost	Redirect or deliberately retire
Redirects to /	Soft 404	Replace with a specific redirect
5xx / timeout	Broken or blocked	Investigate server/WAF

The homepage bucket deserves its own mention. "Redirect everything old to the homepage" is a common migration shortcut, and Google generally treats irrelevant redirects to the homepage as soft 404s, so you lose the equity anyway. Detect it by checking whether final_url normalizes to the site root while the original path didn't.

Step 3: Map each lost URL to its best replacement

This is the part that takes the most human judgment. A good redirect target is the page that best satisfies the same intent, not the closest string.

A practical way to speed it up is to let fuzzy matching propose candidates, then review them:

from difflib import SequenceMatcher
from urllib.parse import urlsplit

def path(u): return urlsplit(u).path.rstrip("/").lower()

def best_match(lost_url, live_urls, threshold=0.6):
    scored = [(SequenceMatcher(None, path(lost_url), path(l)).ratio(), l) for l in live_urls]
    score, target = max(scored, default=(0, None))
    return (target, round(score, 2)) if score >= threshold else (None, round(score, 2))

live_urls should come from your new sitemap or a crawl of the new site, not the old list. Treat every suggestion as a draft:

• High score + same topic → accept.
• Low score → open the archived page (https://web.archive.org/web/*/OLD_URL) to see what it actually was, then pick the target by hand.
• Nothing equivalent exists → a parent category page is usually better than the homepage. If the content is truly gone and has no links or traffic, a 410 is an honest answer.
• High-value page with no equivalent → consider restoring the content. The archived snapshot gives you the old title, headings, and text to start from.

Write the result as a simple CSV: from,to,status.

Step 4: Deploy the redirects

Most platforms take a list in some form:

Nginx

map $request_uri $redirect_target {
    /old-pricing/        /pricing/;
    /blog/2019/my-post/  /insights/my-post/;
}
server {
    if ($redirect_target) { return 301 $redirect_target; }
}

Apache (.htaccess)

Redirect 301 /old-pricing/ /pricing/
Redirect 301 /blog/2019/my-post/ /insights/my-post/

Netlify _redirects

/old-pricing/        /pricing/            301
/blog/2019/my-post/  /insights/my-post/   301

Cloudflare Bulk Redirects and WordPress plugins such as Redirection accept a CSV import.

Rules of thumb: point redirects at the final URL (no chains), use 301 for permanent moves, and keep the map in version control so the next migration doesn't wipe it out.

Step 5: Verify, then make sure the new sitemap is clean

Re-run the Step 2 script against the from column. Every row should now resolve with one hop to the intended to URL and a 200.

Then audit the new sitemap. A migration often ships a sitemap that still lists URLs that redirect, return 404, carry noindex, or have a canonical pointing somewhere else. Each of these sends Google mixed signals and is a common reason for "Crawled – currently not indexed" in Search Console. For each sitemap URL, check:

• status code is 200 without a redirect
• no noindex in <meta name="robots"> or the X-Robots-Tag header
• <link rel="canonical"> points to itself
• it isn't disallowed in robots.txt
• lastmod is a valid date and not in the future

Finally, watch GSC's Pages report and your 404 logs for a few weeks after launch. New 404s with referrers are usually links you missed.

Step 6: Automating it (optional)

Everything above is doable with the scripts in this post and a spreadsheet. If you'd rather not maintain the retries, paging, normalization, and matching yourself, I packaged the workflow into two Apify Actors. Both are paid (pay-per-event), so here's exactly what they do and cost:

• Wayback Machine URL Extractor & Lost Page Finder covers steps 1–3 for the archive source. It pages through the CDX API with retries, filters noise, checks live status, and sorts URLs into categories such as lost, redirects-to-homepage, and unreachable. It suggests a replacement for each lost URL using the same fuzzy path similarity described above, with homepage and a score of 0 when nothing scores 0.6 or higher, and writes a from,to,score,status,category CSV. It can also recover the archived title, meta description, H1, and text. The README puts a typical 1,000 URLs with live checks at about $2.00.
• Sitemap Extractor & SEO Index Audit covers step 5. It finds sitemaps via robots.txt and common paths, follows nested indexes and .xml.gz files, and flags 404s, redirects, noindex, canonical mismatches, robots.txt blocks, and lastmod problems. It outputs an ISSUES CSV. A full audit of 1,000 pages costs about $1.50.

Limitations to know: the redirect suggestions are string similarity, not semantic matching, so you still need to review them. The sitemap auditor doesn't execute JavaScript, so it reads the HTML <head> as served. And the Wayback Machine only knows about pages it happened to capture.

Checklist

• [ ] Old URL inventory from sitemap, GSC, analytics, backlinks, and Wayback CDX
• [ ] Normalized and de-duplicated
• [ ] Live status for every URL, bucketed (lost / soft 404 / chain / error)
• [ ] Redirect map reviewed by a human, no chains, no blanket homepage redirects
• [ ] Deployed and re-verified
• [ ] New sitemap audited: only 200, indexable, self-canonical URLs
• [ ] 404 monitoring for the first few weeks

If you've done a migration that went sideways, I'd like to hear what you missed. That's how this checklist got longer.