DEV Community: Andy Zeng

How to track App Store keyword rankings and competitor reviews across countries (free Python + Apple's official endpoints)

Andy Zeng — Mon, 26 Oct 2026 13:37:10 +0000

Disclosure: I build and sell two paid Apify Actors that automate this workflow. They're mentioned in step 6 as one option. Everything else in this post works with free tools and Apple's public endpoints. This article was drafted with AI assistance; the code and examples were tested.

App Store Optimization tools are expensive, and most indie developers only need three numbers: where the app ranks for its main keywords, where it sits in its category chart, and what users (yours and your competitors') complain about. You also want them per country, because an app can be #15 for "habit tracker" in the US and not appear in the UK at all.

Apple exposes enough free, unauthenticated endpoints to get all three:

What	Endpoint	Limits
Keyword search results	`https://itunes.apple.com/search`	up to 200 results, roughly 20 calls/minute
App details	`https://itunes.apple.com/lookup`	IDs or bundle IDs, batched
Top charts	`https://itunes.apple.com/{cc}/rss/{feed}/limit=100/json`	top ~100 per chart
Customer reviews	`https://itunes.apple.com/{cc}/rss/customerreviews/page={n}/id={id}/sortby=mostRecent/json`	10 pages × 50 = 500 per country and sort

Everything below is plain Python 3 with the standard library, no keys, no browser. I tested it on Oct 26, 2026 against Streaks, a paid habit tracker, in the US, UK and Germany.

Step 1: A polite JSON helper

The iTunes API rate-limits aggressively (around 20 requests per minute per IP) and occasionally returns 403 or empty bodies. Retry with a backoff and send a clear User-Agent:

import json, time, urllib.parse, urllib.request

UA = {"User-Agent": "appstore-tracker/1.0 (personal research script)"}

def get_json(url, params=None, retries=3):
    if params:
        url += "?" + urllib.parse.urlencode(params)
    for attempt in range(retries):
        try:
            with urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=30) as r:
                return json.loads(r.read().decode("utf-8") or "{}")
        except Exception:
            if attempt == retries - 1:
                raise
            time.sleep(2 * (attempt + 1))

You need the app's numeric ID. It's the id963034692 part of the App Store URL, or you can look it up by bundle ID: https://itunes.apple.com/lookup?bundleId=com.burbn.instagram.

Step 2: Keyword rank per country

Search the keyword in each storefront and find your app's position in the results:

def keyword_rank(app_id, term, country, limit=200):
    data = get_json("https://itunes.apple.com/search",
                    {"term": term, "country": country, "entity": "software", "limit": limit})
    ids = [str(r["trackId"]) for r in data.get("results", [])]
    rank = ids.index(str(app_id)) + 1 if str(app_id) in ids else None
    top3 = [r["trackName"] for r in data.get("results", [])[:3]]
    return rank, len(ids), top3

Real output from my test run:

[us] 'habit tracker': rank 15 of 193  (top 3: Habit Tracker, Habit Tracker - HabitKit, Finch: Self-Care Pet)
[gb] 'habit tracker': rank None of 195  (top 3: Habit Tracker, Onrise: Habit Tracker & Focus, Habit Tracker - HabitKit)
[de] 'habit tracker': rank 30 of 181  (top 3: Habit Tracker - Tagesplaner, Habit Tracker - HabitKit, Onrise: Habit Tracker & Focus)

That's already a useful finding: the same app is #15 in the US, #30 in Germany and outside the top ~195 in the UK for the same English keyword.

An important caveat. The iTunes Search API ranking is close to but not identical to what a person sees in the App Store app. The app's search is personalized, mixes in Search Ads, and changes more often. Treat these numbers as a consistent, free proxy: good for trends ("we moved from 30 to 18 after the title change") and for comparing countries, not as the exact position a given user sees. Paid ASO tools have the same problem and estimate it in different ways.

Step 3: Top-chart position

Apple still serves the classic RSS charts as JSON. Feeds include topfreeapplications, toppaidapplications, topgrossingapplications (plus topfreeipadapplications and similar), and you can filter by category with /genre={id}. Get the app's primaryGenreId from the Lookup API; Streaks is 6013 (Health & Fitness).

def chart_position(app_id, country, feed="topfreeapplications", genre=None, limit=100):
    g = f"/genre={genre}" if genre else ""
    data = get_json(f"https://itunes.apple.com/{country}/rss/{feed}/limit={limit}{g}/json")
    entries = data.get("feed", {}).get("entry", [])
    for i, e in enumerate(entries, 1):
        if e["id"]["attributes"]["im:id"] == str(app_id):
            return i, len(entries)
    return None, len(entries)

Output:

[us] Top Paid / Health & Fitness: 1 of 97
[gb] Top Paid / Health & Fitness: 1 of 94
[de] Top Paid / Health & Fitness: 2 of 94

The feed returns about 100 entries even if you ask for 200, so "None" means "not in the top ~100", not "unranked".

Step 4: Competitor reviews per country

The customer-reviews RSS feed gives up to 10 pages of 50 reviews per country and sort order (mostRecent or mostHelpful). Because each storefront has its own reviews, scanning several countries gets you far more data than the US store alone.

def reviews(app_id, country, pages=10, sort="mostRecent"):
    out = []
    for page in range(1, pages + 1):
        url = (f"https://itunes.apple.com/{country}/rss/customerreviews/"
               f"page={page}/id={app_id}/sortby={sort}/json")
        try:
            entries = get_json(url).get("feed", {}).get("entry", [])
        except Exception:
            break
        if isinstance(entries, dict):
            entries = [entries]
        rows = [e for e in entries if "im:rating" in e]  # skip the app-info entry
        if not rows:
            break
        for e in rows:
            out.append({
                "country": country,
                "id": e["id"]["label"],
                "rating": int(e["im:rating"]["label"]),
                "version": e.get("im:version", {}).get("label"),
                "date": e.get("updated", {}).get("label"),
                "title": e["title"]["label"],
                "text": e["content"]["label"],
            })
        time.sleep(0.5)
    return out

Two gotchas: when a page has a single entry the feed returns a dict instead of a list, and some pages include an entry describing the app itself (no im:rating). The code handles both. In my test, 10 pages for the US returned the full 500 reviews.

Then summarize per country, and count words in 1–2★ reviews to see what people complain about:

from collections import Counter

STOP = set("the a an and or to of i it is in for on my this that with app was but not "
           "you have be are so just can its it's they when all at me do as get from no "
           "your if what would".split())

def complaint_keywords(rows, n=10):
    words = Counter()
    for r in rows:
        if r["rating"] <= 2:
            for w in (r["title"] + " " + r["text"]).lower().split():
                w = w.strip(".,!?\"'()[]:;")
                if len(w) > 2 and w not in STOP and w.isalpha():
                    words[w] += 1
    return words.most_common(n)

Output for the 150 most recent reviews per country:

[us] avg 3.57, 1-2 star share 29%
[gb] avg 3.87, 1-2 star share 21%
[de] avg 3.53, 1-2 star share 26%
[us] top complaint words: [('watch', 13), ('more', 11), ('update', 10), ('way', 10), ('use', 9), ('apple', 9)]
[gb] top complaint words: [('streaks', 16), ('watch', 16), ('task', 13), ('sync', 11), ('only', 10), ('time', 9)]

Even this crude count points at something concrete: recent negative reviews in both markets mention the Apple Watch and sync. Note that the recent average (3.5–3.9) is much lower than the app's all-time 4.8 rating from the Lookup API. That gap is exactly the signal you want from a competitor: a weakness you could build on.

Run the word count per country, as above. Mixing languages fills the list with stopwords; in German the top "complaint words" were die, nicht and und. Add a stopword list per language (or send the texts to an LLM to cluster) for non-English stores.

Step 5: Run it daily and keep history

A rank is only useful as a trend. Append one row per keyword, country and chart to a CSV:

import csv, datetime

today = datetime.date.today().isoformat()
with open("ranks.csv", "a", newline="") as f:
    w = csv.writer(f)
    for c in ["us", "gb", "de"]:
        for t in ["habit tracker", "to do list"]:
            rank, n, _ = keyword_rank(APP_ID, t, c)
            w.writerow([today, c, t, rank])
            time.sleep(3)  # stay under the rate limit
        pos, _ = chart_position(APP_ID, c, feed="toppaidapplications", genre=6013)
        w.writerow([today, c, "chart:toppaid:6013", pos])

Schedule it with cron (0 6 * * * cd ~/aso && python3 tracker.py) or a free GitHub Actions schedule that commits the CSV. After a few weeks you can see exactly what a title, subtitle or screenshot change did in each country.

Practical tips:

Keep the keyword list small (10–30 terms) and sleep between calls. 3 countries × 20 keywords is 60 requests, about 3–4 minutes at a safe pace.
Track 2–3 competitors with the same function. Their rank changes after a release often tell you which keywords they targeted.
Pull reviews for competitors weekly, not daily. 500 reviews per country per sort is the ceiling, so mostRecent weekly catches almost everything for most apps.
The full tested script (all of the above in one file) is about 100 lines.

Step 6: Automating it (optional)

If you'd rather not maintain scripts, rate-limit handling and storage, I packaged both halves as Apify Actors. They're paid (pay-per-event) and use the same official Apple endpoints, so here's exactly what they do:

App Store Search & Charts Scraper: keyword search (up to 200 apps per keyword with rank), keyword rank tracking for your apps across countries, top charts (free, paid, grossing, new, overall or per category), developer portfolios and app details. Pricing: $0.005 per run plus $0.001 per app record, so 3 charts × 100 apps is about $0.31.
App Store Reviews Scraper: reviews from up to 76 storefronts in one run, both sort orders merged and de-duplicated, star and date filters, plus analytics (rating by version, by country and by month, top complaint and praise keywords). Pricing: $0.005 per run plus $0.15 per 1,000 reviews, plus $0.003 per app for details and analytics.

Both can be scheduled on Apify and let you set a max cost per run. They're subject to the same limits as the free scripts: the Search API ranking is a proxy for in-app search, and Apple caps reviews at about 500 per country and sort order.

Checklist

[ ] App ID found (URL or Lookup by bundle ID)
[ ] 10–30 keywords × your key countries tracked daily, with backoff
[ ] Category chart position from the RSS feed (top ~100)
[ ] 2–3 competitors tracked with the same code
[ ] Reviews pulled per country; complaints counted per language
[ ] History in a CSV so you can tie rank changes to releases and metadata changes

How to bulk-check domain expiry dates and availability with RDAP

Andy Zeng — Mon, 26 Oct 2026 13:21:41 +0000

Disclosure: I build and sell the paid Apify Actor mentioned near the end. Everything before that section uses free, public services. This article was drafted with AI assistance; the code and examples were tested.

Sooner or later, most people who own more than a handful of domains need the same answers: Which of our domains expire soon? Which of these name ideas are actually free? Is that competitor's domain about to lapse? The traditional answer was "run whois in a loop and parse the output", which breaks constantly because every registry formats WHOIS text differently.

There's a better way. RDAP (Registration Data Access Protocol) is the ICANN-mandated successor to WHOIS, and it returns structured JSON. This tutorial shows how to:

Find the right RDAP server for any TLD.
Look up creation date, expiry date and status for a list of domains.
Tell "available" apart from "not in the registry, but you still can't have it".
Add DNS checks (website? email? SPF/DMARC?) for context.

Step 1: Find the RDAP server for each TLD

IANA publishes a bootstrap file that maps every TLD to its RDAP base URL:

curl -s https://data.iana.org/rdap/dns.json | python3 -m json.tool | head -40

For .com it points to https://rdap.verisign.com/com/v1/. A lookup is just a GET:

curl -s https://rdap.verisign.com/com/v1/domain/stackoverflow.com | python3 -m json.tool

The parts you care about:

events: an array with eventAction values such as registration, expiration and last changed
status: EPP statuses like client transfer prohibited or pending delete
entities: the registrar (with its IANA ID) and sometimes registrant info, which is often redacted
nameservers and secureDNS

For "thin" registries like .com, the registry response has the core dates. The registrar's own RDAP server (linked in the links array) sometimes has more detail, such as abuse contacts.

A 404 from the registry RDAP server means the registry has no record of that domain. That's the start of an availability check, not the end of one (see Step 3).

Step 2: A bulk lookup script

This tested script (needs pip install httpx) loads the bootstrap file once, looks up each domain, computes days until expiry, and prints CSV:

import csv, sys, time
from datetime import datetime, timezone
import httpx

UA = {"User-Agent": "domain-expiry-script (contact: you@example.com)"}
client = httpx.Client(headers=UA, timeout=20, follow_redirects=True)

bootstrap = client.get("https://data.iana.org/rdap/dns.json").json()
RDAP = {tld: svc[1][0] for svc in bootstrap["services"] for tld in svc[0]}

def lookup(domain):
    tld = domain.rsplit(".", 1)[-1]
    base = RDAP.get(tld)
    if not base:
        return {"domain": domain, "availability": "unknown", "note": "no RDAP for TLD"}
    r = client.get(f"{base.rstrip('/')}/domain/{domain}")
    if r.status_code == 404:
        return {"domain": domain, "availability": "available?"}
    if r.status_code == 429:
        time.sleep(5)
        return lookup(domain)
    r.raise_for_status()
    data = r.json()
    events = {e["eventAction"]: e["eventDate"] for e in data.get("events", [])}
    expiry = events.get("expiration")
    days_left = None
    if expiry:
        exp = datetime.fromisoformat(expiry.replace("Z", "+00:00"))
        days_left = (exp - datetime.now(timezone.utc)).days
    statuses = data.get("status", [])
    return {
        "domain": domain,
        "availability": "pending-delete" if "pending delete" in statuses else "registered",
        "created": events.get("registration"),
        "expires": expiry,
        "days_until_expiry": days_left,
        "status": "; ".join(statuses),
    }

rows = [lookup(d.strip().lower()) for d in sys.argv[1:]]
w = csv.DictWriter(sys.stdout, fieldnames=["domain", "availability", "created", "expires",
                                           "days_until_expiry", "status", "note"])
w.writeheader(); w.writerows(rows)

python rdap_check.py stackoverflow.com example.org my-new-startup-idea-zzq.com apify.com > domains.csv

Output from a run I did today (dates come straight from the registries):

domain,availability,created,expires,days_until_expiry,status,note
stackoverflow.com,registered,2003-12-26T19:18:07Z,2027-02-02T11:59:59Z,98,client transfer prohibited,
example.org,registered,1995-08-31T04:00:00Z,2027-08-30T04:00:00Z,307,server delete prohibited; server transfer prohibited; server update prohibited,
my-new-startup-idea-zzq.com,available?,,,,,
apify.com,registered,2009-06-02T17:14:10Z,2035-06-02T17:14:10Z,3141,client transfer prohibited,

Things to add before you run it on thousands of domains:

Rate limits are per server. Registries throttle aggressively and answer with HTTP 429. Group domains by RDAP server, keep a small concurrency per server, and back off on 429. The naive retry in the script is fine for a short list, not for a portfolio.
Input cleanup. Real lists contain URLs, emails and subdomains. Extract the registrable domain with the Public Suffix List (for example the tldextract package), so shop.example.co.uk becomes example.co.uk, not co.uk.
Not every TLD has RDAP. Some ccTLDs still only offer port-43 WHOIS. For those you need a WHOIS fallback with per-registry parsing, or at least a DNS check (Step 4) to infer whether the name is delegated.

Step 3: What "available" really means

An RDAP 404 means the registry has no record. You still might not be able to register the name:

Premium and reserved names can be missing from RDAP and still be unbuyable, or priced far above normal.
Pending delete domains (status pending delete) are about to drop. They're registered right now, but they'll become available within days, which is why domain investors watch for them.
Redemption period / auto-renew grace: for many gTLDs, the registry auto-renews at expiry and the registrar decides later whether to delete. So an expiry date that just jumped forward a year doesn't prove the owner actually renewed.

Always confirm at a registrar before you plan around a name.

Step 4: Add DNS for context

Registration data tells you who holds a domain. DNS tells you whether it's used. DNS-over-HTTPS makes this easy without installing anything:

# Is it delegated at all? (NXDOMAIN = Status 3)
curl -s -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=example.com&type=NS'

# Does it receive email?
curl -s -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=example.com&type=MX'

# Email authentication policy
curl -s -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=_dmarc.example.com&type=TXT'

Useful combinations:

Registered, no NS/A records: parked or abandoned, sometimes a candidate for a purchase offer.
Registered, has MX but no DMARC: a spoofing risk if it's one of your domains.
Your own domain expiring in under 30 days with auto-renew off: fix it today.

Step 5: Turn it into a monitor

For a portfolio you own, run the check weekly and alert on:

days_until_expiry below a threshold (30 is a common default)
a status change, for example losing client transfer prohibited (transfer lock), which is worth knowing about immediately
nameserver changes you didn't make

A cron job that writes the CSV and diffs it against last week's copy is enough for most teams.

Step 6: Automating it (optional)

If you don't want to maintain the bootstrap handling, per-server rate limits, WHOIS fallback and DNS checks yourself, I packaged this as an Apify Actor: Bulk WHOIS & Domain Availability Checker. It's paid, so here's exactly what it does:

takes domains, URLs or email addresses, and extracts and de-duplicates the registrable domain
queries RDAP via the IANA bootstrap and follows the registrar's RDAP server for extra detail, with a port-43 WHOIS fallback for ccTLDs without RDAP
returns availability (registered / available / pending-delete / unknown), registrar, created/expiry dates, domainAgeYears, daysUntilExpiry, an expiringSoon flag, EPP statuses, nameservers and DNSSEC
optionally adds NS/A/MX, SPF and DMARC, plus hasWebsite / hasEmail flags
writes a SUMMARY record and an AVAILABLE CSV. Redacted registrant data stays empty and nothing is guessed.

Pricing from the README: $2 per 1,000 domains checked, plus $0.005 per run, with DNS included. Its "available" has the same caveat as above: no registry record and no DNS delegation, so premium or reserved names can still be unbuyable.

If you're vetting an expired domain before buying it, pair this with a look at its history. I covered that in How to find and fix lost pages after a website migration using the Wayback Machine CDX API.

Checklist

[ ] Use RDAP (JSON) via the IANA bootstrap instead of parsing WHOIS text
[ ] Clean input to registrable domains with the Public Suffix List
[ ] Respect per-server rate limits, and back off on 429
[ ] Treat "available" as "confirm at a registrar"
[ ] Add DNS (NS/MX/DMARC) for context
[ ] Schedule it and alert on expiry, status and nameserver changes

How to audit your XML sitemap for indexing problems

Andy Zeng — Mon, 26 Oct 2026 13:21:35 +0000

Disclosure: I build and sell the paid Apify Actor mentioned near the end. Everything before that section works with free tools. This article was drafted with AI assistance; the code and examples were tested.

Your XML sitemap is supposed to be a list of pages you want indexed. In practice it often turns into a list of pages that redirect, return 404, carry a noindex tag, or tell Google that some other URL is the real one. Each of those sends a mixed signal. When Search Console shows a long "Discovered – currently not indexed" or "Crawled – currently not indexed" list, the sitemap is one of the first places to look.

This tutorial walks through a full sitemap audit:

Find every sitemap the site exposes.
Expand sitemap indexes and gzipped files into one URL list.
Check each URL for the problems that contradict "please index this".
Fix them at the source, usually the CMS or the sitemap generator.

Step 1: Find the sitemaps

Start with robots.txt. The Sitemap: directive can appear anywhere in the file, more than once, and it doesn't have to point to the same host:

curl -s https://example.com/robots.txt | grep -i '^sitemap:'

If there's no Sitemap: line, try the usual locations:

/sitemap.xml
/sitemap_index.xml (Yoast and many other plugins)
/wp-sitemap.xml (WordPress core since 5.5)

Also check Search Console under Indexing → Sitemaps to see which files were actually submitted. A missing Sitemap: line in robots.txt isn't an error, but adding it helps other crawlers find the file.

Step 2: Expand everything into one URL list

A sitemap is either a <urlset> (a list of pages) or a <sitemapindex> (a list of other sitemaps). Large sites nest them and often gzip them (.xml.gz). The protocol limits each file to 50,000 URLs and 50 MB uncompressed, so big sites end up with dozens of files.

For each URL, keep:

loc, the URL itself
lastmod, if present
which sitemap file it came from, so you know where to fix it

Watch for duplicates across files. The same URL listed in two sitemaps usually means two generators are running, for example a plugin plus a custom script.

Step 3: Decide what counts as a problem

A URL in a sitemap should pass all of these:

Check	Why it matters
Returns 200 without redirecting	A redirecting URL isn't the canonical one. List the final URL instead.
Not 4xx/5xx	You're asking Google to index a page that doesn't exist.
No `noindex` in `<meta name="robots">` or the `X-Robots-Tag` header	It directly contradicts the sitemap. The header version is easy to miss because it's invisible in the HTML.
`rel="canonical"` points to itself	If it points elsewhere, you're submitting a duplicate.
Not disallowed in robots.txt	Google can't crawl it to see the content.
`lastmod` is a valid W3C date and not in the future	Google has said it uses `lastmod` only when it's consistently accurate. A bogus date trains it to ignore yours.
HTTPS and on the right host	`http://` or staging-host URLs in a production sitemap are common after migrations.

Two judgment calls:

Stale lastmod (say, older than a year) isn't an error. It's worth reviewing on content-heavy sites, because it often means the date is frozen or the page is neglected.
Canonical comparison should resolve relative canonicals and ignore trailing-slash differences, or you'll get false positives.

Step 4: A script that does all of the above

Here's a compact, tested Python version (needs pip install httpx). It finds the sitemaps, follows nested indexes and gzip, and audits the first N URLs:

import gzip, re, sys
import xml.etree.ElementTree as ET
from datetime import datetime, timezone
from urllib.parse import urljoin, urlsplit
from urllib.robotparser import RobotFileParser
import httpx

UA = "sitemap-audit-script (contact: you@example.com)"
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
client = httpx.Client(headers={"User-Agent": UA}, timeout=20, follow_redirects=False)

def find_sitemaps(site):
    robots_url = f"https://{site}/robots.txt"
    r = client.get(robots_url)
    found = re.findall(r"(?im)^\s*sitemap:\s*(\S+)", r.text) if r.status_code == 200 else []
    if not found:
        for path in ("/sitemap.xml", "/sitemap_index.xml", "/wp-sitemap.xml"):
            if client.get(urljoin(f"https://{site}", path)).status_code == 200:
                found.append(urljoin(f"https://{site}", path))
                break
    return robots_url, found

def read_sitemap(url, seen=None, limit=200):
    seen = seen if seen is not None else set()
    if url in seen or len(seen) >= limit:
        return []
    seen.add(url)
    r = client.get(url, follow_redirects=True)
    body = r.content
    if url.endswith(".gz") or body[:2] == b"\x1f\x8b":
        body = gzip.decompress(body)
    root = ET.fromstring(body)
    if root.tag.endswith("sitemapindex"):
        out = []
        for loc in root.findall("sm:sitemap/sm:loc", NS):
            out += read_sitemap(loc.text.strip(), seen, limit)
        return out
    return [(u.findtext("sm:loc", "", NS).strip(), u.findtext("sm:lastmod", None, NS), url)
            for u in root.findall("sm:url", NS)]

def audit(url, lastmod, robots, site):
    issues = []
    if urlsplit(url).scheme != "https": issues.append("http-not-https")
    if not urlsplit(url).netloc.endswith(site): issues.append("off-domain-url")
    if not robots.can_fetch("Googlebot", url): issues.append("blocked-by-robots-txt")
    if lastmod:
        try:
            d = datetime.fromisoformat(lastmod.replace("Z", "+00:00"))
            if d.tzinfo is None: d = d.replace(tzinfo=timezone.utc)
            if d > datetime.now(timezone.utc): issues.append("future-lastmod")
        except ValueError:
            issues.append("invalid-lastmod")
    try:
        r = client.get(url)
    except httpx.HTTPError:
        return issues + ["unreachable"]
    if r.status_code in (301, 302, 307, 308):
        return issues + ["redirected"]
    if r.status_code >= 400:
        return issues + [f"http-{r.status_code}"]
    head = r.text[:200_000]
    robots_meta = " ".join(re.findall(r'<meta[^>]+name=["\']robots["\'][^>]*content=["\']([^"\']+)', head, re.I))
    if "noindex" in (robots_meta + r.headers.get("x-robots-tag", "")).lower():
        issues.append("noindex")
    m = re.search(r'<link[^>]+rel=["\']canonical["\'][^>]*href=["\']([^"\']+)', head, re.I)
    if m and urljoin(url, m.group(1)).rstrip("/") != url.rstrip("/"):
        issues.append("canonical-mismatch")
    return issues

if __name__ == "__main__":
    site = sys.argv[1]
    robots_url, sitemaps = find_sitemaps(site)
    rp = RobotFileParser(robots_url); rp.read()
    entries = []
    for sm in sitemaps:
        entries += read_sitemap(sm)
    seen = set()
    print(f"{len(sitemaps)} sitemap(s), {len(entries)} URLs")
    for url, lastmod, src in entries[:int(sys.argv[2]) if len(sys.argv) > 2 else 50]:
        dup = url in seen; seen.add(url)
        issues = audit(url, lastmod, rp, site) + (["duplicate-in-sitemaps"] if dup else [])
        if issues:
            print(url, issues)

Run it:

python sitemap_audit.py example.com 200

When I ran it against wordpress.org, it found 9 sitemap files, and the first audited URLs it flagged were /enterprise/... pages carrying noindex while still being listed in the sitemap. That's exactly the kind of contradiction this audit is meant to surface. (Whether those pages should be noindexed is a separate decision. The point is that the two signals disagree.)

Limitations of this script, so you know what to extend:

It audits sequentially. For thousands of URLs, add async requests with low concurrency (around 5) and a clear User-Agent.
It reads every sitemap file before auditing. On very large sites (wordpress.org lists hundreds of thousands of URLs), cap the number of files.
Regex HTML parsing is fine for <head> tags in served HTML but isn't a real parser. It also doesn't run JavaScript, so canonicals or robots tags injected client-side won't be seen.
It checks robots.txt rules for Googlebot only.

Step 5: Fix problems at the source

Patching the XML by hand rarely sticks, because the generator overwrites it. Instead:

Redirected URLs: usually an http→https, trailing-slash, or www mismatch between the CMS "site URL" setting and the real canonical host. Fix the setting, and every URL in the generated sitemap changes with it.
404s: deleted posts or products that the generator still lists. Typical causes are caching plugins or a static sitemap file left behind from an old setup (check that /sitemap.xml isn't a stale file sitting in the web root).
noindex pages in the sitemap: most SEO plugins exclude noindexed content automatically. If they don't, a custom template or a per-page header is overriding them. Check the X-Robots-Tag header at the server or CDN level too.
Canonical mismatches: often paginated, filtered, or parameterized URLs that canonicalize to a parent. Remove them from the sitemap rather than changing the canonical.
Future or invalid lastmod: usually a timezone bug or a "now()" default. Use the real content modification date.

After fixing, resubmit the sitemap in Search Console and re-run the audit. Then schedule it, because sitemaps drift every time someone installs a plugin or migrates a section.

Step 6: Automating it (optional)

If you'd rather not maintain the script, I packaged this workflow as an Apify Actor: Sitemap Extractor & SEO Index Audit. It's paid (pay-per-event), so here's exactly what it does:

finds sitemaps via robots.txt Sitemap: lines, then falls back to /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and other common paths
follows nested indexes and .xml.gz, and keeps lastmod, changefreq, priority, hreflang alternates and image/video counts
flags http-404/http-5xx, redirected, noindex (meta or header), canonical-mismatch, blocked-by-robots-txt, unreachable, stale-lastmod/future-lastmod/invalid-lastmod, duplicate-in-sitemaps, http-not-https and off-domain-url
writes an ISSUES CSV of only the problem URLs, plus a SUMMARY with a health score (the percentage of audited URLs that are indexable)

Pricing from the README: $0.30 per 1,000 extracted URLs and $1.20 per 1,000 audited pages, so a full audit of 1,000 pages is about $1.50. You can set a max cost per run. Like the script above, it doesn't execute JavaScript.

If you just migrated a site, it pairs with my earlier post, How to find and fix lost pages after a website migration. That one covers the old URLs that dropped out of the sitemap. This one covers the URLs that are in it but shouldn't be.

Checklist

[ ] All sitemap files found (robots.txt, common paths, Search Console)
[ ] Indexes and gzip expanded, duplicates found
[ ] Every URL: 200 without redirect, no noindex (meta and header), self-canonical, crawlable
[ ] lastmod valid and honest
[ ] Fixes made in the generator or CMS, not by hand-editing the XML
[ ] Resubmitted and scheduled to re-run

How to find and fix lost pages after a website migration

Andy Zeng — Mon, 26 Oct 2026 13:07:10 +0000

Disclosure: I build and sell two paid Apify Actors that automate parts of this workflow. They're mentioned in step 6 as one option. Everything else in this post works with free tools. This article was drafted with AI assistance; the code and examples were tested.

A migration is the easiest way to lose search traffic you already earned. You redesign the site, change the CMS, move /blog/2019/my-post to /insights/my-post, and forget about a few hundred old URLs. Other sites still link to those URLs, and Google still has them indexed. After launch they return 404, or every one of them redirects to the homepage, and the rankings they held disappear.

The fix is simple to describe and tedious to do:

Build a complete list of URLs the old site had.
Check what each one returns today.
Map every broken URL to its closest live equivalent with a 301.
Verify, deploy, and monitor.

This post walks through each step with commands you can run right now.

Step 1: Build the "old URL" inventory from several sources

No single source has every URL, so combine a few:

Your old XML sitemap. If you still have a backup or a staging copy, export it. It's the cleanest list, but it only has what the CMS chose to include.
Google Search Console. Indexing → Pages and the Performance report (export the "Pages" tab) show URLs Google actually knew about and that actually got clicks. Those are the ones to protect first.
Analytics. Export landing pages from the last 12–16 months.
Backlink data. Any backlink tool (or GSC's Links report) shows "top linked pages". A linked URL that now 404s is wasting link equity.
The Internet Archive's Wayback Machine. This is the one people forget. It often has URLs that never made it into a sitemap: old campaign pages, docs from a previous product version, tag archives.

Pulling URLs from the Wayback Machine (free)

The Internet Archive has a public CDX API that lists every capture for a host. This query returns unique HTML URLs that were archived with status 200:

curl -s "https://web.archive.org/cdx/search/cdx?url=example.com&matchType=host&fl=original,timestamp&collapse=urlkey&filter=statuscode:200&filter=mimetype:text/html" > wayback.txt

Useful parameters:

matchType=host means exactly that host. domain includes subdomains. prefix with url=example.com/blog/ limits it to one section.
collapse=urlkey removes duplicate captures of the same URL.
limit= and the showResumeKey/resumeKey parameters let you page through big sites.

Two things to know before you rely on it:

It's slow and flaky. Large hosts can take tens of seconds per page, and you will see 503/504 or "temporarily offline" responses. Retry with backoff instead of giving up.
It's noisy. You'll get ?utm_source= and ?fbclid= variants, wp-json endpoints, login links, and so on. Normalize before you check anything:

from urllib.parse import urlsplit, urlunsplit

def normalize(url: str) -> str:
    parts = urlsplit(url.strip())
    host = parts.netloc.lower().split(":")[0]   # drop :80
    path = parts.path or "/"
    return urlunsplit(("https", host, path, "", ""))  # drop query + fragment

SKIP = ("/wp-json", "/cdn-cgi", "/wp-login", ".css", ".js", ".png", ".jpg", ".woff")

with open("wayback.txt") as f:
    urls = {normalize(line.split()[0]) for line in f if line.strip()}
urls = sorted(u for u in urls if not any(s in u for s in SKIP))

Dropping all query strings is a blunt choice. If your site uses ?id=123 as real page identity, keep those parameters instead.

Merge this list with your sitemap, GSC, analytics, and backlink exports, and de-duplicate. That's your inventory.

Step 2: Check what every old URL returns today

For each URL, you want the status code, the final URL after redirects, and the number of hops. Be polite: low concurrency, a real User-Agent, and timeouts.

import asyncio, httpx, csv

async def check(client, url):
    try:
        r = await client.get(url, follow_redirects=True, timeout=20)
        return url, r.status_code, str(r.url), len(r.history)
    except httpx.HTTPError as e:
        return url, None, None, type(e).__name__

async def main(urls):
    limits = httpx.Limits(max_connections=4)
    headers = {"User-Agent": "migration-audit (contact: you@example.com)"}
    async with httpx.AsyncClient(limits=limits, headers=headers) as client:
        results = await asyncio.gather(*(check(client, u) for u in urls))
    with open("live_status.csv", "w", newline="") as f:
        csv.writer(f).writerows([("url", "status", "final_url", "hops"), *results])

asyncio.run(main(urls))

Then put each URL into one of these buckets:

Bucket	What it means	Action
`200`, same URL	Fine	Nothing
`301` → relevant page	Fine	Collapse chains longer than one hop
`404` / `410`	Lost	Redirect or deliberately retire
Redirects to `/`	Soft 404	Replace with a specific redirect
`5xx` / timeout	Broken or blocked	Investigate server/WAF

The homepage bucket deserves its own mention. "Redirect everything old to the homepage" is a common migration shortcut, and Google generally treats irrelevant redirects to the homepage as soft 404s, so you lose the equity anyway. Detect it by checking whether final_url normalizes to the site root while the original path didn't.

Step 3: Map each lost URL to its best replacement

This is the part that takes the most human judgment. A good redirect target is the page that best satisfies the same intent, not the closest string.

A practical way to speed it up is to let fuzzy matching propose candidates, then review them:

from difflib import SequenceMatcher
from urllib.parse import urlsplit

def path(u): return urlsplit(u).path.rstrip("/").lower()

def best_match(lost_url, live_urls, threshold=0.6):
    scored = [(SequenceMatcher(None, path(lost_url), path(l)).ratio(), l) for l in live_urls]
    score, target = max(scored, default=(0, None))
    return (target, round(score, 2)) if score >= threshold else (None, round(score, 2))

live_urls should come from your new sitemap or a crawl of the new site, not the old list. Treat every suggestion as a draft:

High score + same topic → accept.
Low score → open the archived page (https://web.archive.org/web/*/OLD_URL) to see what it actually was, then pick the target by hand.
Nothing equivalent exists → a parent category page is usually better than the homepage. If the content is truly gone and has no links or traffic, a 410 is an honest answer.
High-value page with no equivalent → consider restoring the content. The archived snapshot gives you the old title, headings, and text to start from.

Write the result as a simple CSV: from,to,status.

Step 4: Deploy the redirects

Most platforms take a list in some form:

Nginx

map $request_uri $redirect_target {
    /old-pricing/        /pricing/;
    /blog/2019/my-post/  /insights/my-post/;
}
server {
    if ($redirect_target) { return 301 $redirect_target; }
}

Apache (.htaccess)

Redirect 301 /old-pricing/ /pricing/
Redirect 301 /blog/2019/my-post/ /insights/my-post/

Netlify _redirects

/old-pricing/        /pricing/            301
/blog/2019/my-post/  /insights/my-post/   301

Cloudflare Bulk Redirects and WordPress plugins such as Redirection accept a CSV import.

Rules of thumb: point redirects at the final URL (no chains), use 301 for permanent moves, and keep the map in version control so the next migration doesn't wipe it out.

Step 5: Verify, then make sure the new sitemap is clean

Re-run the Step 2 script against the from column. Every row should now resolve with one hop to the intended to URL and a 200.

Then audit the new sitemap. A migration often ships a sitemap that still lists URLs that redirect, return 404, carry noindex, or have a canonical pointing somewhere else. Each of these sends Google mixed signals and is a common reason for "Crawled – currently not indexed" in Search Console. For each sitemap URL, check:

status code is 200 without a redirect
no noindex in <meta name="robots"> or the X-Robots-Tag header
<link rel="canonical"> points to itself
it isn't disallowed in robots.txt
lastmod is a valid date and not in the future

Finally, watch GSC's Pages report and your 404 logs for a few weeks after launch. New 404s with referrers are usually links you missed.

Step 6: Automating it (optional)

Everything above is doable with the scripts in this post and a spreadsheet. If you'd rather not maintain the retries, paging, normalization, and matching yourself, I packaged the workflow into two Apify Actors. Both are paid (pay-per-event), so here's exactly what they do and cost:

Wayback Machine URL Extractor & Lost Page Finder covers steps 1–3 for the archive source. It pages through the CDX API with retries, filters noise, checks live status, and sorts URLs into categories such as lost, redirects-to-homepage, and unreachable. It suggests a replacement for each lost URL using the same fuzzy path similarity described above, with homepage and a score of 0 when nothing scores 0.6 or higher, and writes a from,to,score,status,category CSV. It can also recover the archived title, meta description, H1, and text. The README puts a typical 1,000 URLs with live checks at about $2.00.
Sitemap Extractor & SEO Index Audit covers step 5. It finds sitemaps via robots.txt and common paths, follows nested indexes and .xml.gz files, and flags 404s, redirects, noindex, canonical mismatches, robots.txt blocks, and lastmod problems. It outputs an ISSUES CSV. A full audit of 1,000 pages costs about $1.50.

Limitations to know: the redirect suggestions are string similarity, not semantic matching, so you still need to review them. The sitemap auditor doesn't execute JavaScript, so it reads the HTML <head> as served. And the Wayback Machine only knows about pages it happened to capture.

Checklist

[ ] Old URL inventory from sitemap, GSC, analytics, backlinks, and Wayback CDX
[ ] Normalized and de-duplicated
[ ] Live status for every URL, bucketed (lost / soft 404 / chain / error)
[ ] Redirect map reviewed by a human, no chains, no blanket homepage redirects
[ ] Deployed and re-verified
[ ] New sitemap audited: only 200, indexable, self-canonical URLs
[ ] 404 monitoring for the first few weeks

If you've done a migration that went sideways, I'd like to hear what you missed. That's how this checklist got longer.