Three situations where I have needed the Wayback Machine, and where its website is the wrong tool:
- After a site migration, traffic dropped. Which old URLs used to exist, and which of them now 404 without a redirect?
- An old blog post has 40 outbound links, and a third of them are dead. Each needs an archive link instead.
- An expired domain is for sale with "20 years of history". Was it a real site for 20 years, or a parked redirect for the last ten?
The Internet Archive has a public index API (the CDX server) that answers all of these. It is free, generous and also slow and easy to overload: heavily archived pages can take 30–60 seconds per query, and it starts refusing at about 24 queries a minute. So I wrapped it in an Apify Actor, Wayback Machine Scraper, that queues politely, retries, de-duplicates, and returns clean rows. Here is how each job looks.
Job 1: every URL a site ever had
Mode urls lists every page the Wayback Machine has captured for a domain, a host or a path, with the first capture date:
{
"urls": ["https://blog.example.com/"],
"mode": "urls",
"matchType": "prefix",
"fromDate": "2020",
"maxResultsPerInput": 5000
}
A real row for python.org:
{
"type": "url",
"input": "python.org",
"url": "http://www.python.org/1.5/",
"path": "/1.5/",
"firstCapturedAt": "1998-01-19T01:50:21Z",
"statusCode": 200,
"mimeType": "text/html",
"firstSnapshotUrl": "https://web.archive.org/web/19980119015021/http://www.python.org:80/1.5/",
"latestSnapshotUrl": "https://web.archive.org/web/http://www.python.org/1.5/",
"charged": true
}
By default you get only successful HTML pages, and http://example.com:80/a/ and https://www.example.com/a count once. That is what you want for a redirect map: export the path column, diff it against your current sitemap, and every path that is in the archive but not on the new site is a candidate for a 301. Add urlPattern (a regex) to focus on, say, /blog/ or /products/.
Job 2: an archive link for every dead link
Mode closest returns the snapshot nearest to a date (or the latest one) for each URL you paste:
{
"urls": ["https://www.nasa.gov/missions/", "https://example.com/old-page"],
"mode": "closest"
}
With targetDate and includeContent, you also get the archived page as Markdown, which is handy when you want to quote what a page said rather than just link to it:
{
"type": "closest",
"input": "https://www.nasa.gov/missions/",
"targetDate": "2015-06-01T00:00:00Z",
"capturedAt": "2015-06-12T18:50:45Z",
"statusCode": 200,
"snapshotUrl": "https://web.archive.org/web/20150612185045/http://www.nasa.gov:80/missions/",
"daysFromTarget": 12,
"contentTitle": "NASA Missions A-Z | NASA",
"contentFormat": "markdown",
"charged": true
}
A URL that was never archived gets a free row (errorType: "not_found") instead of a fake answer.
This pairs naturally with a link checker: run Broken Link Checker on your site, feed its dead links into closest mode, and you have a replacement-link sheet for your CMS.
Job 3: every version of a page, with the text
Mode snapshots lists the archived versions of one page. With the default collapse: "changes", you get one row per content change, not one per crawl, which removes most of the noise:
{ "urls": ["stripe.com/pricing"], "mode": "snapshots", "collapse": "changes", "includeContent": true, "maxContentPages": 20 }
{
"type": "snapshot",
"input": "stripe.com/pricing",
"index": 2,
"capturedAt": "2020-01-04T16:56:45Z",
"statusCode": 200,
"digest": "GMSUHBJQMSMCD7CWRFAY4X2OPZGY5BJW",
"lengthBytes": 69301,
"snapshotUrl": "https://web.archive.org/web/20200104165645/https://stripe.com/pricing",
"rawUrl": "https://web.archive.org/web/20200104165645id_/https://stripe.com/pricing",
"contentChanged": true,
"daysSincePrevious": 243
}
digest is the capture's content fingerprint (same digest = same content), and rawUrl is the file exactly as captured, without the Wayback toolbar. With includeContent, each version comes as Markdown, ready to diff or to hand to an LLM: "how did this pricing page change between 2019 and 2022?"
Job 4: is this expired domain what it claims?
Mode history gives one summary row per domain. This is pets.com, a real result:
{
"type": "history",
"input": "pets.com",
"firstCapturedAt": "1996-12-21T00:37:03Z",
"lastCapturedAt": "2026-09-20T01:06:47Z",
"lastOkMonth": "2016-04",
"archiveAgeYears": 29.7,
"totalCaptures": 2391,
"monthsWithCaptures": 268,
"monthsWithOkPages": 40,
"monthsRedirectOnly": 228,
"longestGapMonths": 19,
"distinctVersions": 117,
"redirectShare": 0.494,
"calendarUrl": "https://web.archive.org/web/*/pets.com"
}
"Archived for almost 30 years" sounds impressive. But the last month with a real page (HTTP 200) was April 2016, and 228 of the 268 months with captures were redirect-only. If you were evaluating this as an aged domain for SEO, that is the line that matters, and it is exactly the line a marketplace listing will not show you.
What it costs
| Event | Free / Starter | Scale | Business and higher |
|---|---|---|---|
| Archived URL or snapshot (one row) | $0.001 | $0.0009 | $0.0008 |
| Site history summary (one per domain) | $0.003 | $0.0027 | $0.0024 |
| Archived page content (optional) | $0.003 | $0.0027 | $0.0024 |
No start fee. So 1,000 archived URLs of a site cost $1, and a history check of 100 domains costs $0.30. Inputs that were never archived, invalid lines, duplicates, filtered-out captures and content that could not be downloaded are free.
A real example from my tests: two pages, closest snapshot to June 2015 with their Markdown, plus one never-archived domain = 2 captures + 2 page contents = $0.008.
Limits worth knowing
- The Wayback Machine is a shared free service. The Actor defaults to one index query every 2.5 seconds and one page download every 5 seconds, and backs off when the archive is busy. Big jobs take time; that is the price of not hammering it.
- Only what was archived. Pages that were never crawled, or were excluded or removed at the owner's request, are not available.
- JavaScript-heavy pages may have little text in their archived HTML.
- For live change alerts on a page today, use a change monitor; this tool is for the past.
Terms and responsible use
The Actor reads only the Internet Archive's public endpoints (the CDX index, the availability API and snapshot playback), at a gentle rate, without logging in. Archived pages remain their owners' work, so check copyright before republishing them. I am not affiliated with the Internet Archive; if you rely on it, consider donating.
Actor: https://apify.com/tidytools/wayback-machine-scraper
Disclosure: I built this Actor and earn from its usage on Apify. The sites above are public examples from my tests in September 2026 and have no connection to me.
Top comments (0)