How to Get the Full Wayback Machine History of Any Website (No Code or Python)

How to Get the Full Wayback Machine History of Any Website

If you have ever tried to answer "when did this competitor change their pricing page?" or "what did this domain look like before I bought it?", you have probably clicked around web.archive.org one calendar dot at a time. It works, but it does not scale past a handful of pages, and you cannot put it in a spreadsheet.

This tutorial shows how to pull the complete capture history of a URL or domain into a table you can filter, chart and diff. We will use the Internet Archive's public CDX API through a small hosted tool, the Wayback Machine Scraper on Apify, and then look at the same thing from Python.

What the CDX API actually gives you

The Wayback Machine keeps an index of every capture it has made. Each entry has:

• the original URL and capture timestamp
• the HTTP status code the archive saw (200, 301, 404...)
• the MIME type
• a content digest (a hash of the page body)

The digest is the interesting bit. If two consecutive captures have the same digest, the page did not change. If it differs, something on the page changed. That turns the archive from a photo album into a change log.

The raw API is free and public, but it has quirks: timestamps like 20200109130705, pagination by line, and collapse rules that are easy to get wrong. The actor wraps all of that and returns clean rows.

Step 1: Run it with the example input

Open the actor page and click Try for free. The input is prefilled with apify.com, monthly collapse, from 2020, limit 20. Click Start. In a few seconds you get rows like:

capturedAt	statusCode	contentChanged	snapshotUrl
2020-01-09T13:07:05Z	200	null	https://web.archive.org/web/20200109130705/https://apify.com/
2020-02-03T...	200	true	...

contentChanged is null for the first row (nothing to compare against) and true/false after that.

Step 2: Choose the right collapse mode

The collapse field decides how many rows you get back:

• month: at most one capture per month. Best for a quick overview of a site over years.
• day: one per day. Good for a period you care about, like a product launch.
• digest: only captures where the content changed. This is the one to use for change tracking.
• none: every single capture. Popular sites can have tens of thousands, so set a limit.

Step 3: Pick a match type

• exact looks at one URL, such as example.com/pricing.
• prefix covers everything under a path, such as example.com/blog/. Handy for finding deleted blog posts.
• domain covers the whole domain including subdomains.

A practical example: to see every blog URL a competitor ever published, use matchType=prefix, collapse=digest and a high limit. Then sort by URL and look for posts that return 404 today.

Step 4: Export or automate

Every run produces a dataset you can download as CSV, Excel or JSON. You can also schedule the actor weekly and get only new captures by setting from to last week's date.

From Python with the official client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("scrape-smith/wayback-snapshots").call(run_input={
    "urls": ["example.com/pricing"],
    "from": "2022",
    "collapse": "digest",
    "limit": 200,
})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["capturedAt"], row["statusCode"], row["snapshotUrl"])

That prints one line per version of the pricing page since 2022. Open the snapshotUrl of two neighbours to see exactly what changed.

Use cases that come up a lot

1. SEO audits. When did a page start redirecting? When did the title change? Status-code history answers both.
2. Competitor monitoring. Pricing, plans and positioning pages, version by version.
3. Expired-domain due diligence. See whether a domain you are about to buy used to host spam.
4. Content recovery. Find old copies of pages you deleted by accident.
5. Research and journalism. Show how an organisation's public statements evolved.

Honest limitations

• The archive only has what it crawled. Small sites may have a few captures a year.
• The tool returns snapshot metadata and links, not the page HTML. If you need the HTML, fetch the snapshotUrl yourself (add id_ after the timestamp for the raw page).
• The digest changes on any byte change, so pages with rotating timestamps or ads will show "changed" more often than a human would say.
• This is not an official Internet Archive product. Please be kind to their servers; the actor makes a modest number of requests.

What it costs

Pricing is pay-per-result: $0.0005 per snapshot plus $0.003 per run. A monthly history of one domain since 2020 is around 80 rows, so about four cents. 1,000 snapshots is $0.50. Apify's free plan credit easily covers experimenting.

Wrap-up

The Wayback Machine is one of the most useful free datasets on the web, and the CDX index makes it queryable. If you want to try it without writing any code, the Wayback Machine Scraper has a prefilled example you can run in one click. If you build something with it, I would love to hear what.

Scrape-Smith builds small, cheap data tools on public and official APIs.