DEV Community

Cover image for How to Find Every Archived URL of a Domain in the Wayback Machine
veenkes
veenkes

Posted on

How to Find Every Archived URL of a Domain in the Wayback Machine

How to Find Every Archived URL of a Domain in the Wayback Machine

After a site migration or a lost blog, the Wayback Machine often holds the only copy. This guide lists every URL archive.org captured for a domain, or the full snapshot history of a page.

You will use Wayback Machine Scraper: All Archived URLs & Snapshots on Apify. It runs in the cloud, needs no login or browser on your side, and the Free plan is enough for a first run.

What you get

  • 🗂️ All unique URLs of a domain — find old pages, lost content, forgotten files and URL structures.
  • 🕰️ Snapshot history of any page, deduplicated by content (only captures where the page changed).
  • 🔎 Filters: HTTP 200 only, MIME types (HTML, PDF, images…), date range, subdomains.
  • 🔗 Archive links in normal and raw (id_) form for direct download.
  • 💸 $0.15–$0.30 per 1,000 results, far below other Wayback actors.
  • 💸 From $0.15 per 1,000 results. No monthly rental.

Fields in the output

Field Example
url https://apify.com/agentx/tiktok-live-profile/output-schema
timestamp 2026-02-17T20:16:16Z
statusCode, mimeType, length, digest 200, text/html, 74011
archiveUrl, rawArchiveUrl https://web.archive.org/web/20260217201616/https://apify.com/...

How to run it in the Apify Console

  1. Enter domains (URL mode) or page URLs (snapshot mode).
  2. Optionally set filters and max results.
  3. Click Start and export to CSV, Excel or JSON.

A ready-made example to start from: All archived URLs of a domain.

Sample output

{
    "target": "apify.com",
    "url": "https://apify.com/agentx/tiktok-live-profile/output-schema",
    "timestamp": "2026-02-17T20:16:16Z",
    "statusCode": 200,
    "mimeType": "text/html",
    "length": 74011,
    "archiveUrl": "https://web.archive.org/web/20260217201616/https://apify.com/agentx/tiktok-live-profile/output-schema"
}
Enter fullscreen mode Exit fullscreen mode

The key-value store record SUMMARY lists every input with its status and the number of items saved.

Run it from Python

Install the client with pip install apify-client, create a token in Apify Console > Integrations and keep it out of public repos.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("veenkes/veenkes-wayback-urls").call(run_input={
    "targets": [
        "openai.com"
    ],
    "mode": "urls",
    "mimeTypes": [
        "text/html"
    ],
    "maxResultsPerTarget": 5000
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
Enter fullscreen mode Exit fullscreen mode

The same Actor works from JavaScript, on a schedule, through webhooks, Zapier, Make or n8n, and from AI agents through the Apify MCP server.

Cost

Apify plan tier Price per 1,000 results
FREE $0.30
BRONZE $0.25
SILVER $0.20
GOLD and above $0.15

There is also a $0.00005 start fee per run.

Limitations

  • The Wayback Machine index can be slow: a large domain may take a minute per few thousand URLs, and archive.org sometimes returns temporary errors (retried automatically).
  • In URL mode each URL appears once with its first matching capture.
  • Only what archive.org has captured is available.

Wrap-up

Open Wayback Machine Scraper: All Archived URLs & Snapshots, keep the prefilled input and click Start — the first results arrive in seconds. If something you need is missing, open an issue on the Actor page.

Top comments (0)