DEV Community

Mohamed Fouad
Mohamed Fouad

Posted on

How to Recover a Website's Lost Images from the Wayback Machine (in Bulk, with Python)

#webscraping #python #opensource #tutorial

You migrate a CMS, change a CDN path, or let an old domain lapse — and suddenly every image on the site returns 404. Backups? Not always. But there's a good chance those files still exist in a public web archive: the Internet Archive's Wayback Machine or Common Crawl.

Getting them back one by one through the Wayback calendar is painful. In this tutorial I'll show how the recovery works under the hood, and how to do it in bulk with an open-source CLI I wrote: wayback-image-recovery.

Why manual recovery doesn't scale

A few gotchas show up quickly:

  1. The Wayback UI wraps captures in HTML. You want the original file, not the page with the toolbar.
  2. The newest capture can be fake. Many captures are soft-404 error pages saved with HTTP 200.
  3. Duplicates everywhere. The same logo lives at dozens of URLs.
  4. Wayback doesn't have everything. A file missing there may still be in Common Crawl.

Step 1: Ask the CDX API what exists

The Wayback Machine exposes a CDX index. For one URL you can request the latest capture:

https://web.archive.org/cdx/search/cdx?url=example.com/images/logo.png&output=json&sort=reverse&limit=1&filter=statuscode:200
Enter fullscreen mode Exit fullscreen mode

For a whole domain you can list unique image URLs with matchType=domain, a mimetype:image/.* filter, and collapse=urlkey.

Gotcha: collapse=urlkey keeps the first row per URL, and rows are oldest-first. So the listing is good for discovery, but you must re-query each URL with sort=reverse&limit=1 (latest) or sort=closest&closest=20200101 (nearest to a date) to get the capture you actually want.

Step 2: Download the original bytes

Take the capture timestamp and add the id_ modifier:

https://web.archive.org/web/20210314120000id_/https://example.com/images/logo.png
Enter fullscreen mode Exit fullscreen mode

id_ returns the raw archived bytes with no rewriting or toolbar.

Step 3: Validate before you save

Don't trust the status code. Check that the response isn't HTML, that the magic bytes match an image type, and that the file isn't suspiciously small or truncated. Then hash it (SHA-256) so identical files are stored once.

Step 4: Fall back to Common Crawl

If Wayback has nothing usable, query Common Crawl's index for the URL. A hit gives you a WARC filename, offset and length — and an HTTP Range request fetches only that record instead of a multi-gigabyte file.

Doing all of this with one command

wayback-image-recovery wraps these steps into a resumable pipeline.

Install (Python 3.9+):

pip install "git+https://github.com/mohamd764/wayback-image-recovery.git"
Enter fullscreen mode Exit fullscreen mode

Recover every archived image under a domain:

wir recover --domain example.com --output ./recovered
Enter fullscreen mode Exit fullscreen mode

Or a list of URLs, preferring captures close to a date:

wir recover urls.txt --closest 2020-01-01 --output ./recovered
Enter fullscreen mode Exit fullscreen mode

Or a CSV export of broken links from your SEO crawler (a url, image_url or src column):

wir recover --csv broken-images.csv --min-bytes 2048
Enter fullscreen mode Exit fullscreen mode

Files keep their original host and path, so putting them back on the server is straightforward. Each run writes:

  • manifest.csv / manifest.json — one row per URL: source, snapshot timestamp, status, sha256, path, bytes, MIME type, error
  • manifest.jsonl — an append-only log, so re-running the same command skips files already recovered

A run summary looks like:

Processed 4 URL(s)
recovered=2 duplicate=1 skipped=0 not_found=1 invalid=0 error=0
Enter fullscreen mode Exit fullscreen mode

not_found is normal — archives don't have everything. Use --strict in CI if you want any miss to fail the job.

Be polite to the archives

The Internet Archive and Common Crawl are free public resources. Keep concurrency low, respect rate limits, and only recover content you have rights to. The tool never crawls the live site; it only reads the archives.

Wrap-up

The key ideas: discover with CDX, re-resolve each URL to the right capture, download with id_, validate and dedupe, and fall back to Common Crawl WARC range requests. You can implement that yourself — or use the tool:

👉 github.com/mohamd764/wayback-image-recovery (MIT, tested in CI)

If it saves you a migration headache, a ⭐ helps others find it, and issues/PRs are welcome.

Top comments (0)