Rebuilding a 7,230-URL site from the Wayback Machine into WordPress: CDX, raw bytes and legacy ?p= URLs

Dmitriy Gabdrakhmanov — Wed, 28 Oct 2026 10:41:44 +0000

When a site goes offline without a backup, or you buy an expired domain with history, the usual advice is to run a downloader from GitHub. Tools like wayback-machine-downloader do a fair job of fetching files, but you end up with a folder of static HTML, legacy URLs like ?p=123 saved as files your host won't serve, and no record of what the archive never had.

I've built 150+ sites on expired domains over the past two years, and since spring 2026 I restore sites from the archive into WordPress with my own CLI pipeline. The largest so far was 63,000 pages. Here is how it works, using a real run on criminalbrief.com, a mystery fiction blog archived since 2007.

1. Use the CDX index, not the calendar

The calendar only shows when the crawler visited a given URL, usually the homepage. Deep posts may have been captured once, years apart. The CDX index shows everything the archive holds for a domain:

curl -s "https://web.archive.org/cdx/search/cdx?url=criminalbrief.com/*&to=20250901&output=json&fl=original,statuscode,timestamp,mimetype,digest&collapse=urlkey" > inventory.json

For criminalbrief.com this gave 7,230 known URLs (the CDX index plus URLs linked from saved pages). Every URL gets a status and a reason in a ledger. The result:

4,822 (66.7%) restored on their original URLs: posts, pages, categories and media, about 3,000 of them HTML pages
1,418 filtered by policy (410): ?rest_route= calls (1,228), feeds, filter parameters
922 never captured (410): addresses linked from saved pages that the archive has no copy of, often broken links like /Bob%20Martinez/
32 admin URLs excluded
36 failed: the archive kept returning errors after retries

External files are counted separately: images and scripts hosted on other sites that the archive never saved (261 files on 204 hosts here). They are listed in the report, and nothing is substituted from live sites.

2. Fetch raw bytes with `id_`

A normal archive URL returns rewritten HTML: toolbar, wombat.js, and every link pointing to web.archive.org. Add id_ right after the timestamp and you get the bytes the original server sent:

curl -sL "https://web.archive.org/web/20150920071842id_/http://criminalbrief.com/?p=9666" -o page-9666.html

Two more things matter here:

Gap-fill. A post captured in 2015 may reference a stylesheet or image that was only captured in 2014 or 2017. A second pass looks for those files across the whole window and deduplicates them by the CDX digest.
Rate limits. The archive answers 429 often. I keep about 1 request per second and respect Retry-After.

3. Keep legacy `?p=` and `?cat=` URLs alive in WordPress

Backlinks often point to /?p=9666, /?cat=7, /?m=201405 or .html paths. With pretty permalinks on, WordPress runs redirect_canonical and sends /?p=9666 to a 301 on the new slug. /?cat=7 gets rendered by the theme's archive template, and the original markup is lost.

What I do at build time:

import posts with their original IDs in wp_posts
convert old encodings to UTF-8 and recalculate serialized string lengths (s:N:"...") when replacing the hostname
add an MU plugin router that matches path + query against a route map before WordPress parses query vars

The smallest piece of this you can try yourself is stopping the canonical redirect for legacy query URLs:

<?php
// Minimal example: keep legacy ?p=, ?cat=, ?tag=, ?m= URLs on their exact address.
// Not the full router: it only stops the 301, the theme still renders the page.
add_filter('redirect_canonical', function ($redirect_url, $requested_url) {
    $query = parse_url($requested_url, PHP_URL_QUERY);
    if ($query && preg_match('/^(p|cat|tag|m)=/', $query)) {
        return false;
    }
    return $redirect_url;
}, 10, 2);

With the full router, /?p=9666 and /?cat=7 answer 200 OK with their original markup, stored as editable HTML blocks, and new posts work in the normal editor.

4. 410 instead of redirecting everything to the homepage

Redirecting every missing URL to / hides what was lost, and Google says redirecting many old URLs to an irrelevant page like the homepage may be treated as a soft 404. In this run:

200 for restored pages
301 only where the original site redirected in the archive (0 in this run)
410 for junk and never-captured URLs

For Google, 404 and 410 are handled the same. The point is an honest status instead of a homepage redirect. Every non-200 URL goes into redirects.csv and the QA report with a reason. A few real rows from the report:

URL                                                              status  reason
http://criminalbrief.com/?feed=rss                               gone    not restored by policy (feed, media or non-page address)
http://criminalbrief.com/?antiselect=1031223                     gone    not restored by policy (feed, media or non-page address)
https://criminalbrief.com/Bob%20Martinez/                        gone    no copy of this address in the web archive
http://www.criminalbrief.com/stats/webalizer/usage_200706.html   failed  the archive copy could not be downloaded

Failed URLs answer 410 for now and are retried automatically on the next run. Anyone can run curl -I on any of these addresses and check that the live response matches the report.

Before you launch a restored site

Compare with CDX, not the calendar. Count statuscode:200 + mimetype:text/html URLs in the index and check that deep pages from earlier years made it.
Test legacy URLs. Take 10-20 addresses from your backlink profile, especially ?p=, ?cat= and .html, and run curl -I. They should answer 200 OK with no redirect to a new slug.
Grep for archive traces. Search the files and the SQL dump for web.archive.org, wayback, wombat and /web/20. A clean build has zero matches.
Request a URL that never existed. It should return 410 or 404, not a 301 to /.

More screenshots and report excerpts: wpservicelab.com/samples

DEV Community: Dmitriy Gabdrakhmanov