You can't un-archive a page, so guard the Wayback Machine save

tickerseer — Mon, 26 Oct 2026 15:17:00 +0000

I run tickerseer, a stock research site. Every week it publishes options trade ideas on companies about to report earnings, and when each one expires the code grades it on a public ledger. The page promises that a published call is never edited, withdrawn or removed.

On its own, that promise doesn't prove much, because I control the server and could change the page whenever I liked. So after each weekly settlement a cron job asks the Internet Archive to save the page. Anyone can open two captures and diff them, and I can't touch either copy.

That makes the archive the one place where a mistake sticks for good. A capture can't be edited, and getting one removed means asking the Internet Archive for a favour. The save is about ten lines of curl. Making sure it never stores the wrong page took the other 230 or so lines, most of them comments about the four traps below.

Trap 1: Save Page Now follows redirects

The obvious version looks like this:

curl -s "https://web.archive.org/save/https://example.com/page"

Save Page Now keeps whatever it ends up on. If your page redirects to a login screen, the archive saves the login screen. That happened to us on 15 August 2026: a manual save of a page that needs a paid plan stored our sign-in page in its place.

The fix is to fetch the page yourself first, without following redirects, and refuse to archive anything that doesn't answer 200 directly:

# NEVER add -L here. Following the redirect would read the 200 at the
# end of a redirect to a login gate, and then archive the gate.
resp="$(curl -sS -m 30 -w '\n%{http_code}' "$url" 2>/dev/null)" || resp=""
code="${resp##*$'\n'}"
body="${resp%$'\n'*}"

if [[ "$code" != "200" ]]; then
  echo "[wayback] REFUSED ${url}: answered '${code:-no response}', not 200" >&2
  return 1
fi

A test checks that -L stays out of that line, because a stubbed curl in a unit test can't tell you whether redirects are being followed.

Trap 2: a 200 can still be an empty page

Our ledger handler returns 200 on every path. When its data loader fails, it catches the exception, returns an empty list, and the page renders a polite "being prepared" placeholder, also with a 200.

Archiving that placeholder would be worse than archiving the login page. The page exists to prove nothing was taken down, and the archive would hold a copy showing an empty ledger.

So the caller passes a marker string that only a populated page contains, and the save is refused without it:

if [[ -n "$require" && "$body" != *"$require"* ]]; then
  echo "[wayback] REFUSED ${url}: 200 but no '${require}'" >&2
  return 1
fi

wayback_save "https://tickerseer.com/track-record" 120 "Overall Win Rate"

Trap 3: no snapshot URL doesn't mean no snapshot

When a save works, Save Page Now usually redirects to the new capture, and you read the URL from the Location header. Sometimes it answers 200 with an HTML page instead and saves the capture in the background. On 19 September 2026 both of our weekly saves came back with no Location, and both captures were stored anyway.

So a missing Location is a question, and the CDX index can answer it. It tells you which captures of a URL exist, when they were taken and what status code was recorded.

The catch is that the ledger gets archived every week, so the index is never empty. If you only ask "is there a capture?", the answer is yes forever, including the weeks when the save was broken. The check needs a lower bound, and that bound has to be stamped before the save is requested:

since="$(date -u +%Y%m%d%H%M%S)"   # before the save, not after
# ... request the save ...

body="$(curl -sS -m 30 \
  "https://web.archive.org/cdx/search/cdx?url=${url}&from=${since:0:8}&limit=-5&fl=timestamp,statuscode")"

while read -r ts code _; do
  [[ "$ts" =~ ^[0-9]{14}$ ]] || continue
  [[ "$code" == "200" ]] || continue
  if [[ "$ts" < "$since" ]]; then continue; fi   # fixed-width digits compare fine as strings
  echo "https://web.archive.org/web/${ts}/${url}"
  return 0
done <<< "$body"

from= only filters by date, so a capture that lands just after midnight UTC still comes back. The timestamp comparison is the real bound.

The check only works one way. A hit proves a capture from this run exists. A miss proves nothing, since the capture may just not be indexed yet, so we wait 20 seconds, ask once more, and then log a warning. It can fail to confirm a capture that happened, but it can never confirm one that didn't.

Trap 4: the index has bad days too

On the day I wrote that check, the CDX endpoint was serving a "Temporarily Offline" HTML page with HTTP 200 and curl exit status 0. It also timed out at times and sometimes returned an empty body.

That's why the loop above only accepts a line shaped exactly like <14 digits> <status>. The HTML outage page has no such line, so it counts as "no answer" and never as a capture.

Bash details that bit along the way

The function is best-effort. An outage at a third party shouldn't kill the weekly settlement job that calls it, and three habits keep it from doing that:

local url="${1:?missing}" looks like input validation, but in a non-interactive shell it exits the whole calling script, and a caller's || fallback can't catch that. Use ${1-} and return 2.
[[ "$ts" < "$since" ]] && continue returns non-zero when the test is false. That's harmless mid-loop, but if it ends up as the last command in a function, the false test becomes the function's return status, and under set -e an unguarded call site ends the script. if ...; then continue; fi returns 0 either way.
Every call site is written wayback_save ... || <log a warning>, and the two ledger saves also report to Sentry. A failed archive is something to investigate, and it doesn't fail the job.

What this still can't prove

All of those guards describe what my server sees. The capture that ends up stored is what the archive's crawler saw, from a different IP and a few seconds or minutes later, through a CDN with a firewall in front of it. If a firewall rule ever challenged the crawler, every check above would pass and the archive would store a challenge page.

The CDX check covers half of that. It proves a capture exists and was stored with a 200, but CDX returns metadata, not content, so it can't tell you the stored body is the ledger. Only opening the capture can tell you that.

What changes between two captures

If you diff two captures you will find differences, and the page says where to expect them. The calls stay the same, but the grading changes when I find bugs in the code that reads our own write-ups. In one of them, an entry line reading "Mon 7/6 to Tue 7/7" was parsed as a $7 strike price, so a $62 stock could never lose the trade. Each fix gets a dated entry under Corrections with its effect on the total, and the ones that helped our numbers get the same detail as the ones that hurt. With the archive in place, a bug like that can't be fixed quietly without someone being able to see it.

The helper ended up at about 240 lines of Bash, and most of them are comments. Each one records an incident like these, so whoever touches the file next knows why a guard is there before deleting it.

DEV Community: tickerseer