The Most Dangerous Backup Is the One That Reports Success

TL;DR — I spent today building out a backup subsystem, and almost every design decision that mattered came down to the same question: can this thing fail in a way that still renders as a green tick? A backup that fails loudly is an annoyance. A backup that fails quietly is the reason the restore doesn't exist.

---

The failure mode nobody designs for

Backups have an unusual property among features: the moment you find out whether yours works is the worst possible moment to find out. Everything in between is inference. And inference is exactly where a system gets to lie to you politely.

So the rule I kept coming back to today: a component may not report a success it did not earn. Not "should not" — may not, structurally, because the code path that would produce the false green isn't there.

Here's what that looked like in practice.

Sync is not backup

The first one is the one people get wrong most confidently, because the tool that does it wrong is excellent at its job.

A sync tool mirrors a source into a destination. If a key disappears from the source, it disappears from the destination. That is correct behaviour for a mirror and catastrophic behaviour for a backup, because the thing you are backing up against is somebody deleting the wrong folder at 2am — and a mirror propagates that deletion to the only copy that could have undone it, usually within the hour, and then reports success.

So the bucket driver I wrote today records deletions and never replays them:

/**
 * A key that vanished at the source is marked deleted in the manifest.
 * Its object is LEFT WHERE IT IS until retention expires it.
 * Nothing in this driver deletes an object at a destination.
 */

Two tests pin the pair of consequences, and the second one matters as much as the first:

• an object deleted at the source is still restorable from the last run that had it;
• restoring a later run does not resurrect it.

That second test is the one people forget. A restore reproduces the bucket as of that run. An operator recovering one folder did not also ask to undo every intentional deletion since — and if your restore does that, you've handed them a second incident while they were cleaning up the first.

"Move" is copy, verify, then delete — in that order, always

Moving a backup between destinations is the most obviously dangerous operation in the whole subsystem, because there's a window where the artefact exists in exactly one place and something is actively trying to remove it.

The ordering rule is boring. What's interesting is making the rule checkable:

// source_deleted_at stays null until the destination copy is Verified,
// and is written in its own statement, after verified_at.

Two separate writes, deliberately. If they went out together you could argue forever about whether a bad row was a race or a bug. Kept apart, a query can find the violation:

test('a transfer never deletes the source before the destination verifies', function () {
    expect(BackupTransfer::deletedSourceBeforeVerification())->toBeEmpty();
});

The other half: the destination copy gets its own artefact row, with its own key, size, checksum and verification state — and the driver is asked to verify that row. A move that verified the source's checksum would have proved nothing about the transfer it existed to prove. It would have been a green tick describing the wrong object.

Verification that admits what it didn't check

This is my favourite one, because the honest version is uglier than the dishonest version and that's the whole point.

Verifying a bucket copy properly means re-hashing every object. To do that, the control plane has to read every object back through itself — during the verification of a transfer whose entire purpose was to keep that data out of the control plane. You can't have both.

The dishonest option is to check what's cheap and call it verified. What shipped instead: check object count, total size, and a spread sample of per-object sizes — and then say so.

'verification' => [
    'objects_checked' => 1_284,
    'bytes_expected'  => 4_118_233_712,
    'etags_rehashed'  => false,
],

etags_rehashed: false sits in the record permanently. A passing verification cannot be read, later, by somebody who wasn't there, as proving something it never checked. The same instinct produced a test that scans every operator-visible string in the feature for the words "point-in-time", "versioning" and "snapshot" — because the driver doesn't do those, and prose that merely declines to claim something is not the same as prose that states the opposite. Somebody reading the screen in two years will assume whatever the words allow.

Integrity is not restorability

Every run checks that the artefact is present, the right size, the right checksum, and that it opens. All cheap, all worth having, and none of them prove it restores.

An archive that passes pg_restore --list and then dies part-way through a real restore — a missing extension, an incompatible server version, an encoding mismatch — is indistinguishable from a good one right up until 3am.

So: restore drills. A policy can opt into verify_level = restorability, on its own cadence and its own cursor (drill_cron, next_drill_at — a nightly backup does not want a nightly drill; a drill costs a real target and a real restore).

Two design calls in there I'd defend anywhere:

The drill uses the ordinary restore path. Not a drill-specific one. A drill with its own code path proves the drill works, which is the one thing it was never in doubt about. It builds a restore record in new-target mode and hands it to the same restore() an operator would trigger.

A failed drill flags the policy; it never condemns the artefact. Tempting to auto-demote a backup whose drill failed. That is precisely how your last good copy disappears on the strength of a drill-environment problem. The artefact keeps its verified status. The policy goes red. There's a test pinning that, because it's the kind of rule a future refactor would "fix".

And a bonus the drill gave me for free: restore time is a number nothing else in the system ever measures. An operator planning around a backup needs to know if they're committing to ten minutes or six hours. There's no honest way to find out except doing it. So restore time is recorded from the drill's own clock, or from a real restore's — and where neither exists, the screen says "never measured". Never an estimate. An estimate here is a fabricated green tick wearing a number.

Teardown, incidentally, goes in a finally and is best-effort silent. A teardown error must not replace the real reason a drill failed. And a drill that leaves its target behind fills the server with drill databases — a backup feature causing the outage is a bad look.

The quiet ones: pagination and fakes

Two smaller things, same family.

ListObjectsV2 returns 1000 keys per call. Back up the alphabetical first thousand objects of every bucket, report success, and you have built the worst possible version of this feature — the one that is indistinguishable from the correct one until somebody needs a file starting with "z". Paginate, and pin it with a test that seeds more than a thousand.

A fake is handed out only with a reason. The resolver that picks an object-store client can fall back to a no-op in environments that have no daemon. It never does that silently:

$store = $resolver->for($destination);

if ($store->isFake()) {
    return BackupOutcome::refused($store->reason());
}

The driver turns that reason into a refusal. A silent fake records a beautifully successful backup of nothing — which is, again, the same bug: a green tick with no work behind it.

The incremental baseline has to be verified, not just complete

Objects are compared on (key, etag, size) against the last run that both completed and has a verified copy.

Seeding from a merely-completed run inherits its gaps silently and forever — every future incremental agrees with a baseline that was already wrong, and the gap only surfaces during the restore that needed it.

lastModified is deliberately excluded from the identity triple. A re-uploaded object gets a new timestamp with identical bytes, and comparing on a clock turns an incremental quietly back into a full — the opposite failure, but still the system being wrong about its own work.

The takeaway

Every one of these is the same shape. Somewhere there's a path where the system can report a success it didn't earn, and the fix is never "add a check" — it's usually to make the dishonest path impossible to express:

• delete-after-verify, in separate writes, so the violation is queryable rather than debatable;
• verification that records what it didn't do;
• a drill that shares the production restore path;
• a fake that must carry a reason;
• a baseline that must be verified, not merely finished.

If you're building anything in this space, the question I'd start with isn't "does it work?" It's "if this quietly stopped working, how long until anyone noticed, and what would the screen say in the meantime?"

If the answer to the second half is "success", that's the bug. Everything else is detail.

Next up on my side: the restore-source selector — when several verified copies exist, picking the cheapest one to restore from is its own small pile of trade-offs.