The Recurrence Layer — Or, "We Added a Check" Is Not the Same as "This Cannot Happen Again"

Every postmortem ends with the same sentence in a different tense: "We have added a review step so this cannot happen again."

Fourteen months later it happens again. Different service, different engineer, different quarter. The review step is still there. It was there the whole time. It was simply never the thing standing between the failure and production — it was the thing standing between the failure and being noticed.

That gap is the recurrence layer. Restitution pays for the last error. The recurrence layer is the only part of the response that says anything about the next one.

Checks are attached to paths, failures move

A check is added where the failure happened. That is also the one place the failure is least likely to happen again — because everyone who was there remembers it, the code around it was rewritten, and the incident review made that path the most-watched surface in the system.

The next occurrence arrives somewhere else: a second service that reads the same config, a batch job written before the convention existed, a partner integration that copies the behavior, a new region where the default differs. The check guards a door. The failure comes through the wall, because the wall was never part of the design.

This is why "we added a check" is not evidence of anything. The question is not whether a control exists. It is whether the class of failure — not the instance — can still produce the same class of outcome.

Name the class first. Not "the pricing page showed the wrong limit," but "a value documented in one place and enforced in another, with no mechanism that compares them." That class has many instances, most of which are not on your list. A fix attached to one instance is a fix attached to one instance.

Prevention is unfalsifiable; containment is not

"We made it impossible" is a claim about the absence of events. You cannot test for absence, so the claim can never be verified, only believed. It also has a predictable failure mode: the moment someone needs an exception, the impossibility becomes a documented override, and the override becomes the normal path within two quarters.

What can be tested is a four-part property, and it does not require anyone to promise a probability of zero:

1. Injection. Take the initiating condition — verbatim, the same values — and run it against the current system. Does it still reach production?
2. Independent detection. If it reaches, is it caught by something that is not the person who caused it, and not the same component that produced it? A check inside the failing system is self-attestation: when the system is wrong, the check agrees.
3. Bounded propagation. When it is caught, what is the maximum it can touch before it stops? "We would notice" is not a bound. A blast radius is a number: accounts, regions, euros, hours.
4. Attribution. If it fires at 3 a.m., does the alert name the control, the owner, and the class — or does it name a symptom and a dashboard?

A failure that still reaches production but is caught in minutes, bounded to one tenant, and attributed to a named control is a materially different system from one where the same failure closes the company. Neither is "impossible." Only one is defensible.

Three ways recurrence programs rot

1. The control is a ritual. "Reviewed in the weekly change meeting." Rituals are performed by people, and people are absent, overloaded, new, or measured on shipping. A control whose only enforcement is attention has a half-life measured in staff turnover. If the control has no artifact — a diff, a signed record, a test result with a timestamp — it did not run; someone remembers it running.

2. The control has an override nobody counts. Emergency bypasses are legitimate. Uncounted bypasses are how a control becomes decorative: the guard is real, the exception process is real, and after a year 40% of changes take the exception. The number that matters is not "how many were blocked" but "how many went around, by whom, and was each one reviewed afterward."

3. The control is never exercised. A check that never fires is indistinguishable from a check that is broken. Both produce zero alerts. The only way to tell them apart is to make the failure happen on purpose, on a schedule, in a place where it is safe — and confirm the guard fires, the page goes out, and the runbook step is still correct. If you have not fired the alarm since it was installed, you have an untested alarm, not an alarm.

What regression evidence actually looks like

"Tests pass" is a statement about the tests. Regression evidence is a statement about the failure:

• The initiating condition, reproduced verbatim in a harness that is not the production path.
• The guard that fired, with its name and the build it ran in.
• The artifact: log line, diff, alert ID, screenshot of the page, whatever a third party could open without asking you.
• The date, because evidence decays.
• The guard's own failure mode: if the guard is down, does the system fail open or closed, and who is paged? A guard whose outage silently disables protection is a single point of failure wearing a safety vest.

Anything short of that is a claim about intent. Intent is not a control.

Detector decay

Every detector has a shelf life. Thresholds tuned to last year's volume, dependencies that changed shape, a service that was renamed, a log field that stopped being emitted. The detector still exists, still returns success, and no longer looks at the thing it was built to look at.

The measurable version of this: expected fires versus actual fires. If history says this class of event happens roughly monthly and the detector has fired zero times in a year, one of two things is true — the class stopped occurring, or the detector stopped working. Both are worth knowing, and neither can be distinguished by looking at the dashboard, which shows green in both cases.

Put a synthetic recurrence on a schedule. Same class, safe blast radius, expected to fire. If it does not fire, the detector is the incident.

The recurrence ledger

Seven columns, one row per failure class — not per incident, not per fix.

1. Failure class (one sentence). The mechanism, not the symptom. "A limit documented in one place and enforced in another, with no comparison between them."
2. Initiating condition (verbatim). The exact configuration, input, or sequence that starts it. If you cannot write it as a value, you do not know the class yet.
3. Entry path. Where it enters: which service, which deploy, which partner, which region. A class that can enter through four doors needs four rows or an honest note.
4. Control + owner. What catches it, named, with a person. "Monitoring" is not a control; "billing-delta alert, owner @dana, fires when invoiced ≠ rated for 15 min" is.
5. Proof it fires (artifact + date). The last time it was exercised, and the thing you could show someone. No date, no proof.
6. Blast radius bound. What stops propagation and the maximum it can touch before it does. A number, not an adjective.
7. If it fires again: who is paged, max cost, review date. The cost ceiling you are willing to accept, in public, with a name attached.

The row is complete when someone who was not in the room could read it and run the test.

The honest sentence

You cannot say "this will never happen again." You can say:

If this happens again, it will be caught by this control, contained to this radius, attributed to this owner, and cost no more than this amount — and here is the date we last proved that, and here is what happens when the control itself is down.

That sentence is checkable, falsifiable, and dates itself. It is also much harder to write than "we added a review step," which is precisely why it is worth writing.

Friday version

Take the last incident. Write the failure class in one sentence, then find every other place in the system where that class could occur — services, batch jobs, partner integrations, regions. Count them. Then take the control you added and answer three questions in writing: what artifact proves it ran, what happens when it is down, and when did you last make it fire on purpose?

If the answer to the third is "never," you have installed a belief, not a control.

What comes next

Even a complete recurrence ledger makes a claim about now. It says nothing about how long the proof stays true — the test that passed against last quarter's schema, the alert whose threshold was tuned to a volume you no longer have, the runbook validated before the service was split in two.

That is the next layer: the half-life layer — every piece of evidence has an expiry date, and the only honest question is whether anyone is tracking it.

If you have a failure class you closed with "we added a check" and you are not sure the next instance would be caught, say so in the comments. We keep a one-page recurrence ledger — the failure class, the initiating condition verbatim, the path it entered by, the control and its owner, the evidence it fired, the blast-radius cap, and who gets called when it fires again — and I will send it over. No form, no email list.