Your Audit Log Agrees With Itself. That's the Problem.

@remdore ran an experiment this week that I have not stopped thinking about.

One account, balance 100. Ten workers, each withdrawing 10. Every worker does the obvious thing: read the balance, subtract, write it back, log what happened.

Final balance: 90. Nine withdrawals gone.

That part is a race condition and every backend engineer has met it. The part I want to talk about is the audit log.

Ten rows. Every one of them says the balance went from 100 to 90. No gaps. No nulls. No orphaned foreign keys. Nothing a schema validator, a checksum, or a nightly reconciliation job would flag.

Hand me that table during an incident and I would read it as one withdrawal retried ten times. I would go and look at the retry logic. The retry logic is fine. I would spend the afternoon there.

The log is not wrong. That's what makes it dangerous.

We are in regulated payments. The audit log is not a debugging convenience for us. It is the artefact. It is what an auditor reads, what a dispute resolves against, what we would put in front of the FCA if someone asked how a payment reached the state it did.

So the standard we hold it to is not "does it look sensible". It is "is this the record of what happened".

And the failure in that experiment passes every test I know how to write against a log.

Think about what you can actually check automatically:

• Schema conformance. Passes.
• No nulls in required columns. Passes.
• Referential integrity to the account. Passes.
• Monotonic timestamps. Passes.
• Balance arithmetic internally consistent per row. Passes, ten times.
• Row count matches the number of operations attempted. Passes. Ten operations, ten rows.

Every one of those questions is about whether the log is coherent. None of them is about whether the log is true. Ten processes each believed they were alone, and each one wrote down a faithful account of the world as it saw it. Coherence is exactly what you get when ten honest narrators all have the same wrong information.

The thing you would need to detect it is a row that disagrees with another row. There isn't one. Disagreement is the signal, and the bug removed the signal.

This is the same failure I wrote about two days ago

I published something on Monday about guardrails nobody checks the liveness of. A lint rule that stopped running, a policy check that matches nothing, an eval that skips half its corpus and reports the half it ran. The thread on it got long, and the argument that came out of it was sharper than the article.

The shape is the same here.

A check that never fires and a log that never contradicts itself both produce the same reading: nothing is wrong. In both cases the absence of a complaint is being taken as evidence of health, and in both cases the mechanism that would produce the complaint is the thing that broke.

You cannot fix that by inspecting the output more carefully. The output is clean. That is the whole point.

The one-statement version, and why it matters more than it looks

Postgres 18 lets you name the row before and after a change in the same statement that makes it:

UPDATE accounts SET balance = balance - 10 WHERE id = 1
RETURNING old.balance AS was, new.balance AS now;

The syntax is nice. It is not the interesting part.

The interesting part is that the log entry is now derived from the write instead of being a second write that describes the first. Before this, the audit row was a separate statement carrying a separate copy of the truth, and two copies of the truth can drift. After it, there is one statement, and the record and the change are the same event. There is no version of the run where the balance moves and the log says something else, because there is no seam between them for a lie to get into.

That is the general principle and it is worth more than the syntax. Any time the record and the thing being recorded are two separate writes, you have two versions of the truth, and only one of them is the one that moves money.

You can apply that without Postgres 18. The change is to stop thinking of the audit log as something you write about an operation and start treating it as something the operation emits. If a code path can complete without producing its record, the record is optional, and optional loses to deadline pressure every time.

Where this stops working

I want to be honest about the limit, because the single-statement version solves a single-database problem and most of us do not have a single-database problem.

The moment the operation crosses a service boundary, the seam is back. The money moves in one place and the record is written in another, and now you are relying on an outbox, or a queue with delivery guarantees you have read about but not tested, or in the worst case a developer remembering to call the logging function. Convention again.

The only thing I have found that helps at that scale is narrowing the number of routes. Anything touching money goes through one path with the record emitted inside it, so there is no version of the call that reaches the effect without producing the evidence. Everything outside that path is convention, and I am not comfortable about it, but I would rather have one path I can defend than six I have to trust.

The question I would actually like answered

Here is what I do not have a good answer for.

Every detection method I know for this class of bug is a disagreement detector. Two sources that should match and don't. A count that should equal another count. A hash that should still verify. All of them need two things to compare.

The read-modify-write bug produces one thing, ten times. There is nothing to compare it against, because the second source was never independent.

So: has anyone got a check that catches a log which is internally consistent and collectively false, without needing an independent second record to diff against? Not a stronger schema. Not more constraints on the row. Something that notices that ten narrators agreeing is itself suspicious.

I think the honest answer might be that there isn't one, and the only real defence is making the seam impossible rather than detecting what crawls through it. But I would like to be wrong about that, because "make the seam impossible" is expensive and does not retrofit.

---

Credit where it is due: the experiment and the Postgres 18 walkthrough are @remdore's. Go and read the original. I have only taken the part that scared me and pointed it at my own industry.