How do you actually test for a failure that leaves no trace?

Code and results: github.com/ashwin-sridhar/silent-failure-harness

There's a specific kind of bug I have a hard time explaining to people who haven't hit it. Not a crash. Not a stack trace. A test suite that's green, a system that looks "up," and a business event that just — isn't there. Nobody's system logged an error, because from its point of view nothing happened. That's not a metaphor. That's the literal failure mode.

I wanted to know whether the pattern I'd designed around this was actually safe, or just safe-sounding. So I tried to break it on purpose.

What "silent failure" actually means

At-least-once delivery plus a crash at exactly the wrong instant produces a state where the sender believes the event was delivered and the receiver has no record it ever arrived — and both of them are technically correct, from where they're standing. There's no error anywhere in that story. That's what makes it hard to test for: you're not looking for an exception, you're looking for an absence.

The setup

Two variants of a minimal webhook handler, and a way to kill the process on demand at an exact, repeatable point:

Variant A — ack-then-persist
  1. Receive request
  2. Return 200
  3. Write event to the database

Variant B — persist-then-ack
  1. Receive request
  2. Write event to the database (INSERT ... ON CONFLICT DO NOTHING)
  3. Return 200

The crash is a SIGKILL sent to the process at a pinned point right after step 2 in each variant — not "at some point during the request," but deterministically after the ack or after the write, using an env flag. No graceful shutdown. The same kind of failure a host reboot or an OOM kill actually produces.

For each trial: send the event, let the process kill itself, restart it clean, resend the identical event ID — exactly what a provider does when it never gets a 200. Then check the actual source of truth: how many rows exist for that event ID. Not the logs. The row count.

Finding 1: ack-ordering is not a "usually" — it's unconditional

50 trials per variant.

variant	lost	correct	duplicate	/ 50
ack-then-persist	50	0	0	50
persist-then-ack	0	50	0	50

Ack-then-persist lost the event on every single trial. The 200 went out, the process died before the insert ran, and there was no retry to save it — the provider saw a successful delivery and had no reason to send it again. Fifty for fifty, not a tendency.

Persist-then-ack didn't lose one. One trial out of fifty is worth noting honestly rather than ignoring: the crash request itself returned a 200 anyway, meaning the kill didn't land in time to suppress the ack that trial — a timing artifact of the harness, not a case where the retry path saved a loss. It still counted as correct, since the write had already committed either way, but it's not evidence of the same mechanism as the other 49.

Finding 2: the race I expected to see wasn't the race that was actually there

I set up a second test for the dedup mechanism itself — ten concurrent requests, same event ID, fired at once, comparing a naive check-then-insert against an atomic INSERT ... ON CONFLICT DO NOTHING. I expected check-then-insert to sometimes let a duplicate row through.

It never did. Twenty trials, ten concurrent requests each, for both mechanisms:

dedup	exactly 1 row	2+ rows	/ 20
check-then-insert	20	0	20
atomic-conflict	20	0	20

Zero duplicate rows, either way — but that's not the dedup logic being safe. That's event_id TEXT UNIQUE at the schema level doing its job regardless of what the application code above it does. A second insert physically cannot land, no matter how the race plays out.

The race still happened. It just didn't show up as a row — it showed up as an error. Across 200 individual concurrent requests against check-then-insert (20 trials × 10), 181 came back 200 and 19 came back 500. Two goroutines both ran the SELECT, both saw no existing row, both proceeded to INSERT — one won, one hit the unique constraint it never checked for and blew up. Against atomic-conflict, all 200 requests came back 200. Same exact race, same ten-way collision — ON CONFLICT DO NOTHING just absorbed it.

What this actually means for you

1. Persist before you ack. Not usually — always. Fifty trials, zero exceptions, in either direction.
2. Put a real UNIQUE constraint on your dedup key at the schema level, regardless of your application logic. It's the thing that actually stopped a duplicate row from ever landing here, independent of whether the code above it was written carefully.
3. Prefer atomic INSERT ... ON CONFLICT DO NOTHING over check-then-insert — not because it prevents data corruption, the constraint already does that — but because it decides what happens to the losing side of a race. Check-then-insert hands the loser a raw 500. From the sender's side, that looks like a failed delivery, which means another retry, which means another race. Atomic conflict resolves it cleanly on the first attempt.

The part where I admit something

I built this test expecting to catch a data-integrity bug — duplicate rows from a sloppy dedup implementation. The schema-level constraint made that outcome structurally impossible before either mechanism got a chance to prove anything. What the test actually caught was a behavioral difference, not a data one: whether the loser of an inevitable race gets an idempotent-feeling 200 or an ugly exception. That's a more useful thing to know than what I went in looking for, and I wouldn't have found it without running the concurrent version instead of trusting that "no duplicate rows" meant "no race."

The constraint saved the data. It didn't save the response.

Full harness and raw results if you want to run it yourself: github.com/ashwin-sridhar/silent-failure-harness