I ran a stupid little experiment a few months ago. I built the same app twice.
The first time, the slow way — by hand, the way I used to build eve...
For further actions, you may consider blocking this person and/or reporting abuse
The detail I keep coming back to is that version two passed edge-case tests version one didn't and still shipped ack-before-persist. That isn't a contradiction, it's what that bug looks like to a test suite: the failure needs a retry or a crash to land between the ack and the save, and a test that calls the handler and checks the response never opens that gap. So the map you lost isn't only where things break, it's which breakages your tests are structurally blind to. A cheap partial substitute might be walking each write path line by line asking "what has the client been told here, and what has actually been saved?", since ordering bugs are the ones scrolling hides best.
Yeah, you put your finger on the exact thing that bugged me about it. The green test suite wasn't lying, it just wasn't being asked the right question. It checked "did the handler return the right thing" and never "did the right thing return before the save landed." The gap only exists in wall-clock time, and a unit test collapses wall-clock time to zero — so the bug is invisible by construction, not by accident.
And "which breakages your tests are structurally blind to" is better than how I put it. I was treating the map as one thing — where it breaks — when it's really two: where it breaks, and where your safety net has holes you can't see from inside the net.
Your line-by-line fix is the move. Walk each write path asking "what has the client been told at this point, and what's actually durable right now?" Ordering bugs hate that because it forces you to read in execution order instead of top-to-bottom — and scrolling only ever gives you top-to-bottom. That mismatch is exactly where the eye slides past.
Might steal "structurally blind." Good comment.
The discussion around tests being "structurally blind" hits the nail on the head, but there's a second trap people probably fall into: "we'll just catch this with E2E or negative test cases."
In reality, they almost never do.
Negative tests look for malformed payloads or missing auth, whereas
ack-before-persistships 100% valid data. And in a healthy CI pipeline, your E2E test sends the payload, gets the premature 200 OK, queries the DB a few milliseconds later—and by then, the row has already quietly landed. 100% green, 100% false confidence.The only test that actually bought the "failure map" back for me was temporal fault injection: intentionally holding the connection or killing the worker in that tiny micro-gap between the response and the durable write, and asserting that the UI doesn't prematurely celebrate. Unless you deliberately pry that 2ms gap open in tests, the AI's blind spot simply becomes your 2am fire.
Yeah, this is the trap under the trap, and the detail that makes it vicious is the timing you named: the E2E queries the DB a few milliseconds late, the row has already landed, so the test doesn't just miss the bug — it actively certifies the opposite. Green for the wrong reason is worse than no test at all, because now you've got a receipt telling you not to look there. A false negative that hands you confidence is the most expensive kind.
Temporal fault injection is the real answer, and I want to name the one catch in it, because it's the whole post biting its own tail: you can only pry open the 2ms gap you already suspect is there. The injection has to be aimed. Which means the test that would've caught ack-before-persist requires you to already know ack-before-persist is the failure mode — you have to hold the map before you can write the test that would've given you the map. Fault injection doesn't hand you the map for free. It lets you take a scar you already earned and compile it into something durable, so the next person inherits the caution without the 2am phone call. That's not a net that catches unknowns. It's the map, encoded.
Which is maybe the honest version of the whole thing: you can't test your way to a map you don't have — you can only write tests to keep a map you already earned from rotting. The AI's blind spot becomes your 2am fire in exactly the case where neither of you had the scar to aim the injection, and no amount of coverage closes that, because coverage is just answers to the questions you already thought to ask.
"Temporal fault injection" is the precise name for the move I've been hand-waving at. Stealing that one. Great comment.
Fair point that you can only aim the injection at a gap you already suspect, and I don't have a clean answer to it. The best partial one I can think of is to aim at a category instead of a line: in a test environment, delay every write acknowledgement by a random amount, without knowing which one matters, and see what falls over. That is still a guess, just a much broader one than a single 2ms window. So I'd mostly agree with your framing, tests keep an earned map from rotting, with a little room left for cheap, unaimed poking at the boundaries.
"Tests keep an earned map from rotting" — I'm stealing that, it's the whole argument in six words. And I think your category-not-line move is the honest answer to the problem you raised. You can't aim injection at a 2ms window you haven't imagined yet, but you can say "every write ack in this environment gets a random delay" and let the thing that falls over tell you where you weren't looking. That's not cheating the unknown-unknown, it's widening the net until it catches shapes you didn't name. The fact that you won't pretend it's a clean answer is exactly why I trust it more than one that claims to be.
Interesting framing: trust as a map of where things break. I ran a similar experiment from the other side: built a full app (backend, web, PWA, Chrome extension) without ever reading the code, and had to get the "map" some other way.
For me it came from design instead of typing: written module boundaries (a short note per module on what it owns and must not touch), standards in files, and constantly using the product like a user. It is not proof of correctness, and bugs I found along the way went into a known-issues file rather than being fixed on the spot. But it did make me trust the result more than I expected.
Wrote it up here as one more data point: dev.to/nomad4tech/i-built-a-time-t...
This is the version of the argument I was hoping someone would make, because it's the part I left implicit. If the map is the real deliverable, then typing was only ever one way to produce it — and you found another. Written module boundaries, "what it owns and must not touch," standards in files: that's the map, just authored up front instead of accumulated line by line. You paid for it deliberately, which is the whole point I was fumbling toward.
The known-issues file is the detail I like most. You resisted the reflex to fix on the spot, and that's exactly the discipline — a fix closes the ticket but doesn't necessarily enter the map, whereas writing it down is the map getting drawn. Most people would've patched and forgotten, and the knowledge would've evaporated the same way version two's did for me.
The one place I'd poke: boundaries and standards tell you where things are supposed to break — the planned seams. My scar stuff is about the breaks nobody designed, the ack-before-persist that looks fine against every rule you wrote down. "Constantly using the product like a user" is probably what catches those for you, the way the 2am phone call caught it for me. Curious whether the bugs in your known-issues file came more from the design notes or from just living in the thing.
Reading your write-up now. Good to see it from the other side.
The ack-before-persist example nails it, clean idiomatic code with green tests that only blows up on a mis-timed retry. Your framing that trust is the mental map of where it breaks, not a property of the code, is what I keep failing to explain about AI-generated code: when you didn't type it, you never built that map. That reconstruction is exactly what tracing and evals end up doing after the fact.
Yeah, that's exactly the part that stings — the "after the fact." Tracing and evals are me buying the map back at retail, after I've already shipped the thing that should've come with one.
And they're not a clean substitute, which is what I keep running into. When I typed the code, the map got built at the one moment I could still change my mind about the line. Tracing builds it after the line is already live. So evals catch the break I thought to write a case for — but the ack-before-persist bug never had a test, because the whole failure was that I never paused on it in the first place. You can't write an eval for a failure mode you never imagined. That's the gap I haven't figured out how to close yet — the scar fires on lines you wrote, not lines you review.
Glad that example landed, anyway. The phone call was real, and I still don't fully trust that write path in either version.