The break never happens at the demo
The demo always works. Stakes are zero, the input is clean, one user, one pass down the happy path. The break arrives after launch — when the export has a blank row, when two people use the tool at once, when the number on screen has to agree with the number on the invoice.
Nate B Jones gave the pattern a name in a video this month that hit with a lot of people: vibe-coded apps can "look right and fall apart after launch." The screen was convincing. The plumbing under it was never built — because it was never in the conversation.
The instinctive fix — go back to the chat and prompt harder — fails for a structural reason. The missing pieces weren't omitted by the model; they were omitted by the description. Error handling, state that survives a restart, what happens when the input is wrong — production software carries a layer of behavior nobody asks for, because nobody demos it. As the write-up that accompanied the video put it: a convincing screen tells you very little about what should happen next, what needs to be remembered, or which numbers have to agree.
The new lever: your agent can look inside real software
What changed this month is that the missing reference material is now reachable. REA — "Reverse Engineer Anything," an open-source project that has passed 200,000 GitHub stars — gives coding agents like Claude Code and OpenAI Codex a set of tools to inspect installed software and explain how a feature actually works. The flagship example from the REA team's published work: a developer pointed the agent at QuickBooks Desktop, investigated how the real thing handles transactions and reports, and then built a local accounting app — Ledger Reborn — with its own transaction engine.
The point of that story isn't "clone QuickBooks." The point is that the investigation turned years of taken-for-granted behavior into a visible list of decisions: which records to keep, how a result is calculated, what happens when something changes. That's exactly the layer your AI-built app is missing — and now there's a repeatable way to go look at it.
Here's the drill for the day the break arrives.
The 30-minute post-mortem
Step 1 — Reproduce the break (5 minutes)
Write down the exact steps that trigger it: which screen, which input, what you expected, what happened instead. Not "the invoicing tool is broken" — the specific row of input that breaks it. If it can't be reproduced, it can't be fixed, and there is nothing precise to hand an agent. The reproduction is your spec for the repair and your acceptance test after it.
Step 2 — Diff against a real one (10 minutes)
Ask the agent what an app that does this job for a living has that yours doesn't. If you run Claude Code or Codex, REA can point the agent at a real installed program and ask how that feature survives the situation that broke yours. If that's past your line, say so plainly in the chat: "Here is the broken path. What does production software typically have on this path that a generated app might be missing?" The answer is usually a short list: input validation, error handling instead of a crash, state that persists, defined behavior for empty or duplicated input. The goal is learning what production apps carry, not copying one — behavior ideas are fair game; lifting proprietary code or assets is not.
Step 3 — Rebuild the one path (10 minutes)
Not the whole app. The short list from step 2 becomes a mini-spec for the single path that broke: what goes in, what comes out, what happens when the input is wrong. Rebuild that path, then run the exact reproduction from step 1 as the acceptance test. One path, one fix, one test — a whole-app rewrite in this moment is how a 30-minute post-mortem becomes a bad weekend.
Step 4 — Make the keep-or-retire call (5 minutes)
This is the step everyone skips, and it's the entire point. If the fix holds and the failure was one path: keep the app, and write down the fix while it's fresh. But if this is the third week in a row the app breaks somewhere new, that's not bad luck — that's a tool doing production work it was never scoped for. The honest options are retire (buy the boring, supported tool that absorbs this class of failure) or promote (write the real spec — inputs, outputs, edge cases, rollback — and rebuild deliberately, treating the first version as the prototype it always was).
The honest boundaries
- The deep inspection is a developer tool. If you don't already run Claude Code or Codex, step 2's full version is a one-afternoon job for whoever built the app or a freelancer, not a DIY evening. The post-mortem structure still works with the plain conversation version — you lose depth, not the drill.
- One repair is data; two is a pattern. Don't let the keep-or-retire question drift just because the fix worked tonight. A tool that needs a post-mortem every week is a subscription you're paying in hours.
- The post-mortem is the floor, not the ceiling. It exists so a single break costs you 30 minutes instead of a rebuild. It is not a substitute for writing the spec before the next build — the cheapest post-mortem is the one you never need.
Tonight's version
Pick the broken thing — you know the one. Write the reproduction steps. Run the diff conversation, even if it's the plain-chat version. Rebuild one path and test it against the reproduction. Then make the call out loud: keep, or retire.
If the app survives the night, write down what it does and who owns it — so the next post-mortem isn't the only documentation the tool has.
Want the frameworks that prevent this repair loop? The Boring Automation Pack includes the spec templates, acceptance-test prompts, and guardrail checklists that keep AI-built tools useful without becoming a liability.
Top comments (0)