The Zapier failure took one second to notice. The n8n one took five minutes. The invoice-reading one I might never have noticed at all, and that gap is the entire point of this post.
I build one automation a week and publish the real numbers, including the runs that fall apart. After the first handful of builds I had a small pile of failures, and they sorted cleanly along a single axis: how long it took before I knew something was wrong. That axis turns out to be the most useful way to think about automation risk. The failures you notice instantly are annoying. The failures you notice late are expensive.
Here they are, ranked. They get quieter and scarier as you go down.
Type 1: The Wall (you notice in 1 second)
Zapier's free tier killed a Code step at a hard 1-second runtime cap. Not a timeout you can raise, a wall. The job could not finish, and there was nothing in the step to fix, because the constraint lived in the pricing tier, not in my code.
Same category, different platform: Make got stuck mid-build. I sat there on camera waiting for it to recover, and eventually abandoned the build.
The Wall is the best kind of failure. It fails loud, it fails now, and it fails before you have shipped anything to anyone. You lose an afternoon, not a customer. The fix was never a cleverer script. The fix is choosing a platform whose limits you can actually live with, and finding those limits on purpose before you depend on them.
Type 2: The Timeout (you notice in 5 minutes)
I ran a 100-run stress test and it died at exactly 5:00. Not roughly five minutes. 5:00.
When a failure lands on a round number, it is almost never your logic. It is a default. This was n8n's default 300-second task timeout. One environment variable raised it and the stress test finished clean.
The lesson is cheap and worth keeping: when something breaks at a suspiciously round time, count in seconds and go read the platform defaults before you touch your own flow. Someone picked 300 for you. You are allowed to pick a different number.
Type 3: The Confident Wrong Answer (you notice at your next audit, maybe never)
Now it gets quiet.
A vision model read an invoice date, "02/08", as February 8 instead of August 2. That is the whole bug. It looks tiny. It is not.
Here is why it is dangerous: the wrong date passed every check I had. The arithmetic reconciled. Totals matched line items. Nothing downstream threw an error, because the value the model invented was internally consistent, just attached to the wrong month. There was no error state. There was a clean, confident, wrong record sitting in the output, looking exactly like a correct one.
You do not catch this in the demo. You catch it during a quarterly audit when something does not tie out, or you never catch it and it quietly rots your data.
No smarter model saves you here, because the model was not confused. It was confident. The fix is a deterministic cross-check that lives outside the model: pull the date from a second source, enforce the format you actually expect, and reject anything ambiguous instead of guessing. The model can read the invoice. It does not get to decide what is true.
Type 4: The Flake (you notice when it costs you)
The quietest one.
I built a lead-qualifier with a gate that scored inbound leads and flagged the hot ones. One test lead claimed a team of 200 while also saying it was 3 people. Obvious garbage. The gate agreed and rejected it, most of the time.
In 2 of 5 identical reruns, same input, same prompt, the gate marked that lead HOT and passed it through.
This failure breaks every intuition you carry over from normal software. Any single run looked perfect. If I had tested it once, which is how most people test an agent, I would have shipped it and believed it worked. The bug only exists across repetitions, and in production it surfaces as the occasional expensive mistake you cannot reproduce.
You cannot debug a flake by staring at one run. You have to measure consistency: run the same input many times and count the disagreements. A gate that is right 60% of the time is not a gate. It is a coin you have not weighed.
The pattern under all five
Here is what I did not expect. Not one of these was fixed by a better model.
- The Wall: pick a platform whose limits you can live with.
- The stuck build: rehearse before you depend on it.
- The Timeout: change one config variable.
- The Confident Wrong Answer: add a deterministic cross-check outside the model.
- The Flake: measure consistency, not correctness on a single run.
Four of the five fixes are plain engineering, and the one that touches the model takes power away from it. That is the line I keep coming back to: the model votes, the code decides. The model is a fast, fluent, occasionally confident liar, and your job is to build the boring deterministic scaffolding that catches it.
Then rank your own failures by time-to-notice. The loud ones at the top are a tax you pay up front. The quiet ones at the bottom are the ones that actually cost you, because by the time you notice, the wrong data is already downstream.
The builds and the raw numbers are in the repo: https://github.com/Ships-Itself/builds
If you would rather watch these break in real time than read the postmortem, that is the whole channel: youtube.com/@shipsitself
Top comments (0)