DEV Community: Ships Itself

5 automation failures, ranked by how long until you notice

Ships Itself — Wed, 12 Aug 2026 12:31:22 +0000

The Zapier failure took one second to notice. The n8n one took five minutes. The invoice-reading one I might never have noticed at all, and that gap is the entire point of this post.

I build one automation a week and publish the real numbers, including the runs that fall apart. After the first handful of builds I had a small pile of failures, and they sorted cleanly along a single axis: how long it took before I knew something was wrong. That axis turns out to be the most useful way to think about automation risk. The failures you notice instantly are annoying. The failures you notice late are expensive.

Here they are, ranked. They get quieter and scarier as you go down.

Type 1: The Wall (you notice in 1 second)

Zapier's free tier killed a Code step at a hard 1-second runtime cap. Not a timeout you can raise, a wall. The job could not finish, and there was nothing in the step to fix, because the constraint lived in the pricing tier, not in my code.

Same category, different platform: Make got stuck mid-build. I sat there on camera waiting for it to recover, and eventually abandoned the build.

The Wall is the best kind of failure. It fails loud, it fails now, and it fails before you have shipped anything to anyone. You lose an afternoon, not a customer. The fix was never a cleverer script. The fix is choosing a platform whose limits you can actually live with, and finding those limits on purpose before you depend on them.

Type 2: The Timeout (you notice in 5 minutes)

I ran a 100-run stress test and it died at exactly 5:00. Not roughly five minutes. 5:00.

When a failure lands on a round number, it is almost never your logic. It is a default. This was n8n's default 300-second task timeout. One environment variable raised it and the stress test finished clean.

The lesson is cheap and worth keeping: when something breaks at a suspiciously round time, count in seconds and go read the platform defaults before you touch your own flow. Someone picked 300 for you. You are allowed to pick a different number.

Type 3: The Confident Wrong Answer (you notice at your next audit, maybe never)

Now it gets quiet.

A vision model read an invoice date, "02/08", as February 8 instead of August 2. That is the whole bug. It looks tiny. It is not.

Here is why it is dangerous: the wrong date passed every check I had. The arithmetic reconciled. Totals matched line items. Nothing downstream threw an error, because the value the model invented was internally consistent, just attached to the wrong month. There was no error state. There was a clean, confident, wrong record sitting in the output, looking exactly like a correct one.

You do not catch this in the demo. You catch it during a quarterly audit when something does not tie out, or you never catch it and it quietly rots your data.

No smarter model saves you here, because the model was not confused. It was confident. The fix is a deterministic cross-check that lives outside the model: pull the date from a second source, enforce the format you actually expect, and reject anything ambiguous instead of guessing. The model can read the invoice. It does not get to decide what is true.

Type 4: The Flake (you notice when it costs you)

The quietest one.

I built a lead-qualifier with a gate that scored inbound leads and flagged the hot ones. One test lead claimed a team of 200 while also saying it was 3 people. Obvious garbage. The gate agreed and rejected it, most of the time.

In 2 of 5 identical reruns, same input, same prompt, the gate marked that lead HOT and passed it through.

This failure breaks every intuition you carry over from normal software. Any single run looked perfect. If I had tested it once, which is how most people test an agent, I would have shipped it and believed it worked. The bug only exists across repetitions, and in production it surfaces as the occasional expensive mistake you cannot reproduce.

You cannot debug a flake by staring at one run. You have to measure consistency: run the same input many times and count the disagreements. A gate that is right 60% of the time is not a gate. It is a coin you have not weighed.

The pattern under all five

Here is what I did not expect. Not one of these was fixed by a better model.

The Wall: pick a platform whose limits you can live with.
The stuck build: rehearse before you depend on it.
The Timeout: change one config variable.
The Confident Wrong Answer: add a deterministic cross-check outside the model.
The Flake: measure consistency, not correctness on a single run.

Four of the five fixes are plain engineering, and the one that touches the model takes power away from it. That is the line I keep coming back to: the model votes, the code decides. The model is a fast, fluent, occasionally confident liar, and your job is to build the boring deterministic scaffolding that catches it.

Then rank your own failures by time-to-notice. The loud ones at the top are a tax you pay up front. The quiet ones at the bottom are the ones that actually cost you, because by the time you notice, the wrong data is already downstream.

The builds and the raw numbers are in the repo: https://github.com/Ships-Itself/builds

If you would rather watch these break in real time than read the postmortem, that is the whole channel: youtube.com/@shipsitself

I ran my AI agent 1,200 times. The bill was $1.20.

Ships Itself — Wed, 12 Aug 2026 12:25:23 +0000

Everyone selling you an AI agent shows you the demo. Almost nobody shows you the bill. So I took a support agent I'd built in n8n and ran it 100 times in a row — the same 12-ticket inbox, over and over — to find out what an agent actually costs when it's doing real work.

That's 1,200 real model calls. Here's what the meter said, and the two things the exercise taught me that the pricing page never will.

The setup

The agent is three code nodes in self-hosted n8n:

Load the tickets and a small knowledge base.
Answer or escalate — the model may only answer from the knowledge base, and it must cite which entry it used. No citation, no answer: the ticket escalates to a human. (This is the rule that keeps it from being a hallucination hose.)
Gate + receipts — deterministic code decides whether the model's answer counts, and writes answered.json / escalated.json / summary.json.

To measure cost, I wrapped the whole inbox in a loop and ran it 100 times. Every model call goes to meta-llama/llama-4-scout through fal, which bills a flat rate per request, and every response carries an x-fal-billable-units header you can check yourself.

The bill

1,200 calls (100 runs × 12 tickets)
fal's public rate: $0.001 per request, flat
Total: $1.20

Per ticket, that's a tenth of a cent. Scale it out: a thousand tickets a month is about $1 in model calls, plus a ~$4 box to run n8n on. Call it five dollars a month, all in.

For comparison, Intercom's Fin — the market leader — charges $0.99 per resolution (their public price). Fin is a whole platform: inbox, routing, analytics, compliance. This build is none of that. But the raw model work inside that 99 cents? A tenth of a cent. The other 98.9 cents is packaging. Sometimes worth it — now you know what it's for.

The part nobody measures: consistency

Cheap is worthless if it's random. So I ran the same 12 tickets through all 100 passes and asked a different question: did the agent make the same decisions every time?

98 of 100 runs came back identical: 9 answered, 3 escalated, the same tickets every time.
The 2 runs that differed? The agent got more careful, not less — it escalated one extra ticket instead of answering it.

That's the number I actually care about before trusting an agent with customers. A model that's cheap but non-deterministic in a decision path is its own kind of outage. The citation gate is what holds it steady: the model votes, and deterministic code decides whether the vote counts.

The honest part

Two things this bill leaves out, and one real failure:

It excludes your build time, keeping the knowledge base current, and connecting a real helpdesk (that's OAuth and webhooks I didn't pay for here).
Short tickets are cheap. Longer conversations mean more tokens, so token-priced providers will cost more than this flat-per-request math suggests.
My first run died at exactly 5:00. n8n ships a default task timeout of 300 seconds, and 1,200 sequential calls blew past it. One environment variable (N8N_RUNNERS_TASK_TIMEOUT) fixed it. Round-number failures are almost always a config default, not your logic.

The math is the part that lasts: count your calls, multiply by your provider's public rate, and read your own bill. Don't estimate it — measure it.

Try it yourself

The whole workflow, the demo tickets, and the run receipts are here:
👉 https://github.com/Ships-Itself/builds/tree/main/ep05-cost-teardown

Import the workflow, point EP05_DIR and FAL_KEY at your setup, and hit execute. It writes its own receipts so you can check my numbers against yours.

I build one of these on camera each week and publish the real numbers — including the failures. If that's your thing, the video version of this teardown is on the Ships Itself channel.