DEV Community

Cover image for I Audited 50,669 n8n Runs. 85% of Failures Alerted Nobody.
אחיה כהן
אחיה כהן

Posted on

I Audited 50,669 n8n Runs. 85% of Failures Alerted Nobody.

One of my own workflows — the gate that decides whether a new WhatsApp contact gets the bot or gets me — failed 24 times between Thursday and Sunday. Seventy-one hours. I found out on Monday morning, by opening the executions list for an unrelated reason.

That is embarrassing for someone who sells automation. So instead of fixing the one workflow and moving on, I went and measured the actual state of error handling across all 225 workflows on my main production instance. Here is every execution my instance still retains — pruning is on, so this is a six-day window, 18–24 August:

Metric Value
Executions (6 days) 50,669
Failed executions (error + crashed) 106
Failure rate 0.21%
Active workflows 60
Active workflows with an error workflow attached 26 (43%)
Failures that occurred in a workflow with no error handling 90 of 106 (85%)

n8n 2.35.5, queue mode, Postgres. A 0.21% failure rate is fine. The 85% is not.

Why silence is the default

Error handling in n8n is opt-in, per workflow. You build one workflow whose trigger is the Error Trigger node, then in every other workflow you open Settings → Error Workflow and point it at that one. There is no switch that applies it to everything you have already built, or to everything you build next.

That is a per-workflow checkbox on a list that grows every week. Nobody forgets on purpose. You just build workflow #61 at 11pm, it works, you activate it, and you move on. The failure mode of an opt-in safety net is that coverage decays quietly while the number in your head stays at "yeah, we have error handling."

Mine had decayed to 43%.

What the gap actually cost

I pulled the failures grouped by workflow, with a column for whether that workflow had an error handler attached:

Workflow Guarded Failures Window
WhatsApp group lead detection (AI) no 34 36.5 h
Personal-line bot gate no 24 70.9 h
Client error-handling workflow no 13 1.2 h
Campaign-failure watchdog no 11 8.7 h
Website lead intake form no 4 53.0 h
SMS dispatcher (outbox) yes 8 1.2 h
Appointment sync + no-show recovery yes 4 1.1 h
6 others mixed 1–2 each ~0 h

Read the last column, not the failure counts. Every window longer than nine hours belongs to an unguarded workflow. The longest is 70.9 hours — a WhatsApp bot gate that failed 24 times over three days while I was working on other things. Nothing was on fire. No customer wrote in. The bot simply did not answer new contacts, and the only reason I found it is that I went looking.

The guarded workflows failed too — 16 times between them. Their longest window was 1.2 hours, because something told me.

That is the whole value proposition of the Error Trigger, and it is not "fewer failures." It is failures that end in hours instead of days.

The part that actually scared me

Three of my workflows contain an Error Trigger node. Those are the handlers — the things that send the Telegram alert. Here is how many active workflows depend on each one:

Handler Workflows it guards Guarded itself?
Client error handler 15 no
Generic error alert 6 no
Product owner alert 5 no

Every handler was unguarded. And the one covering 15 workflows is the same row from the table above: it failed 13 times inside that same six-day window.

For those 13 failures, 15 workflows had no alerting at all and no way to find out. The error handler cannot report its own errors, because the thing that reports errors is the error handler. Quis custodiet ipsos custodes, in YAML.

You cannot fix this by pointing handler A at handler B, either. That just moves the single point of silence one hop and adds a cycle you will forget about. The catcher has to be watched from outside n8n.

The pattern, in four parts

1. One handler, not one per project. Fan-in beats fan-out. A single Error Trigger workflow that formats {{ $json.workflow.name }}, {{ $json.execution.id }} and {{ $json.execution.error.message }} into one alert is easier to keep correct than six near-copies.

2. Audit coverage with a query, not with your memory. This is the whole audit — run it against your n8n Postgres:

SELECT w.name,
       (w.settings->>'errorWorkflow' IS NOT NULL
        AND w.settings->>'errorWorkflow' <> '') AS guarded
FROM workflow_entity w
WHERE w.active = true
ORDER BY guarded, w.name;
Enter fullscreen mode Exit fullscreen mode

Everything with guarded = f is a workflow that can fail into the void. If you are on n8n Cloud, the same check runs off GET /api/v1/workflows?active=true and reading settings.errorWorkflow per item.

That query is now the first line of my weekly runbook, next to the backup check. A new f row gets treated the way a failing test does.

3. Make the handler loud about workflows, not just about errors. The first thing I changed in mine was putting the workflow name in the subject line instead of in the body. That sounds trivial. It is the difference between "an automation failed" (which you snooze) and "the lead intake failed" (which you do not).

4. Guard the guard from outside. The Error Trigger has a blind spot that no amount of coverage fixes: it fires when an execution fails. It does not fire when an execution never starts — a deactivated trigger, a dead cron, a webhook whose URL changed, a worker that is not consuming the queue. A workflow that stopped running entirely produces zero failed executions and therefore zero alerts. It looks exactly like a quiet week.

So the handler pings a dead man's switch on every run, and a plain cron on a different machine screams if the ping stops:

# in the error handler workflow, and in one heartbeat workflow per instance
curl -fsS -m 10 "https://your-monitor/ping/$SLUG" || true

# elsewhere — not on the n8n box
LAST=$(stat -f %m /var/lib/heartbeat/n8n1 2>/dev/null || echo 0)
if [ $(( $(date +%s) - LAST )) -gt 3600 ]; then
  notify "n8n1 heartbeat is $(( ($(date +%s) - LAST) / 60 ))m stale"
fi
Enter fullscreen mode Exit fullscreen mode

The || true matters: a monitoring call that can fail the run it is monitoring is worse than no monitoring.

That is it. Four parts, none clever. The reason it is worth writing down is that I had parts 1 and 3 for as long as I have run this instance and still ate a three-day outage, because I never had part 2 and never imagined I needed part 4.

Where I landed

I am not going to pretend this is finished. As I write this, coverage is still 43% — what changed is that I now have the number, the query that produces it, and a list of 34 workflows in the order I am going to fix them, longest silent window first. The handler-watching-the-handler problem is the part I am fixing today, because it is one workflow and it covers fifteen.

The honest reason I am writing this before finishing it: the audit took twenty minutes and the three-day outage took three days. Those twenty minutes are the cheapest thing in the entire stack I maintain — cheaper than the WhatsApp bots I run in production for clients, and a rounding error against the numbers in this breakdown of what business automation actually costs. Run the query before you finish reading this post and you will probably learn something uncomfortable too.

One genuine question, because I do not think I have solved this part: how do you monitor the trigger that never fires? A heartbeat proves the instance is alive and proves that workflow ran. It does not prove that a webhook from a third party still arrives, or that a specific cron survived the last upgrade. Per-workflow "expected minimum run rate" alerting is the obvious answer and it also sounds like a maintenance burden that decays exactly like error-workflow coverage did.

If you run n8n in production: do you alert on absence of runs, and if so — how do you keep the thresholds from rotting?

Top comments (1)

Collapse
 
max_quimby profile image
Max Quimby

"Coverage decays quietly while the number in your head stays at 'yeah, we have error handling'" is the whole post in one sentence — and it's true of every opt-in safety net, not just n8n's Error Workflow. We run a fleet of scheduled agent tasks and hit the identical decay: the count in your head is the count from the last time you checked, and nobody re-checks.

The structural fix that worked for us was to stop treating "is this guarded?" as a per-workflow property you remember and start treating it as a query you run on a schedule. You already wrote it — the SQL that found the gap is the monitor. Invert it: a tiny meta-workflow that lists active workflows with no error handler attached and pings you weekly. Opt-in coverage you have to remember will always decay to whatever % you last enforced; a periodic audit that alerts on the gap turns "remember to add it" into "get told when it's missing."

The detail that'll haunt people is your error handler itself failing silently — the classic "who watches the watchman." Did you end up with an external dead-man's-switch (heartbeat that alerts on absence) so the monitor's own death is detectable?