I run a handful of n8n workflows unattended. A few weeks ago I noticed one had been finishing successfully for days while producing nothing at all, and n8n was perfectly content about it. That turned into a small tool and a longer education in how many ways I had been thinking about the problem wrongly.
The starting point was something about n8n's execution data that genuinely surprised me. On self-hosted 2.36.7, a node on a branch whose IF condition no longer matches is absent from resultData.runData entirely. Not present with a failed status. Not present at all. Meanwhile the execution reports status: success.
Rather than assert that, I published a four-node reproduction so anyone could check it against their own instance: a Schedule Trigger, a Code node emitting rows, an IF that can never match, and a Code node on the true branch. The true-branch node never appears in the run data. The workflow JSON is here https://gist.github.com/moneywithjjcom-del/b404b315d3444668c15da09ce2c6fbd7.
Two people in r/n8n confirmed it on their own instances within a day, and one reported it holds across 2.x rather than only the version I had tested, which is a wider claim than I could have made on my own.
Here is what I got wrong.
The first mistake was checking for bad values when the signal was a missing key. My earliest version walked runData looking for nodes with a non-success status. Given the behaviour above, that finds three healthy nodes and reports that everything is fine. The workflow is broken and the check disagrees. I think most hand-rolled monitors make this mistake, because a missing key is not what you go looking for when you sit down to think about failure.
The second was assuming a floor would catch a bad run. I added a minimum row count, which does catch zero rows. It does not catch the run that quietly returns sixty percent of normal, and that passes every non-empty assertion including the floor. A source changes shape, half the records stop matching, and everything downstream processes what it is given, correctly and pointlessly.
What works is comparing each run against a rolling median of recent runs. Median rather than mean, so one freak day does not permanently raise the bar.
The third was treating absent data as zero. n8n prunes execution data on a retention schedule, so once a workflow's history ages out, its output data is simply gone. My checks read that as "produced nothing" and alerted on everything at once, which is a good way to teach yourself to ignore your own alerts.
Absent data has to read as unknown. Silence detection via stoppedAt is unaffected, because that is metadata rather than run data, but anything counting rows is bounded by your retention window and should say so out loud.
The fourth thing I got wrong I did not find myself.
I posted about the monitor, and someone replied within the hour:
someone legitimately edits a workflow and your expected node set goes stale, suddenly you're swimming in false positives
He was right, and it was the weakest part of the whole design. The monitor learned which nodes usually run and then flagged the ones that stopped. A node the client had deliberately deleted or renamed looked identical to a node that had silently failed. Every legitimate edit would have produced a false alarm, and false alarms are how monitoring dies.
The fix needed no timestamps and no extra bookkeeping: read the workflow's currently declared node set fresh on each run, and skip anything that is no longer declared. A deleted node is an edit, not a failure. Five tests, shipped the same day, and his objection is now a test case with his name against it.
That is the part I would actually want someone to take from this. I had been looking at that code for a week. Somebody who had never seen it found the hole in under an hour, because I put the thing in public rather than describing it in public. The difference between those two turned out to matter more than any of the design decisions above.
The monitor is MIT licensed if it is useful to you, or if you want to find the fifth thing.The monitor is MIT licensed if it is useful to you, or if you want to find the fifth thing.
Top comments (0)