Last month I lost half a day to a workflow that was working perfectly. It just was not running.
An order-sync job on my self-hosted n8n had been s...
For further actions, you may consider blocking this person and/or reporting abuse
Your point that an error workflow needs an execution to exist before it can call it failed is the cleanest framing of this I've read. One gotcha with the heartbeat layer: if the ping is the last node after the email step, an SMTP failure stops the ping and you get paged for the wrong reason. I'd put it on its own branch right after the API reads and send the problem count in the body, like
curl -fsS -m 10 --data-raw "problems=2" "$HEARTBEAT_URL". Then the outside service shows that the check ran and what it saw. Will the heartbeat fire from n8n itself, or from host cron?That's a good catch. I hadn't considered the SMTP failure scenario causing a false alert at the heartbeat layer. Keeping the heartbeat on a separate branch after the API checks makes more sense.
In my current implementation, the monitoring logic runs separately using Python and the n8n API. The external heartbeat is something I've identified as the second layer, but I haven't tested it end to end yet.
I'm leaning towards keeping it independent of the email notification path so that an SMTP issue doesn't affect the heartbeat. Passing the problem count in the heartbeat payload is a useful addition too.
Would you recommend running that heartbeat through host cron rather than n8n itself, to keep the monitoring independent of the instance being monitored?
I've seen similar issues with cron jobs and even my own scraping agents when a dependency failed silently. For n8n, I'd build a simple external liveness check that pings a health endpoint or checks a specific DB record to ensure the workflow actually ran, then alert on its absence. FNT vProxy Studio uses a similar heartbeat system for its multi-VPN tunnels and tethered Android 4G connections, ensuring my proxy farm is always alive before I even think about routing traffic.
That's the direction I'm exploring with the external heartbeat layer. Checking whether the monitoring process itself is alive is important, but independently verifying that the actual workflow completed is what makes the difference.
I like your suggestion of checking a specific DB record as well. It could provide another signal beyond the n8n execution history.
For your proxy setup, do you use the same heartbeat mechanism to detect connectivity issues and distinguish them from an application-level failure?
The detail that the error workflow only fires on failed executions is the whole lesson. An unpublished workflow creates no execution, so there is nothing to fail and nothing to alert on. Silence looked exactly like health.
The comments make the fix stronger. AgentSearch's per-workflow cadence table covers the case where runs happen but process zero items. Amorizz's advice to put the heartbeat on its own branch matters too, because otherwise an SMTP failure late in the flow also kills the signal that would have told you about it. Keying the watch list on workflow ID instead of name, as Viet suggested, avoids the rename problem.
At Ascendra Ventures we treat this as part of handover on automation builds. The client gets a list of every scheduled job with its expected interval, and the monitor checks for missing runs, not only failed ones.
Did you settle on 3x the interval for all workflows, or do the slow weekly ones need a different rule?
Thanks for bringing all those points together. I agree that a single threshold won't necessarily work for every workflow.
In my current implementation, I'm using a configurable freshness threshold based on roughly 3x the expected execution interval. I haven't yet implemented separate cadence rules for weekly or other less frequent workflows, so that's something worth improving.
How do you usually handle weekly workflows in your handover setup? Do you allow a larger grace window, or calculate the threshold based on the expected next execution time?
Silent deadlocks in cron triggers will ruin your day faster than any explicit error. When your upstream proxies or scraper IPs drop offline and stall the queue, you need independent watchdog pings running outside the primary scheduler. For heavy multi-account workflows routing through isolated local ports, I pipe outbound requests through FNT vProxy Studio to handle rotating /64 IPv6 subnets and Android USB 4G tethering without paying per-gigabyte SaaS fees.
That's an interesting approach. I'd be interested in understanding more about your setup and how you're handling these scenarios in practice. Could you share some technical details and use cases over email at simplynadaf@gmail.com? Happy to take a closer look.
"It does nothing when an execution never starts" is the whole problem in one sentence. One thing that's worked for me alongside a liveness check: store an expected cadence per workflow (every 15 minutes, daily by 9 AM) outside n8n, so the check knows what "late" means for each one instead of using one global threshold. The other quiet case is a workflow that runs on time but its upstream source stopped returning items, which looks healthy to both an error workflow and a liveness check. Do you alert on runs that produced zero items too, or only on missing executions?
That's a really good point. I haven't implemented per-workflow expected cadence yet. In my current setup, I'm using a configurable freshness threshold based on the expected execution interval, so having an individual schedule for each workflow would definitely make the checks more accurate.
On zero-item executions, that's another scenario I haven't covered in this implementation. Right now, I'm checking whether executions happen and whether the last successful execution is within the expected window, not validating the actual output.
Curious how you handle that second case. Do you usually define a minimum expected item count for each workflow, or validate the output against some business-specific condition?
A small table of workflow id plus expected interval works well here, so the checker reads each workflow's own cadence instead of one global threshold. For zero-item runs, flag a successful execution that returned nothing when that workflow normally returns items. That catches the quiet failures that still look green.
That makes sense. Keeping the expected interval in a separate configuration table is a simple approach and avoids adding unnecessary complexity to the monitoring workflow. I also like the idea of checking zero-item executions separately. I'll keep that in mind as a next improvement. The current implementation focuses on execution liveness and failure detection, but adding output validation would make it more useful for catching silent data issues.
Great write-up. The distinction between a failed execution and a missing one is the important part.
For scheduled workflows, I’d also consider a heartbeat from the end of each watched workflow, alongside the heartbeat from the checker. That gives you an independent signal that the actual work reached its completion point. Unpublishing the workflow, stopping n8n or losing the server would all eventually show up as a missed check-in.
Disclosure: I build Doyouok, which has an n8n-verified community node for this. It can also evaluate a result condition locally, such as
processedCount > 0, and report failure even when n8n considers the execution successful—where zero is genuinely unexpected. It’s free to start and available on n8n Cloud too.Your API check adds useful context, such as whether the workflow is unpublished or its last execution failed. The external heartbeat answers a different question: did the expected completion signal arrive? I see the two as complementary.
Absolutely. I see the heartbeat and API check as complementary signals too. The API check tells us what happened inside n8n, while the external heartbeat confirms whether the expected workflow completion signal actually arrived. I also like the idea of evaluating result conditions locally, because a successful execution doesn't always mean the expected work was completed. Combining both gives us much better visibility.
the fix and the trap are the same object here. the check you built is the right shape, but it runs inside n8n, so when n8n goes down it goes silent too — your own blind-spot section says that, and i suspect it's the part most readers will skim past. the outside heartbeat is not an add-on, it's the actual fix; the internal check just makes the outside one cheap to run.
i ran into the same gap on my homelab a different way: a container that reported up but was serving stale data, and nothing was watching the watcher. i kept a small monitor that reads the machine's own state per process (github.com/SikamikanikoBG/homelab-monitor) mostly so i'd get paged when that watcher stopped reporting.
one question on the watch list: is max_quiet_minutes per workflow, or do you also want a global 'nothing ran at all' alarm?
The "check what ran, not what failed" framing is exactly right, and the 3x-interval rule for max_quiet_minutes is a sensible default.
Two gaps I've hit with this kind of watcher:
+1 on the external heartbeat being mandatory rather than optional. A watcher that lives inside the box it watches is the classic way to end up with a monitor that was dead for a week.