I run a benchmark harness that measures how often automation platforms silently drop webhook events. The self-hosted part of it lives on a Google Cloud e2-micro: 1 GB RAM, 0.25 vCPU baseline, 2 GB swap, free tier, $0/month. It handles about 108 events a day, roughly 4.5 an hour, and has been up since July.
Someone pointed out that the burst test I had published (10 events at once) says nothing about sustained load, which is true and is the better question. So I held one event every 5 seconds against it for 4 hours and 42 minutes. 2,880 events. 136x its normal rate, continuously, while sampling host telemetry every 15 minutes.
Scale caveat before anything else, because 136x sounds more impressive than it is: 0.17 events/second is a long duration at a modest rate. This tests whether a small box degrades or leaks over hours. It is not a throughput benchmark and tells you nothing about 50/second.
Nothing broke
fired 2,880
accepted 2,876
refused at send 4 (loud, client errored)
accepted then lost 0 (silent)
Zero lost in five hours. No memory leak, no progressive slowdown, no queue backing up. That's the boring headline and it's the one I expected.
It got faster, which I did not expect
Same workflow, same box, compared against normal operation:
median p95 p99
normal, last 7d (n=260) 782 ms 10,577 ms 16,937 ms
normal, last 30d (n=1120) 822 ms 10,322 ms 25,517 ms
under soak (n=2876) 610 ms 1,673 ms 5,392 ms
The tail improved 6.3x while load went up 136x, and it holds against 7, 14 and 30 day baselines, so it isn't a stale-comparison artifact.
My explanation is cold path: at 4.5 events an hour the instance is idle essentially always, and every arriving request has to wake something up. Under a 5-second cadence nothing gets a chance to go cold.
I'm labelling that a hypothesis, not a finding. The experiment that would confirm it is trivial (fire one event after six hours idle, compare) and I haven't run it. What I'm claiming is the table, not the reason.
If it does hold, it's a mildly uncomfortable thought for capacity planning on low-traffic self-hosted services: the ugly p99 you see in production may be an idleness cost rather than a capacity cost, and adding load would improve it.
Then, at 06:30, a 56-second request
Thirty-minute buckets across the run:
bucket fired refused undeliv p50 p95 worst
04:00 253 0 0 521 1728 32,831
04:30 319 0 0 581 796 4,032
05:00 308 4 0 608 782 13,100
05:30 319 0 0 579 783 2,445
06:00 314 0 0 590 2174 5,617
06:30 273 0 0 715 4727 56,726 <--
07:00 312 0 0 658 1595 4,778
07:30 317 0 0 612 818 2,985
08:00 289 0 0 680 2363 36,194
08:30 176 0 0 536 965 7,669
56.7 seconds. 93x the median for that bucket.
My first hypothesis was an OOM restart, because container memory had dropped from 252 MiB to 88 MiB in a single sample, below where it started the run.
That hypothesis was wrong:
n8n | Up 7 weeks
restarts=0 started=2026-07-12 oomkilled=false exit=0
host up 54 days, no OOM in dmesg
The container never restarted. The memory reading fell because the kernel reclaimed page cache, which docker stats counts as container memory.
The journal had the real answer:
06:04:38 Starting apt-news.service - Update APT News...
06:04:39 Starting packagekit.service - PackageKit Daemon...
06:49:49 packagekit.service: Deactivated successfully.
The machine's own package-metadata refresh, running for 45 minutes. sar shows what it did to the disk:
time %user %iowait %idle dirty pages (kB)
06:10 1.65 2.66 95.05 204
06:20 1.46 2.03 95.93 388
06:30 4.09 20.20 71.54 41,316
06:40 4.04 34.24 57.02 524
06:50 1.66 29.12 68.12 336
07:00 1.61 3.64 93.88 664
I/O wait went from 2.7% to 34.2% for three quarters of an hour on a shared-core VM with a slow persistent disk, and the request tail went with it.
My 136x load did not drop a single event in five hours. The box's package manager produced a 56-second request.
The part worth carrying away
Look at the %user column again during that window. It goes from 1.6% to 4%.
And free memory increased, from 317 MB to 396 MB, because the kernel was reclaiming cache under write pressure.
So for the entire 45 minutes in which p99 was 56 seconds:
- CPU said the machine was idle. By CPU it was idle. It was blocked on disk.
- Memory said the machine was improving. An alert threshold on free memory would have fired in the good direction.
- Only iowait showed anything, and iowait is not on most default dashboards.
A standard CPU-and-RAM monitoring setup would have reported a perfectly healthy machine throughout.
There's a load-average wrinkle in the same direction: I sampled a load of 2.72 at 06:36 alongside 4% user CPU, which reads as a contradiction until you remember Linux counts processes blocked in uninterruptible disk wait toward load average. The one metric that did look alarming looked alarming for a reason most dashboards don't explain.
I spend my time measuring failures that don't announce themselves, so I should have seen this coming, and I still assumed OOM first. The instrument was working correctly and reporting on the wrong layer. That's the same shape as the thing I built the harness to catch.
The part I can't explain
Four sends were refused, at 05:23:43, 05:23:58, 05:24:41 and 05:24:56. Sequences 825, 826, 829 and 830 failed while 827 and 828 succeeded between them, so it was flapping, not an outage window.
Four sources for that moment, all quiet:
- CPU: 1.51% user, 3.07% iowait, 94.69% idle
- Network: 5.12 packets/s in, flat across the window
- Container memory: 257.9 MiB, steady
- The service's own container log for 05:20 to 05:30: empty
I don't know why those four were refused.
And I made it harder on myself, because my harness recorded that each send failed but not why: the error was caught and thrown away. Connection reset, timeout and a non-2xx response are indistinguishable in my own ledger for those four events. Fixed now, but it's a fair thing to hold against this run, and a decent argument for logging the error string on every failure path even when you're sure you'll never need it.
Limits
One box, one config, one trivial workload (webhook in, one HTTP call out). Nothing here transfers to a heavy multi-step workflow or a queue-mode deployment. Duration test, not throughput. Single run on a single day, so the apt-news window is a scheduled event I happened to catch rather than a rate. And the cold-path explanation for the latency result is unproven.
If you self-host on a small VM
Three things I'd take from this:
-
Check your
apt-daily,apt-newsandunattended-upgradestimers before blaming your application for mystery latency spikes. On a shared-core instance with a network disk, package metadata refresh is a real workload. - Put iowait on the dashboard. CPU and memory will both look fine through a disk stall, and memory will look like it's improving.
- A small box is likely fine for sustained low-rate work. It may even be faster busy than idle. The thing that hurt me was maintenance, not traffic.
Method, per-run data and the full bucket table: https://benchtruth.com/self-hosted-n8n-under-load
I publish per-run data for all of this and pay for my own subscriptions. No vendor pays me, and there are no affiliate links on the data pages.
Top comments (0)