DEV Community: אחיה כהן

My sudo Logs Hit 10 GB. Session Count Told Me Nothing.

אחיה כהן — Thu, 24 Sep 2026 07:41:26 +0000

A disk-usage alert sent me looking at /var/log/sudo-io on a Chatwoot production box. It held 5.9 GB and was adding roughly 654 MB every week. Nothing on that server had changed. No new services, no new admins, no new cron jobs.

My first instinct was the wrong one: I assumed the server was simply busier than the others. It isn't. It runs the fewest sudo sessions of any box in the fleet that logs them.

Here is what the numbers look like, measured live across the whole fleet this morning.

The measurement

Two commands. The first counts sessions, the second measures bytes:

# how many sudo sessions were recorded
sudo find /var/log/sudo-io -name log -type f | wc -l

# how much disk those sessions occupy
sudo du -sk /var/log/sudo-io

Ten hosts in my SSH config. Only four of them have /var/log/sudo-io at all. The other five never had log_output enabled, and one was unreachable during the run. That inconsistency is its own finding, and I'll come back to it.

Host	Log volume	Sessions	KB per session
chatwoot	5.48 GB	16,168	356
supabase	2.33 GB	56,794	43
website	1.71 GB	14,289	126
chatwoot_admon	0.49 GB	8,788	59
Total	10.02 GB	96,039	109

Read the last two columns together. supabase recorded 3.5x more sudo sessions than chatwoot and produced less than half the volume. Per session, chatwoot is 8.3x heavier.

Session count and log size are not measuring the same thing, and I had been reading the wrong one.

Why bytes and sessions diverge

log_output in sudoers does not record which command ran. It records everything the command writes to its pty. A short admin command that prints two lines costs a few hundred bytes. A command that streams a file costs the size of that file.

So this:

sudo systemctl restart nginx

costs almost nothing, while this:

sudo cat /opt/chatwoot-backups/dump-2026-09-14.sql

costs 60 MB. The dump gets written to disk a second time, into the audit log, carrying no information the original didn't already have.

That was the leak. The offsite backup job pulls a database dump every night by reading it through sudo. supabase and website already had those reads exempted in /etc/sudoers.d/iolog-exemptions. chatwoot did not. chatwoot_admon had no exemptions file at all. Same fleet, same backup design, three different configurations — because the exemption file was written by hand on each host, one host at a time, and nothing ever checked that they matched.

The fix is an exemption, not a shutdown

The tempting fix is to turn log_output off. Don't. The recording is the point; you want to know what a human typed at 2am on a box holding customer conversations. What you don't want is a byte-for-byte copy of files you already have.

sudoers lets you carve out specific commands:

Cmnd_Alias IOLOG_EXEMPT = /usr/bin/cat /opt/chatwoot-backups/*, \
                          /usr/bin/stat /opt/chatwoot-backups/*, \
                          /usr/bin/zcat /var/log/caddy/*, \
                          /usr/bin/tail /var/log/caddy/*

Defaults!IOLOG_EXEMPT !log_output

The session still gets logged: who ran what, when. Only the byte stream is dropped.

Two things worth saying out loud about this file:

Validate before you apply it. visudo -cf /etc/sudoers.d/iolog-exemptions parses the file without installing it. A syntax error in sudoers.d can lock you out of sudo on a machine you may only reach through sudo. Keep a timestamped backup next to it.

The paths matter more than the commands. Exempting /usr/bin/cat outright would let anyone pipe any file past the audit log. Exempting cat restricted to the backup directory keeps the hole the size of the actual problem.

What ten days later looks like

I applied the exemptions across all four hosts on September 14 and measured again today, September 24.

At 654 MB/week, chatwoot should have been near 6.8 GB by now. It reads 5.48 GB. The curve turned over. Two things are doing that work together. New writes stopped, and the 60-day retention cron keeps deleting old sessions, so I'd expect the number to keep falling into mid-October as the pre-fix backlog ages out. I'm not claiming the exemption alone recovered 1.3 GB. I'm claiming the growth stopped, and that's the part I was chasing.

The part I'd actually change

The exemption fixed one symptom. The real defect is that five of ten hosts don't record sudo I/O, two of the four that do had drifted apart, and I only noticed because a disk filled up. A config that exists on some machines and not others isn't a policy, it's a habit that happened to repeat.

So the check I added isn't "is the log too big." It's this, run across the fleet:

# does this host record sudo I/O at all, and is the exemption file present?
grep -rl log_output /etc/sudoers /etc/sudoers.d/ 2>/dev/null | head -1
test -f /etc/sudoers.d/iolog-exemptions && echo "exemptions: yes" || echo "exemptions: MISSING"

Two lines, and it answers the question the disk alert was too late to ask. This is the same reasoning behind most of the fleet automation I build at achiya-automation.com — the alert that fires when a threshold breaks is always a worse version of the check that confirms a configuration is still what you think it is.

The metric to steal

If you have log_output enabled anywhere, don't look at total size and don't look at session count. Look at bytes per session:

sudo bash -c 'kb=$(du -sk /var/log/sudo-io | cut -f1); \
  n=$(find /var/log/sudo-io -name log -type f | wc -l); \
  echo "$((kb / (n>0?n:1))) KB per session across $n sessions"'

Across my fleet the number is 109 KB. Anything in that range is ordinary admin work. A host sitting at 350+ has a command streaming a file through sudo, and you can find it in one pass. A host at 43 is doing plenty of work and costing you nothing.

A question for anyone running log_output in production: what's your bytes-per-session number, and did the outlier turn out to be a backup job like mine, or something you'd never have guessed? I'm especially curious whether anyone exempts by path the way I did here, or whether you found a cleaner pattern than hand-maintained Cmnd_Alias blocks that drift between hosts.

My Monitoring Cron Never Ran Once. crontab -l Showed It Fine.

אחיה כהן — Sun, 20 Sep 2026 06:49:59 +0000

I write monitoring so I don't have to watch things by hand. So there is a special kind of dread in discovering that the monitor you installed to watch production has itself been dead since the day you installed it — and that everything you would normally check said it was fine.

This one was dead for five days. Here's the autopsy, because the failure mode is boring, extremely common, and almost invisible.

The setup

I run WhatsApp automation for businesses, and a session can silently drop: the container stays "up", the health check stays green, but the line stops receiving messages. So I built a small watchdog, waha-session-watch, that polls the sessions API every ten minutes and alerts if a line goes quiet. It runs as a dedicated unprivileged user, waha, and needs sudo for one docker call.

The crontab line looked completely ordinary:

*/10 * * * * sudo /usr/local/bin/waha-session-watch >> /opt/waha-watch/cron.log 2>&1

Install it, watch cron.log fill up, move on. Except cron.log never filled up. And I didn't notice, because I wasn't looking at cron.log. I was looking at the tool's own log, watch.log, which had plenty of recent entries. Green across the board.

Five days later a production line went dark for the better part of a day and nothing alerted. That's when I actually went digging.

The first lie: `crontab -l`

$ crontab -l -u waha
*/10 * * * * sudo /usr/local/bin/waha-session-watch >> /opt/waha-watch/cron.log 2>&1

There it is. It's installed. crontab -l printing your line feels like proof the job runs. It is not. It only proves the line is stored. Whether cron can actually execute it is a completely separate question, and the answer here was no. Not once.

The second lie: a log full of runs

watch.log had entries. Recent ones. So the tool clearly was running... right?

2026-09-08 21:40:12  OK  7 sessions healthy
2026-09-08 21:52:03  OK  7 sessions healthy
2026-09-08 21:59:47  OK  7 sessions healthy
2026-09-08 22:22:15  OK  7 sessions healthy
2026-09-08 22:32:51  OK  7 sessions healthy

Look at the minutes: :40, :52, :59, :22, :32. A */10 cron job fires at :00, :10, :20, :30, :40, on the round ten. These timestamps are ragged. Every single one of them was me, running the tool by hand from the terminal on install day to confirm it worked. Not one line in that log came from cron.

That's the trap in one sentence: a log full of successful runs is not proof of scheduled runs. Manual runs and cron runs land in the same file and look identical unless you read the clock.

The tell: one line in syslog

Here's where it actually gives itself up. I grepped the system log for the job:

$ grep CRON /var/log/syslog | grep waha
... (CRON) info (No MTA installed, discarding output)
... CMD (sudo /usr/local/bin/waha-session-watch >> /opt/waha-watch/cron.log 2>&1)

No MTA installed, discarding output.

That message is the whole case. cron ran the job, the job produced output on stderr, and cron tried to email that output to the user, because that's what cron does with anything a job writes when it can't otherwise capture it. There's no mail server, so it threw the text away. Which means: something printed an error, every ten minutes, into a void, for five days.

If your command's output were really being captured by >> cron.log, cron would have nothing to mail and this line would never appear. (No MTA installed, discarding output) sitting next to a CMD is always a sign the command printed something nobody saw. It is one of the highest-signal lines in all of syslog and almost nobody greps for it.

The root cause: who opens the `>>`

The bug is the redirect. Specifically, who opens it.

sudo /usr/local/bin/waha-session-watch >> /opt/waha-watch/cron.log 2>&1

It's intuitive to read this as "run the whole thing as root and append to the log." That's not what happens. Redirections are set up by the shell that launches the command, before the command is exec'd, and cron runs this line as the waha user. So the >> /opt/waha-watch/cron.log is opened with waha's permissions, before sudo ever runs.

And the directory:

$ ls -ld /opt/waha-watch
drwxr-xr-x 2 root root 4096 Sep  8 /opt/waha-watch

root:root, mode 755. The cron.log file didn't exist yet, and waha has no write permission in a 755 directory it doesn't own. So the shell can't create the file, the redirection fails, the shell bails out before executing sudo, and the command never runs at all. cron dutifully mails the "permission denied" to nobody.

sudo was never the problem. The unprivileged redirect target in front of it was. The privilege you granted the command is irrelevant when the shell dies setting up its plumbing first.

The fix

Anticlimactic, as these always are:

sudo touch /opt/waha-watch/cron.log
sudo chown waha:waha /opt/waha-watch/cron.log

Create the log file up front, owned by the user cron runs as. First round-numbered timestamp showed up ten minutes later and the watchdog has been real ever since. No code changed. The script was always fine.

What I actually changed

The fix took ten seconds. The lesson took five days, so it's worth more than the fix.

After installing any cron job or systemd timer, verify the first scheduled run in two independent places:

The scheduler's own record — grep CRON /var/log/syslog and confirm a CMD line with no discarding output next to it, at a round-numbered minute.
The tool's own log — an entry whose timestamp lands on the schedule, not a ragged manual one.

If those two don't agree, you have a job that is stored but not running. crontab -l is not on this list on purpose: it can only ever confirm storage.

And the redirect rule specifically: if a scheduled command redirects into a log, create that file ahead of time, owned by the user the scheduler runs as. Don't assume sudo on the command buys you anything for the >>. That file is opened by the caller, not the callee.

I build this kind of infrastructure for a living — production WhatsApp automation for Israeli businesses at Achiya Automation — and this watchdog exists precisely so a silent line doesn't go unnoticed. The irony of the watchdog itself going silent for five days is not lost on me. That's exactly why the two-place verification is now non-negotiable in my install checklist.

Your turn

What's the longest one of your "installed" monitors has been quietly dead before you noticed, and what finally gave it away? I'm collecting the tells that actually work in the wild, beyond the syslog one above. Drop yours below. #discuss

I Audited 38 Production Containers. Zero Use depends_on restart.

אחיה כהן — Thu, 17 Sep 2026 07:52:44 +0000

My n8n upgrade runbook has a line in it that I wrote by hand, sometime in 2026:

After the main container reports healthy, run docker restart n8n-docker-caddy-n8n-worker-1.

I have run that line at every version bump for months. It works. I never questioned it.

Last week I finally read the Compose specification closely enough to notice that I had hand-implemented a field that has existed since Compose 2.17.0, released in 2023. The field is depends_on.<service>.restart. My runbook is a human being standing in for one boolean.

So I went and measured how widespread the gap actually is — not in a demo repo, in the fleet that runs my clients' work.

The measurement

Eight servers, read-only, on 2026-09-17. I pulled two things from every running container: its healthcheck configuration, and the com.docker.compose.depends_on label that Compose v2 writes at create time. That label is the ground truth — it records the resolved dependency as service:condition:restart, so I didn't have to trust the YAML on disk matching what is actually running.

for c in $(docker ps --format '{{.Names}}'); do
  docker inspect "$c" --format \
    '{{index .Config.Labels "com.docker.compose.project"}}|{{.Name}}|{{index .Config.Labels "com.docker.compose.depends_on"}}'
done

Results across 38 running containers, in 14 Compose projects, on 8 servers:

	count
Containers with a healthcheck	33 / 38
Containers with any `depends_on`	7 / 38
Of those 7, using `condition: service_healthy`	7 / 7
Of those 7, using `restart: true`	0 / 7

Every dependency I had bothered to declare was declared well — all seven wait for health, none of them settle for service_started. And every single one of them ends in :false.

That trailing false is the restart flag. It means: when Compose updates the dependency, do not restart me.

The 31 containers with no depends_on at all are a separate confession, and I'll come back to them.

What depends_on actually promises

The long syntax takes three sub-options, and it is worth being precise about the scope of each:

condition: service_started — the dependency's container has been created and started. Says nothing about the process inside it.
condition: service_healthy — the dependency's healthcheck has passed. This is the one everybody reaches for.
condition: service_completed_successfully — the dependency ran to exit 0. For migration jobs and seeders.
restart: true — "Compose restarts this service after it updates the dependency service."
required: false — downgrade a missing dependency from an error to a warning.

Read the restart line again. It is scoped to Compose-controlled updates — the docker compose up -d that replaces a container with a new image. It is explicitly not about runtime crashes; your restart policy covers those.

So condition and restart answer two genuinely different questions:

condition → in what order do these containers come up the first time?
restart → what happens to me when the thing underneath me gets replaced?

A stack with condition: service_healthy everywhere and restart nowhere is fully ordered on a cold boot and fully unordered on every deploy after that. Which, on a server that boots once a year and deploys twice a month, means the feature is protecting the rare case and sitting out the common one.

Where condition alone was not enough

Two dated incidents from this fleet, both of which I originally filed as something other than what they were.

2026-07-31 — the healthy database that wasn't ready. Upgrading n8n from 2.30.6 to 2.32.6 on the queue-mode server. The worker's declared dependency reads, to this day, postgres:service_healthy,redis:service_healthy. Postgres was genuinely healthy. The worker still died — stack trace in MigrationExecutor.executePendingMigrations, then unhealthy and stuck there.

The reason is that service_healthy asked Postgres whether it was accepting connections, and Postgres honestly said yes. Nobody asked the question that mattered: has the schema been migrated to the new version yet? The main container and the worker both came up against the same database and both started running migrations. Main won. The worker crashed on a half-applied schema.

There is no healthcheck you can write on Postgres that answers "is the application's schema current", because the answer doesn't live in Postgres — it lives in whichever sibling container is doing the migrating. The fix I actually deployed was that runbook line: wait for main, restart the worker. Which is depends_on: { n8n: { condition: service_healthy, restart: true } } performed manually.

2026-09-04 — the update that kept everything running and broke one thing anyway. Docker engine 29.7.2 → 29.8.0 across the fleet. live-restore did its job: every container survived, 8→8, 5→5, 3→3. Except supabase-docker-proxy, which went unhealthy with:

ERROR Docker API proxy error error="dial unix /var/run/docker.sock: connect: connection refused"

It mounts /var/run/docker.sock. Replacing the daemon replaces the socket; the process inside kept holding the old one and never reconnected. live-restore keeps the container running — it cannot heal a client that lost its connection to the API.

The tell was ugly: docker ps showed Up 3 days (unhealthy). The uptime was true. It really had not fallen over. FailingStreak was 75, and that was the number that said when it actually broke. A docker restart fixed it in fifteen seconds.

Note what both incidents have in common. Neither container crashed. Both needed to be restarted because something underneath them changed, and in both cases the thing that noticed was a person.

The 31 with no depends_on

The honest part of the audit. Four of them are application containers in production: the Rails and Sidekiq containers of two self-hosted Chatwoot instances. They have no declared dependency on their own Postgres or Redis. On a cold boot they race, lose, crash, and get resurrected by restart: always.

That works. It has worked for a year. It is also the reason that stack's real recovery time is invisible — the recovery is a crash loop that happens to terminate, and nothing records how long it took.

While I was in there I measured the other number that deploys care about. Those Rails containers run start_period: 60s, interval: 60s, retries: 3. Add it up: a Rails container that comes back up broken can present as starting, then healthy-adjacent, for up to four minutes before Docker is willing to call it unhealthy. If your deploy script polls health to decide whether to roll back, that is the size of your blind spot.

What I changed

For the dependencies that already exist, adding one line each:

services:
  n8n-worker:
    depends_on:
      postgres:
        condition: service_healthy
      n8n:
        condition: service_healthy
        restart: true     # ← replace main, and I get restarted after it

Two changes in that block, not one. The restart: true is the field I had been simulating by hand. The new n8n entry is the fix for the 2026-07-31 incident: the worker's real dependency was never Postgres, it was whoever migrates Postgres.

Run this on your own fleet before you assume you're clean — it reads the resolved labels, not the YAML you think is deployed:

for c in $(docker ps --format '{{.Names}}'); do
  d=$(docker inspect "$c" --format '{{index .Config.Labels "com.docker.compose.depends_on"}}')
  [ -n "$d" ] && echo "$c -> $d"
done

Anything ending in :false is a container that will keep running against a dependency you just replaced.

Two caveats I'd rather state than have corrected in the comments. First, restart: true is not free — it widens each deploy, because more containers cycle. On a stack where the dependency changes often, that is the trade you are making. Second, it is still not zero-downtime; it is ordered downtime. Real zero-downtime means two versions serving at once behind a proxy, and Compose is not the tool for that. What this fixes is the cheaper, more common failure: the deploy that finishes green while something behind it is quietly talking to a socket that no longer exists.

Most of the infrastructure work at Achiya Automation looks like this — not clever, just the boring field nobody read the docs for.

The question

I'm curious whether the :false ratio is a me problem or an everyone problem. Run the loop above and post your two numbers: how many of your running containers declare depends_on at all, and how many of those end in :true?

And if you're above zero on the second number — what made you go looking? I only found it because I noticed I'd written the feature into a runbook by hand.

I Measured 27,257 MCP Connections. The p90 Session Waits 35s.

אחיה כהן — Wed, 16 Sep 2026 05:45:26 +0000

In July I wrote a 56-line shell script called mcp-optional whose entire job was to remove two MCP servers from my config.

Not fix them. Remove them.

Both were stdio servers, so each one spawns its own Node process per session. With around 13 Claude Code sessions open on a 16GB M4, that was roughly 1.4GB of duplicated RAM, which meant swap thrash, which meant a hot laptop. Disabling them by default and re-enabling on demand fixed it.

I filed that under "memory problem, solved" and stopped thinking about it.

This morning, chasing something unrelated, I started reading the MCP connection logs Claude Code leaves on disk. It turns out one of those two servers wasn't just the RAM tax. It's also the slowest thing I connect to, by a wide margin — and the reason isn't its code at all.

Here's what 35 days of logs actually say.

What I measured

Claude Code writes a JSONL log per MCP server, per session, under ~/Library/Caches/claude-cli-nodejs/<project>/mcp-logs-<server>/. Two lines matter:

Starting connection with timeout of 30000ms
Connection established with capabilities: {"hasTools":true,...}

Pair them inside one file and you get a real connection latency. I did that across every project directory on this machine:

33,599 log files, 2026-08-13 → 2026-09-16 (35 days)
38,876 connection attempts started
27,257 clean start→established pairs, across 22 servers
2,888 sessions

This is one developer's machine, not a lab. That's the point — it's the distribution you actually live in.

The headline numbers

percentile	connect time
median	582 ms
p75	1,650 ms
p90	3,678 ms
p95	6,428 ms
p99	14,049 ms

The median looks great. Under six tenths of a second — nobody would ever file a bug.

But 39% of connections take longer than a second, and the p99 is fourteen seconds. Against a client timeout of 30,000ms, the tail is not a rounding error. It's most of the way to the wall.

Cause #1: it's OAuth, not your server

This is the finding that reframed the whole thing for me. I split every connection by whether a token refresh happened inside that same connection window:

	n	median	p90
token refresh in window	748	3,170 ms	9,619 ms
no refresh	26,509	550 ms	3,400 ms

5.8× slower. Same servers, same network, same machine. The only difference is whether the client had to go get a new access token first.

Your MCP server didn't take three seconds to start. Your MCP server took 550ms to start, and an OAuth round-trip took the other 2.6 seconds while the server sat there doing nothing.

That's why "why is this server slow" is usually the wrong question. Only 2.7% of my connections hit a refresh — but that 2.7% is where a disproportionate share of the visible pain lives, because it's the path that turns a fast connection into a slow one non-deterministically. You can't reproduce it on demand, so you blame the server.

Cause #2: transport, with a twist

transport	n	median	p90
HTTP	19,894	676 ms	3,395 ms
stdio	7,363	222 ms	4,693 ms

stdio is 3× faster at the median — no surprise, there's no network. But look at p90: stdio is worse in the tail. Spawning a process has a floor of roughly nothing and a ceiling of "npm decided to do something." An HTTP server that's already running is slower on average and far more predictable.

If you're choosing a transport, that trade is the actual decision. Not "stdio is faster."

Cause #3: nobody connects to one server

Per-connection numbers hide the thing that actually costs you time, which is fan-out. Per session on this machine:

median 8 MCP servers, max 16
median total connect time: 5.9 s
p90: 35.5 s
p99: 81.8 s
33.7% of sessions spend more than 10 seconds just connecting

That p90 is the number that made me stop and re-read my own script. A third of the time I open a session, I'm waiting a meaningful fraction of a minute before the first token — not because any single server is broken, but because eight of them each drew from that fat tail and nothing amortizes.

Worst offender by median, unsurprisingly: one of the two servers I'd already disabled for RAM reasons, at 2,261 ms median and 10,413 ms at p90. I removed it in July for the wrong reason and got the right outcome.

The 29.7% nobody talks about

Of 38,876 connection attempts, 11,556 never logged an established line at all — 29.7%.

Some of that is log rotation cutting a file mid-handshake, so treat it as an upper bound rather than a failure rate. But it isn't evenly spread: a single remote connector accounts for 7,422 of them on its own. That's not noise, that's one integration failing over and over while everything upstream stays quiet about it.

The trap I almost published

My first pass had a max connect time of 119 seconds and a p99 of 14.5s, and I nearly wrote a paragraph about it.

Then I re-read the client's own log line: timeout of 30000ms. A connection cannot establish at 119 seconds if the client gives up at 30. Those samples weren't slow connections — they were a closed laptop. The wall clock kept running through sleep; the connection didn't.

So I capped every measurement at the client's declared 30s timeout and threw the rest out. That removed 24 of 27,281 samples — 0.09%. Every number above is post-cap.

I mention it because the uncapped version would have been a better story and a false one. If you run this on your own machine, cap it.

What actually helps

Count your servers before you tune any of them. Median 8, p90 35 seconds. Removing one mediocre server beats optimizing a good one.
Make heavy servers opt-in. That's all mcp-optional does — claude mcp remove by default, claude mcp add when a task needs it. It's the highest-leverage 56 lines I have written this year, and I wrote it for the wrong reason.
Look at your refresh path, not your server code. If connections are erratic rather than uniformly slow, that's the OAuth signature: 550ms most of the time, 3.2s when the token expired.
Prefer an already-running HTTP server for anything heavy. Worse median, much better p90.
Read the logs you already have. Nothing here required instrumentation. It was sitting in a cache directory the whole time.

The reason I care about any of this: at Achiya Automation I run agent tooling against real client infrastructure all day, and a 35-second session start that lands on one session in ten — with a third of them over 10 seconds — is a tax I pay dozens of times a day without ever seeing it on a bill.

One question

If you've got MCP servers configured right now — how many, and when did you last check whether you still use all of them?

I'd genuinely like to know whether median-8 is normal or whether I'm the outlier. And if you've measured a refresh-vs-no-refresh split on a different client, I want to see that number, because I only have one machine to look at.

3 of 7 WhatsApp Lines Died in 9 Days. Docker Said 'Healthy'.

אחיה כהן — Mon, 14 Sep 2026 05:47:45 +0000

At 08:33 this morning I ran GET /api/sessions against my own WhatsApp gateway and read the line I least wanted to read:

{
  "name": "session_<redacted>",
  "status": "FAILED",
  "me": { "pushName": "<insurance agency>", "id": "<redacted>@c.us" },
  "timestamps": { "activity": 1789052954786 }
}

That timestamp is 10 September, 18:09. The check ran on 14 September. The line had been dead for eighty-six hours, and every dashboard I own was green.

I run WhatsApp automation for Israeli small businesses. Each client's business number lives as a session inside a self-hosted WAHA container, which bridges their line into a CRM and a pile of n8n workflows. Seven sessions, seven businesses, one container.

Between 4 and 13 September, three of those seven lines went down. Here is what each one taught me, and what I check now.

The health check was answering the wrong question

My monitor checked that the waha container was running and responsive. It was. It stayed running, responsive and healthy through all three outages, because a container that is perfectly happy to serve you a list of dead sessions is, by every definition Docker cares about, working.

This is the ordinary liveness-versus-readiness mistake, except the readiness that mattered was not the service's. It was the readiness of seven independent things inside the service, each of which can fail on its own, for unrelated reasons, without the process noticing or caring.

One client's line failed on 4 September at 20:47. I found it on the 8th. Three and a half days, during which the container reported itself healthy roughly five hundred times.

The failure that looks like nothing at all

The second one is the reason I am writing this.

On 5 September at 11:04 the gateway logged a device removed stream error for a clinic's line. The linked device had been removed from the client's phone — deliberately, or while clearing out old sessions. I never found out which, and it does not matter.

What matters is what the clinic saw afterwards: nothing. Messages kept arriving on the owner's phone exactly as before. Customers got replies, because a human was reading the phone. From the business's side the line was completely, visibly fine.

Only the CRM went blind. Only the automations stopped firing. The reminders that go out before appointments, the routing, the logging, the follow-ups — all silently not happening, behind a line that looked perfectly alive to the only person in a position to notice.

If your failure mode is invisible to the user and invisible to your health check, you do not have a monitoring gap. You have no monitoring.

You cannot always tell whose line just died

When I sat down to write the alerting, I hit a problem I had not anticipated: a dead session may not be able to tell you who it belongs to.

While a session is WORKING, /api/sessions gives you me.id (the number) and me.pushName (the business name). When a session is torn down by an unlink, those fields are wiped. The identity disappears at precisely the moment you need it to address an alert.

I want to be careful here, because I wrote this down as an absolute rule in my own notes last week and it is not one. Checking the live API this morning while drafting this post, the currently-failed insurance line does still carry its me block, as you can see in the JSON above. So it depends on how the session died: a removed device wipes the identity, a connection-level failure does not.

Which is worse than either rule on its own, because it means you cannot depend on the field being there. The fix is the same either way: keep your own roster. My watcher rewrites a roster.json on every pass — but only from sessions currently in WORKING, because a single pass during an outage would otherwise overwrite good identity with nulls.

And you cannot recover the mapping from billing, either. I scanned all 141 customers in our accounting system looking for these business numbers. Zero matches. What sits in an invoice is the owner's personal mobile, which is a different number from the business line the bot runs on. The mapping is manual, or it does not exist.

The watchdog that never ran

On 8 September I built the thing that should have caught all of this: a watcher on a ten-minute cron that diffs session states, notifies the business owner when their own line drops, and sends them a pairing code to reconnect it themselves.

I tested it. Ten test scenarios, three real bugs caught before install. I watched it run. I marked the problem solved.

It did not run once from cron until 13 September.

*/10 * * * * sudo /usr/local/bin/waha-session-watch >> /opt/waha-watch/cron.log 2>&1

/opt/waha-watch is owned by root, mode 755. The cron user is waha. The redirect is evaluated by the shell before sudo runs, as the unprivileged user, so creating cron.log failed, so the shell exited, so the command never executed at all. Cron dutifully tried to mail me the error and found no MTA installed, so it dropped it:

(CRON) info (No MTA installed, discarding output)

Five days of no monitoring, hidden behind a crontab entry that reads perfectly.

What fooled me during the install was watch.log. It had runs in it — 21:40, 21:52, 21:59, 22:22. Non-round minutes, every one of them, because they were my own manual test runs. Scheduled runs land on round minutes. I looked at a log full of evidence that the tool worked and read it as evidence that the schedule worked.

The insurance line fell on the 10th, into that exact blind window. The first genuinely scheduled run, on the night of the 13th, found it immediately — three days late.

The fix was touch cron.log && chown waha:waha. That is the entire fix. The lesson costs more than the fix: that No MTA installed, discarding output line in syslog is never noise. It means your scheduled command printed something nobody has ever read.

What I check now

Four things changed:

Health-check the units, not the process. The container being up says nothing about the seven sessions inside it. Whatever your service multiplexes — tenants, sessions, connections, device links — that is the thing with a state worth alerting on.

Verify the first scheduled run, in two places. Not "is it in crontab -l", which was true and meaningless for five days. Check syslog for the CMD line, and check the tool's own log for an entry on a round minute. If a cron entry redirects to a file, create the file first, owned by the cron user.

Keep a persistent roster, written only from healthy state. Never let a failure pass overwrite identity you will need to send the alert.

Alert the person who can actually fix it. A dropped line is repaired by someone holding the phone, not by me. The watcher now messages the business owner directly with a pairing code, inside working hours, once per incident. The one-message-per-incident rule matters as much as the alert: the fastest way to get a monitoring channel ignored is to send the same thing every ten minutes.

It matters that this is WhatsApp specifically. Bezeq's 2025 Internet Report — the annual survey of Israeli internet use, published in December 2025 — notes in passing that reaching 90% WhatsApp usage in Israel took about a decade. For a small business here, that line is not a channel. It is the front door. Which is exactly why the customer-service bots running on those lines need a liveness signal independent of the phone in the owner's pocket, because the owner's pocket will never report the outage. It does not look like one from there.

Three lines, nine days, one container that never once said anything was wrong.

The question I am actually asking

I want to hear about the second category specifically — not the outage your monitoring missed, but the one where the user experience stayed completely normal while the system underneath was dead, so nobody had any reason to report it.

What was the failure, and what signal finally caught it? I am collecting these, because I suspect the honest answer for most of them is "a human noticed weeks later," and I would like to be proven wrong.

My safety guard was checked in 6 places. Nothing ever set it.

אחיה כהן — Fri, 11 Sep 2026 06:43:45 +0000

A user of my Safari MCP server — the user was me, from a scheduled run — hit a failure I could not talk my way out of.

Right after the background daemon restarted, the server opened a browser tab and handed back a capability token for it. The token is how every later command proves it is allowed to touch that tab, because the whole point of the ownership system is that the server must never act on a tab you opened. Thirty-three seconds later, the very next command was refused:

Tab safety: receipt is forged, stale, ambiguous, or not valid for this origin

Forged. By me. For a tab I had just opened, four lines earlier.

Worse, it was unrecoverable from the client side. The token could not be rotated. Switching to the tab by index was refused — "not opened by this MCP session." Closing it was refused too, which is correct, since closing is the one action you cannot undo and ownership could no longer be proven. So the tab sat there, stranded and unusable, and the agent had to reopen its work somewhere else.

Ruling things out

I had two obvious hypotheses and both died in the same session.

"A newer tab invalidates the older token." Minted a token, opened a second tab, used the first token afterwards. Worked fine.

"Switching by index corrupts the next command." Switched by index, ran a command with no token at all. Worked fine.

The one thing every failure had in common was timing. Everything opened once the daemon had been up for a couple of minutes behaved perfectly. Only the tabs created inside the restart window went bad. The first tab creation in that window took 22 seconds — the extension's background worker was still cold-starting — and that number turned out to be the entire story.

The actual mechanism

Tokens are stamped with a browser-session epoch: a random identifier that ties a token to one continuous run of the browser. If the browser restarts, the epoch rotates, and every old token is correctly rejected instead of being allowed to point at a recycled tab id. Good design. The epoch lives in session storage, which is the authority.

Here is the function that reads it:

async function _ensureBrowserSessionEpoch() {
  if (!_browserSessionStorageAvailable) {
    throw new Error("Browser session storage is unavailable; refusing durable tab authority");
  }
  if (/^[a-f0-9]{36}$/.test(_browserSessionEpoch)) return _browserSessionEpoch;
  // ... otherwise read storage, or mint and persist a new epoch
}

That last if is the problem. Once the worker has resolved an epoch into memory, it never consults storage again for the rest of its life. The cached copy is treated as the truth.

So when the epoch rotates in storage while the worker is still alive — which is exactly what a daemon restart plus an extension re-handshake does — the worker keeps stamping brand-new tokens with the old epoch, and writes them to durable storage that way. Nothing complains, because nothing is looking.

The complaint arrives later, from the one path that does read storage. When the worker eventually cold-starts, it rehydrates its tables and filters the stored tokens:

const receiptsFresh = receiptEnvelope &&
  receiptEnvelope.version === _TAB_RECEIPTS_VERSION &&
  receiptEnvelope.browserEpoch === browserEpoch && ...

Every stored record carries the stale epoch. receiptsFresh is false. The entire record array is discarded, the in-memory tables are cleared and repopulated from that empty set, and every token minted during the stale window dies at the same instant. Which is precisely what the report showed: two tabs, created five seconds apart, rejected together.

The part that actually bothers me

There was a guard for this. It just could not fire.

The code tracks a counter whose whole job is to notice that the browser-session identity changed underneath an in-flight operation. It is compared at six call sites, across the three functions that own the epoch:

2885, 2896, 2920   _ensureBrowserSessionEpoch
3043               _hydrateOwnedTabs
3075, 3095         _persistOwnedTabs

Each comparison guards a real invariant, and each one has a hand-written error message explaining what went wrong: "Browser-session identity changed during initialization." "Browser session changed during tab ownership recovery."

The variable is assigned exactly once, on line 2819:

let _browserEpochGeneration = 0;

That is the declaration. Nothing increments it. Ever.

$ grep -nE '_browserEpochGeneration\s*(\+\+|--|=[^=])' extension/background.js
2819:let _browserEpochGeneration = 0;

One hit, and it is the let. Six comparisons against a constant, plus three sites that dutifully capture it into a local first.

Why it survived review, including mine

I have been staring at this file for months, and I want to be precise about why I did not see it, because "you should have looked harder" is not a lesson anyone can apply.

The guards read as protection. Every site looks like careful concurrency code. There is a captured local, an await, and a re-check afterwards — the exact shape you are trained to pattern-match as correct. Reading for shape rather than for reachability, it passes.

Three of the six had a live half. They are compound:

if (generation !== _browserEpochGeneration || browserEpoch !== _browserSessionEpoch) {

The second clause works. So those three guards do fire, sometimes, on the epoch comparison alone — which is exactly why this bug shows up at rehydration time rather than never at all. Half of the protection was real, and the working half masked the dead half. If all six had been dead, the failure would have been louder and I would have found it sooner.

The other three, the fully dead ones, were the ones that mattered most. They sit inside _ensureBrowserSessionEpoch and have no second clause, because that function is the thing that sets the value they would compare against. Those three were supposed to catch the rotation at mint time, before a doomed token is ever handed to a caller. Instead the rotation is caught at rehydration, long after the bad tokens are in the client's hands and in durable storage.

A dead guard is worse than a missing one. A missing guard is an obvious gap. A dead guard is a gap plus a comment asserting the gap is covered, and it makes the next person to read the file — me, repeatedly — stop looking.

What would have caught it

Not a unit test. Every behavioural test over these paths passes, and would still pass with all six comparisons deleted, because the code under test never rotates the epoch. The defect is an omission, and you cannot observe an omission by exercising what is there.

This project already had one test for exactly that category, written after a different omission bug, and its own comment says so:

This is a source-level check on purpose: the failure is omission at a call site, which no behavioural test over the existing sites can see.

That test reads the source as text and asserts a property about the call sites. The same trick works here, and the rule is embarrassingly simple:

A variable that is compared must be assigned somewhere other than its declaration.

That is a grep. It is cheap enough to run over every mutable module-level variable in a file, and it would have failed loudly the day this landed. Not a linter rule I had to invent either — "assigned once, read many, never reassigned" is the shape, and the moment you write it down you realise how many state machines and generation counters and dirty-flags could rot this way without a single test turning red.

What I have not done

I have not shipped the fix.

The change itself is small: stop trusting the cached epoch, re-verify it against storage, and increment the counter when the two disagree — which lights up all six existing guards at once. I know what to type.

But this is the tab-ownership guard, and the reason it exists in the first place is that early versions of this server genuinely clicked into and closed people's tabs. The failure mode of getting the fix wrong is inverted from the current bug: today a good token is rejected rarely, in a narrow restart window. An over-eager rotation check would reject good tokens routinely. That is a strictly worse trade, and I am not making it against a theory. I want a staged reproduction that captures the epoch from both storage and the persisted record at mint time, and shows them differing, before I touch the file.

So for now the bug is open, documented, and worse than the fix — on purpose. The guard is dead either way; at least now it is dead and labelled.

If you want to look: the server is safari-mcp, and the full analysis lives on issue #105.

The question I am genuinely stuck on: how do you catch a dead guard in your own code? Grepping for compared-but-never-assigned finds this exact shape, but it is one instance of a bigger family — the if that can never be true, the catch that can never be reached, the flag nobody flips. Has anyone found a check for that family that is cheap enough to run on every commit and quiet enough that people do not start ignoring it?

I Rotated 4 Secrets in 6 Days. 3 of Them Broke Production.

אחיה כהן — Mon, 07 Sep 2026 05:39:55 +0000

Every guide on secret rotation ends with "and then update the consumers." That sentence is where production dies.

I run two self-hosted n8n instances for an automation studio in Israel. This morning I counted what they hold: 114 credentials, referenced from 1,311 nodes across 260 workflows. The single most-used credential (a Postgres connection) sits in 233 nodes in 80 workflows, 40 of them active. The WhatsApp gateway key is in 106 nodes.

Between August 20 and August 25 I rotated four secrets. Three of them took something down. Each one failed in a different way, and none of the three failures was inside n8n. That is the part worth writing down.

Failure 1: the copy you forgot (2.5 days, zero alerts)

August 20. I rotated a Chatwoot API token. The token is used by a WAHA→Chatwoot bridge, and the bridge keeps its own copy of the token per inbox app. Six apps, six copies. I updated four.

The two inboxes on the old token went quiet. Inbound messages hit POST /contacts/filter, got 401 Invalid Access Token, and vanished. Outbound looked fine from the agent's side (the row was created in Chatwoot), then stalled on the same 401. No alert fired anywhere, because nothing was "down": the process was up, the endpoint answered, the queue just filled with failed jobs.

I noticed on August 23. Two and a half days.

The side effect was worse than the outage. Looking up a contact needs the token; creating one goes through a public endpoint that doesn't. So every inbound message during the outage created a fresh contact_inboxes row. When I finally counted, the table held 18,489 duplicates (some older, most from this), and those later produced 404s on update_last_seen, because the bridge picked one mapping and the conversation sat on another.

Fix that stuck: a daily check that validates every bridge app's token against Chatwoot, one line per app, non-zero exit if any fails. The hash comparison runs without printing a value: sha256(config.accountToken) against left(encode(sha256(token::bytea),'hex'),16) in access_tokens.

Failure 2: the derived value (14 hours of rejected calls)

Same night, August 20, 22:55. I rotated the signing key of the Hebrew AI call-answering service I run. The key signs magic links and admin cookies, which I knew. It also derives the URL token that Telnyx uses to deliver inbound calls to /voice/telnyx/<token>, which I had forgotten.

Telnyx kept posting to the old URL. Every inbound call got 403 bad URL token until 13:07 the next day. Six calls from four numbers, rejected by a service whose only job is answering calls.

Nothing in the rotation touched Telnyx, so nothing in it could have warned me. Three layers now: on every boot the service compares its webhook URL with what Telnyx has and corrects it; any 403 on the token triggers an immediate resync plus a Telegram alert; and a fallback URL at Telnyx points to an n8n workflow on a different server that plays an apology recording. Detection gap went from 14 hours to seconds. I verified that by firing a call at it.

Failure 3: the shadow override (858 × 401 in 38 hours)

August 23. WAHA API key. This one lives in six places: two lines in the .env, an nginx snippet that injects the key on the public media path, two n8n credentials, and the local keychain. I had a tested procedure for all six.

The procedure updated the nginx snippet. But the vhost file had a second, hardcoded proxy_set_header X-Api-Key <old value>; line inside the /api/files/ location, left over from before the snippet existed. The hardcoded line won. The snippet variable was correct, and nothing read it.

Result: 858 requests returned 401 over 38 hours, and every media message (voice notes, images, stickers) on five WhatsApp sessions stopped syncing to Chatwoot. In the inbox this shows up as "unsupported message type" notes, not as errors. The public vhost stayed green the whole time. Nothing on it fetches media except real clients.

The verification I use now: from inside the WAHA container, curl a real media path through the public domain. Localhost 200 plus public 401 means an injection layer is holding an old value.

The one that worked (4 destinations, verified in order)

August 25. A client website's webhook secret, after it leaked: the webhook trigger node stores the full request headers in execution data, X-Webhook-Secret included, and I had opened that execution to debug something else. Four destinations: a config row in Supabase, a file on the client's WordPress host, an n8n credential, and the keychain.

What was different: I wrote the verification before the rotation. New secret → the endpoint returns 200. Old secret → 401. Then wait for the two scheduled workflows that consume it (a 2-minute and a 5-minute cron) to log success. Only then delete the old value.

It took longer than the other three. It was also the only one where I didn't have to explain anything to a client.

The pattern

Three failures, three different mechanisms:

Failure	Where the stale copy lived	What was silent
Chatwoot token	2 of 6 bridge apps	Inbound messages dropped, no alert
Signing key	A value derived from it, registered at a third party	Inbound calls rejected at the carrier
WAHA key	A hardcoded override under the variable I updated	Media sync, while health stayed green

The rotation itself was never the problem. The inventory was. None of my consumer lists was complete, and two were wrong in ways that only showed up after the old value stopped working.

What I run before touching a credential now

The n8n side of the inventory is a query. Workflows store nodes as JSON, and each node with a credential carries credentials: { <type>: { id, name } }. This flattens that into a blast-radius table:

WITH n AS (
  SELECT w.id AS wid, w.active, w."isArchived" AS arch,
         jsonb_array_elements(w.nodes::jsonb) AS node
  FROM workflow_entity w
), c AS (
  SELECT wid, active, arch, e.key AS ctype, e.value->>'id' AS cid
  FROM n, LATERAL jsonb_each(node->'credentials') e
  WHERE node ? 'credentials'
)
SELECT c.cid, coalesce(ce.name, '<deleted>') AS name, c.ctype,
       count(*) AS nodes,
       count(DISTINCT wid) AS workflows,
       count(DISTINCT wid) FILTER (WHERE active AND NOT arch) AS active_workflows
FROM c LEFT JOIN credentials_entity ce ON ce.id = c.cid
GROUP BY 1, 2, 3
ORDER BY nodes DESC;

Names and counts only. It never touches the data column.

Three things it told me this morning that I didn't know:

8 credential IDs are referenced by workflows but no longer exist in credentials_entity. The <deleted> rows. All in inactive or archived workflows, but one of them is a Telegram credential still referenced from 43 nodes.
44 of the 100 credentials on the main instance were modified in the last 30 days. "We rotate rarely, so let's do it carefully" was the wrong mental model. We rotate constantly. It has to be routine.
The Postgres credential with 233 references is a single point of failure that no dashboard shows. If I ever rotate that password, the order of operations matters more than the password.

For the update itself: on n8n 2.36 the public API accepts PATCH /api/v1/credentials/{id} with a data object. I checked the route this morning with a body it had to ignore and got a 200 back. Ten days ago I was still creating a new credential, rewriting every node reference, and deleting the old one: three writes and a chance to miss a node. One PATCH, then confirm versionId equals activeVersionId on each consuming workflow, because a saved workflow is not necessarily the running one.

The secret you already leaked

One more thing from that fortnight, because it changed how I build the AI agents I build for small businesses.

An insurance agency's reply bot pulled its API key from a config table with a Postgres node: SELECT value FROM config WHERE key = 'api_key'. Clean, no hardcoding.

It sat in every execution instead. The node's output JSON contained the value, and n8n saves node output to execution_data. When I went looking: 980 rows held that key, and 649 held a second one (a Chatwoot token loaded the same way). Every time anyone opened an execution to debug it, the secret was on screen. That is how it leaked to me.

The fix is to keep the secret out of node output entirely. httpCustomAuth merges its body straight into the request (requestOptions.body = { ...requestOptions.body, ...customAuth.body } in HttpRequestV3), and httpHeaderAuth does the same for a header. Drop the column from the SELECT, attach the credential to the HTTP node, and the value never enters JSON. Then scrub what is already stored:

UPDATE execution_data
SET data = replace(data, :'sec', '***REDACTED***')
WHERE position(:'sec' IN data) > 0;

(Run it from a script that reads the value from stdin, so it doesn't land in your shell history either.)

The question

For your single most-referenced credential: how many copies of it exist outside the system that owns it? Bridges, proxies, a file on a client's host, a URL registered at a third party, a derived token.

If you have the number, I'd like to hear how you keep it current. If you don't have the number, that is the number.

The commit that fixed my only security advisory failed its security audit. The audit never ran.

אחיה כהן — Sun, 06 Sep 2026 05:45:07 +0000

On 4 September I cut two releases of safari-mcp ten minutes apart. The second one exists because of a mistake in the first, and its CI run went red for a reason that had nothing to do with either.

Here is the sequence, because the order is the point.

Release one: shipped with the alerts open

v2.18.0 was a real release: receipts that survive a reconnect, screenshot downscaling, a queue for parked extension workers. CI was green, 227 tests passed locally, the changelog was written. I tagged it.

What I had not done was look at the Dependabot tab. There was one open PR there, bumping qs from 6.15.3 to 6.16.0 to close two moderate advisories. Lockfile-only, CI green, sitting there for a day. I released around it. So the fresh version that landed on npm carried two advisories that anyone running npm install would see in the summary line.

I noticed within minutes, merged the bump, waited for CI on main, and cut v2.18.1 with a one-line ### Security entry. The release workflow ran, npm got the new tarball, the registry picked it up. Fine. Lesson written down: merge the green Dependabot PRs before cutting, not after.

Release two: red CI on the commit whose only job was security

Then the CI run for that release commit failed.

The workflow runs the test matrix on three Node versions. One of them failed at this step:

- name: Audit production dependencies (fail on high+)
  run: npm audit --audit-level=high --omit=dev

The log:

npm warn audit 503 Service Unavailable - POST https://registry.npmjs.org/-/npm/v1/security/audits/quick
npm error audit endpoint returned an error

The npm ci before it had taken seven minutes. npm was having a bad morning. The audit endpoint was unreachable, npm audit exited non-zero, the step failed, and because the matrix has fail-fast on by default, GitHub cancelled the other job that was still running. Final tally on the commit that fixed my only security advisory: one job passed, one failed, one cancelled. Red.

The same thing happened again that afternoon on the next commit, on a different Node version. Two days later I re-ran both without changing a line. All green.

What the gate actually measured

I put npm audit in CI so that a release cannot go out with a known high-severity vulnerability in production dependencies. That is a reasonable thing to want. But look at what the step can and cannot tell me:

Exit 0: the audit ran and found nothing at high or above.
Exit 1: the audit ran and found something — or the audit could not run at all.

Those two are not the same finding, and the step collapsed them into one red X. On 4 September the repository had zero open Dependabot alerts. There was nothing to find. The gate went red anyway, because it is a network call to a third party, and the third party was down.

A check that fails when it cannot run is not a gate. It is a coin flip weighted by someone else's uptime. And the weighting is not small: npm ci plus npm audit are two round trips to the same registry, in the same job, with no retry between them.

fail-fast then made it worse in the specific way that amplifiers do. One transient error on one Node version became three red jobs, two of which had no opinion about anything. The whole point of a matrix is to learn something per version. Cancelling the siblings on the first failure throws that information away exactly when it would have been useful, because the one thing a green sibling would have told me is "this is not your code".

Why nothing was stopped

Here is the part I am least comfortable with. The red CI did not block the release. The release workflow triggers on the tag, not on CI status, and I had cut the tag after checking that the previous commit was green. So v2.18.1 was on npm before its own CI had finished, and the CI then went red on a commit that was already published.

I had thought of CI as a gate. It was a lagging indicator. A gate that runs after the door has closed is documentation, and documentation that is wrong is worse than none — for two days the commit history said "this release failed CI", and it had not failed anything.

The fix

Two changes, both small, both shipped this morning:

strategy:
  fail-fast: false

so each Node version reports its own result, and the audit step now tells the two failure modes apart:

set -o pipefail
for attempt in 1 2 3; do
  if npm audit --audit-level=high --omit=dev 2>&1 | tee audit.log; then exit 0; fi
  grep -q "audit endpoint returned an error" audit.log || exit 1
  echo "npm audit endpoint unavailable (attempt $attempt/3) — retrying in 30s"
  sleep 30
done
exit 1

A real finding still fails immediately. An endpoint error retries three times, then fails — which is correct, because after ninety seconds of no answer I genuinely do not know whether the release is clean, and "I don't know" should be red. What I have removed is the case where a single 503 pretends to be a vulnerability.

The thing I have not fixed is the ordering problem: the release still does not wait for CI on its own commit. That is a bigger change to a workflow that has finally been reliable for two weeks, and I am not going to touch it on the same morning I am writing about touching things too fast.

The question

The two mistakes on 4 September are opposites. The first was a gate I forgot to look at (Dependabot). The second was a gate that looked at nothing and reported anyway (npm audit during an outage). I over-trusted one and under-checked the other, and the release that came out of it was fine — which is the uncomfortable bit, because "it was fine" is how these habits survive.

So: when a CI check depends on an external service, do you fail the build when that service is down, or skip the check and mark it? I chose "retry, then fail", on the argument that an unknown security state should be red. I can see the case for a yellow.

I Measured What Self-Hosted Chatwoot Actually Uses. 348,703 Messages, 1.8 GB of RAM.

אחיה כהן — Wed, 02 Sep 2026 16:49:33 +0000

Disclosure: I run a small automation shop, I set Chatwoot up for clients, and ACHIYAVS is my affiliate code — if you use it on Chatwoot Cloud you pay 5% less and I earn a commission. Self-hosting, which is most of this post, earns me nothing.

Every "Chatwoot pricing" article copies the same four numbers off the pricing page. None of them say what the free option actually consumes, because none of them are running it.

I am. So I measured mine.

The install

One box. Not a cluster, not a managed database, no autoscaling group.


Client accounts	8
Inboxes	20
Seats	34
Conversations, all time	10,321
Messages, all time	348,703
Messages, last 30 days	125,521
Collecting since	25 Nov 2025

What it uses

docker stats, steady state, mid-afternoon on a working day:

chatwoot-rails-1       827 MiB
chatwoot-postgres-1    595 MiB
chatwoot-sidekiq-1     368 MiB
chatwoot-redis-1         7 MiB
                     ---------
                      1.80 GiB

Postgres is 997 MB on disk. The whole box sits at 20 GB of 38 GB used and 2.4 GB of 3.7 GB RAM — and that box hosts other things too.

348,703 messages across 20 inboxes fit inside about 1.8 GB of RAM. A 4 GB VPS runs this without strain.

The structural point matters more than the number: that footprint tracks message volume, not seat count. Adding an agent to a self-hosted install costs nothing. On Cloud, every seat is another line on the bill, every month, for as long as you have it.

The line item nobody quotes

I went looking for what my install would cost on Cloud, and found the thing that actually decides the tier — and it is not the price.

Each Cloud tier caps how long your conversation history is kept. Thirty days on the free tier, six months on the cheapest paid one, longer above that.

My history goes back to 25 November 2025. That is 282 days. On the entry paid tier, roughly three months of client conversations would already be gone — not archived, gone.

A support inbox is a record. "You told me in March that this was included" is a question someone has to be able to answer. That constraint, not the sticker price, is what pushes most teams up a tier.

I put the full tier-by-tier breakdown, with the retention caps and the crossover math at different team sizes, on my own site: what Chatwoot actually costs. It is the version I wish I had found.

The cost that is not in anyone's table

Self-hosting is not free. It costs the thing hardest to price: your attention, at inconvenient hours.

In the months I have run this I have handled Postgres connection-pool exhaustion, a Redis eviction policy that quietly dropped background jobs, a log rotation config that corrupted an access log badly enough to break my own monitoring, and an upgrade that reverted files I had mounted into the container.

None of that was Chatwoot's fault. All of it was the price of the $0 licence.

If your team does not already have someone comfortable in a terminal at 11pm, the honest answer is that Cloud is cheaper for you than self-hosting — because the alternative is not "a server," it is "a server plus a person."

So which one

Self-host if you have ops capacity, more than a handful of seats, and you want history you control. The economics improve with every seat and never get worse.

Take Cloud if support is not your engineering problem and you would rather pay for that boundary. That is a legitimate purchase, not a defeat.

And if you take Cloud, the code is ACHIYAVS for 5% off — which is the disclosure at the top of this post, and the real number. Search for a Chatwoot code and you will find "70% off" and "$100 off" on half a dozen coupon sites. The published affiliate terms give the customer 5%. Anything larger is expired or was never real.

On 34 seats that discount is worth about $33 a month. It is not a reason to choose Cloud. It is just the correct number, which turns out to be surprisingly hard to find.

Figures measured on a production install, 2 September 2026.

If you self-host Chatwoot — what does your install actually use, and did the retention cap ever catch you out?

I Run n8n in Both Modes. Queue Mode's Peak Load: 19 Jobs.

אחיה כהן — Mon, 31 Aug 2026 05:42:13 +0000

I run two production n8n instances. Both sit on identical 2 vCPU / 4 GB VPSes. One runs queue mode with a dedicated worker, Redis, and Postgres. The other runs plain single-process mode.

Last week the queue-mode instance handled 76,100 executions. The regular one handled 322.

Here's the part that should change how you think about queue mode: at the single busiest moment of those 76,100 runs, exactly 19 executions were in flight at once. My worker runs with --concurrency=5. The box never broke a sweat.

Queue mode is the most recommended "scaling" step in every n8n thread, and most people who follow that advice are buying RAM and operational complexity they will never use. I have the numbers, and two production incidents, to show where the line actually is.

The numbers, queried this morning

Everything below comes from live SQL against both instances' Postgres, run on the day I wrote this (2026-08-31), plus docker stats snapshots.

	Instance A (queue mode)	Instance B (regular)
n8n version	2.36.7	2.36.7
Host	2 vCPU / 4 GB	2 vCPU / 4 GB
Executions, last 7 days	76,100	322
Active workflows	69	3
Success rate	99.65%	100%
Median run duration	0.20 s	n/a
p95 / p99 duration	1.59 s / 8.02 s	n/a
Longest single run	7,320 s (~2 h)	n/a
Peak concurrent executions	19	1
RAM: n8n processes	456 MiB main + 445 MiB worker	507 MiB
RAM: Redis	4 MiB	none

Two things jump out.

First, 88% of the retained executions on instance A finish in under one second. This is what real automation traffic looks like: webhooks arrive, a filter node rejects most of them, done. It is not a compute workload. It's a plumbing workload.

Second, the queue stack costs me roughly 900 MiB of n8n processes to do what the regular instance does in 507 MiB. Redis is a rounding error, but the worker is a whole second n8n. That's the price of admission, before you've gained anything.

What queue mode actually bought me

Not speed. A sub-second execution is sub-second in either mode. The queue adds a Redis hop; nobody notices it, but nobody gains from it either.

What it actually bought me, in order of how often it mattered:

1. Crash isolation. See that 7,320-second run in the table? That was an AI workflow hanging on a model call for two hours before dying with an error, and it wasn't alone: the three longest runs in my window are all AI workflows that hung and eventually errored. In regular mode, runs like that live inside the same process that serves your editor and your webhooks, and a memory-hungry one takes everything down with it. In queue mode each of them burned one worker slot out of five while everything else kept flowing. This is the real feature. It's an availability feature, not a performance feature.

2. Restart behavior. I can restart the worker (after an update, after a leak) without dropping incoming webhooks. The main process keeps accepting; the queue holds jobs until the worker returns.

3. A scaling path I've never used. If I ever need a second worker, it's one compose line. Peak of 19 in-flight against concurrency 5 says that day is far away. That in-flight count also includes parent workflows sitting idle waiting on sub-workflow calls, so the true CPU-busy number is lower still.

What queue mode charged me

The migration race. Every version upgrade, docker compose up -d starts main and worker together, and both immediately try to run database migrations on the same Postgres. n8n has no cross-instance migration lock. The main wins; the worker crashes mid-migration with a MigrationExecutor.executePendingMigrations stack trace and sits there unhealthy. First time it happened (2.30.6 → 2.32.6, during a routine fleet upgrade), the worker sat unhealthy while I worked out whether my database was half-migrated.

It wasn't. The fix is boring and permanent: wait for main to report healthy, then docker restart the worker. It rejoins cleanly because the migrations are already done. But nobody tells you this before you switch, and an unhealthy worker after an upgrade looks exactly like a disaster until you know it's choreography.

A second thing to monitor. A dead worker in queue mode fails quietly: webhooks still return 200, jobs still enqueue, and nothing executes. In regular mode, when n8n is down, everything visibly fails, which is ugly but honest. Queue mode converts loud failures into silent backlogs. You need a health check on the worker specifically, not just on the UI.

The incident that queue mode couldn't touch

The one time this instance actually hurt, no execution mode would have saved it.

One gateway workflow, a filter in front of a WhatsApp bot, was receiving every event on the line and rejecting most of them in its first node. 11,880 executions per day, 96% finishing under a second. Pure no-op traffic.

The damage wasn't CPU. The box was idle. The damage was writes: n8n persists every execution by default, and my pruning cap (EXECUTIONS_DATA_PRUNE_MAX_COUNT=50000) quietly became the effective retention window. At that rate, 50K rows is 2.5 days. Sixteen of my active workflows had zero saved executions left: a workflow failed three days earlier and there was no evidence it ever ran. Meanwhile execution_data grew to 92% of a 1.6 GB database and the nightly dumps inflated the backup directory to 9.8 GB.

The fix was two settings on one workflow: saveDataSuccessExecution: none, saveDataErrorExecution: all. You can see it in my daily counts: 18,238 saved executions on August 24, between 6,300 and 9,000 a day since. Same traffic, half the writes.

Queue mode has no opinion about any of this. Workers don't reduce writes; they just move where the writing happens. If your n8n feels heavy, check what it's storing before you scale what it's computing.

Where the line actually is

My rule after running both modes side by side:

Stay in regular mode while all of these are true: your runs are mostly short (check with the query below), a single stuck execution taking the UI down for a minute is survivable, and you're on one box anyway.

Switch to queue mode when any of these arrives: individual runs that go multi-minute or memory-heavy (my 2-hour hung AI call is the poster child), webhooks that must stay up while you restart things, or genuine horizontal scaling.

Notice what's not on the list: raw execution count. 76,100 a week fits through --concurrency=5 because almost all of it is sub-second. Volume is the wrong trigger. Duration and blast radius are the right ones.

Two queries to run against your own instance before you add containers:

-- What fraction of your runs are sub-second no-ops?
SELECT (EXTRACT(EPOCH FROM ("stoppedAt" - "startedAt")) < 1)::int AS under_1s,
       count(*)
FROM execution_entity
WHERE "stoppedAt" IS NOT NULL
GROUP BY 1;

-- Your true peak concurrency (sweep line over start/stop events)
WITH ev AS (
  SELECT "startedAt" AS t,  1 AS d FROM execution_entity
   WHERE "startedAt" IS NOT NULL AND "stoppedAt" IS NOT NULL
  UNION ALL
  SELECT "stoppedAt", -1 FROM execution_entity
   WHERE "startedAt" IS NOT NULL AND "stoppedAt" IS NOT NULL
)
SELECT max(c) FROM (SELECT sum(d) OVER (ORDER BY t, d DESC) AS c FROM ev) x;

If the first query says most of your load is sub-second, fix your filters before your architecture. If the second says your peak is under 10, queue mode is an availability decision, not a capacity one — make it for the crash isolation or don't make it at all.

These two instances run client-facing automation at the automation studio behind 69 production n8n workflows, so the "survivable downtime" bar is set by paying customers, not by my patience. That's why the busy instance got queue mode long before it needed a second worker.

What's your peak? Run the sweep query on your own instance and post the number. I am collecting evidence for a follow-up on how oversized most n8n deployments are. If you've crossed your worker concurrency in production, I especially want to hear what the workload was.

I Ran 89,479 WhatsApp Messages Through WAHA. Twilio: $604.

אחיה כהן — Thu, 27 Aug 2026 06:54:18 +0000

Last month my WhatsApp stack moved 89,479 messages. I got no invoice for any of them.

That is not a brag, it is the setup for an honest accounting. Because "self-hosting is cheaper" is the least interesting sentence in infrastructure, and it is usually said by someone who has never been paged at 7am by a bot that went quiet at 2am. I want to put a real number on both sides of that trade: the money Twilio would have charged, and the money self-hosting quietly takes back.

All the numbers below were pulled or fetched on August 27, 2026. The rate cards move quarterly, so check yours.

The traffic, measured rather than estimated

Five WhatsApp inboxes, bridged from WAHA into a self-hosted Chatwoot. Thirty days:

	messages
Total	89,479
Inbound (from users)	45,563
Outbound (from us)	43,916

Most benchmarks stop here, multiply by a per-message rate, and publish. That answer is wrong, because Meta does not charge per message. It charges per template sent outside an open customer service window. Multiplying my full 89,479 by a template rate overstates the Meta line by about 3x. Multiplying just the outbound half still overstates it by about 1.5x.

Since November 1, 2024 non-template messages are free. Since July 1, 2025 utility templates answering a user inside an open 24-hour window are also free. So the only line that costs money is the outbound message that goes out when nobody has written to you in the last day.

Which means the number you actually need is not "how many messages," it is "how many outbound messages had no inbound message from that contact in the preceding 24 hours."

The query that produces the real bill

Here it is against Chatwoot's schema. It uses a window function rather than a correlated NOT EXISTS, because on a messages table of any size the correlated version will happily eat your connection pool.

WITH src AS (
  SELECT m.conversation_id, m.created_at, m.message_type
  FROM messages m
  WHERE m.inbox_id IN (27, 23, 46, 50, 48)   -- your WhatsApp inboxes
    AND m.created_at > now() - interval '33 days'
    AND m.message_type IN (0, 1)             -- 0 = incoming, 1 = outgoing
), w AS (
  SELECT *,
    max(created_at) FILTER (WHERE message_type = 0) OVER (
      PARTITION BY conversation_id ORDER BY created_at
      ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
    ) AS last_in
  FROM src
)
SELECT
  count(*) FILTER (WHERE message_type = 1) AS outbound,
  count(*) FILTER (WHERE message_type = 1
        AND last_in IS NOT NULL
        AND created_at - last_in <= interval '24 hours') AS inside_window,
  count(*) FILTER (WHERE message_type = 1
        AND (last_in IS NULL
             OR created_at - last_in > interval '24 hours')) AS outside_window
FROM w
WHERE created_at > now() - interval '30 days';

Two details worth stealing: pull 33 days of history but count only the last 30, otherwise messages near the boundary look like they have no preceding inbound and you overstate your bill. And ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING is what keeps a message from counting itself as its own window opener.

My result:

 outbound | inside_window | outside_window
----------+---------------+----------------
    43916 |         14314 |          29602

So of 43,916 outbound messages, 29,602 (67%) would have been billable templates. The other third rode free inside an open conversation.

What that costs on Twilio

Twilio's WhatsApp pricing is two layers. Its own fee is $0.005 per message, inbound or outbound, and that one applies to all 89,479. On top sits Meta's template fee, passed straight through, which varies by the recipient's country calling code.

I'm in Israel. Meta's USD rate card effective July 1, 2026:

Template category	Israel	UK	Germany	India
Marketing	$0.0353	$0.0635	$0.1365	$0.0118
Utility	$0.0053	$0.0220	$0.0550	$0.0014
Authentication	$0.0053	$0.0220	$0.0550	$0.0014

Israel is a cheap market for utility and a mid market for marketing. Germany is over ten times Israel's utility rate. If you are benchmarking, your country's row is the whole story. A US-centric blog post about WhatsApp costs is nearly useless to you.

The month, priced out:

Twilio handling   89,479 × $0.005   = $447.40
Meta (utility)    29,602 × $0.0053  = $156.89
                                    ---------
                                      $604.29

Meta (marketing)  29,602 × $0.0353  = $1,044.95
                                    ---------
                                    $1,492.35

$604 if every billable template is a utility template. $1,492 if they are marketing. The gap between those two numbers is not a pricing question, it is a template-categorization question, and Meta decides the category, not you.

Scaled to the 10,000-messages-a-month shape people usually ask about, holding my ratios (49% outbound, 67% of that outside the window): $67.53 utility, $166.77 marketing.

The other side of the ledger changed in June, quietly

If you last looked at WAHA a year ago, you priced it wrong. WAHA used to sell a "Plus" license for the features anyone running it in production actually needs. In release 2026.6.1, announced June 21, 2026, every Plus feature moved into the free Core image and the tiers collapsed into a single optional $5/month Community subscription. The project's own words: "100% free and open source, with no limits on messages or time, and no license expiration."

So the software line is now genuinely $0.

The hardware line: WAHA is running on a 2 vCPU / 4 GB / 40 GB x86 VPS. It is not alone on that box: Redis and a second internal service share it. While carrying those 89,479 messages, the machine's load average was 0.05, 0.22, 0.23. This workload is not compute-bound, it is session-bound. Whatever your provider charges for the cheapest 4 GB instance, single digits a month, it does not move the comparison. Round the infra to $10 and self-hosting is still about 60× cheaper than the utility-rate scenario.

Which is exactly the point at which an honest post has to stop being a sales pitch.

What self-hosting actually charges you

It bills in incidents, not invoices. Here is last week's, in full.

On August 23 I rotated WAHA_API_KEY. The rotation updated /opt/waha/.env and the nginx snippet that injects the header. What it did not update was a second vhost, an internal tailnet-only copy created weeks earlier by duplicating the public one, where the key had been written as a literal instead of the $waha_api_key variable.

Result: every GET /api/files/… over the internal path started returning 401. Bots that download an image, OCR it, and act on the contents stopped being able to fetch anything. Of the three code nodes involved, one threw and killed its run outright; the other two caught the error and skipped silently, so those pipelines produced output — just output with no images in it. The public vhost was untouched, so every check that went through the domain came back green.

The failure surfaced the next morning, because nobody had sent a photo overnight. A one-line fix, roughly twelve hours of quiet wrongness, and a monitoring path that was structurally incapable of seeing it.

Twilio would have absorbed that entire class of problem. That is what the $604 buys. Not messages. The absence of a category of 2am.

And there is a risk that no amount of care removes. WAHA drives a real WhatsApp Web session. Its own homepage says it plainly: "WhatsApp does not allow bots or unofficial clients on their platform, so this shouldn't be considered totally safe." A banned number is not a support ticket, it is a dead asset. That is a business risk, not a technical one, and it does not belong in a cost table. It belongs in whatever conversation you have before you build.

The line I actually draw

After enough of these deployments, the rule I use with clients is not about volume at all:

The number is the business. A clinic's only phone line, a store's published contact. Official API, pay Twilio. The ban risk is unpriceable.
Internal, operational, or a secondary channel. Courier groups, staff notifications, an intake line that can be replaced in an afternoon. WAHA. The savings are real and the downside is survivable.
Marketing blasts at any volume. Official API, and check the marketing rate for your country before you promise anyone a budget. At $0.0353 a message in Israel and $0.1365 in Germany, the same campaign has wildly different economics depending on who receives it.

Volume is the wrong axis. Replaceability is the right one. This is roughly the split behind the pricing I quote Israeli businesses for WhatsApp automation: the channel decision comes first, and the bill follows from it.

One more thing worth knowing: Meta has further pricing changes landing October 1, 2026, covering service and utility messages. Whatever you calculate today has a shelf life of about five weeks.

Run the query

The number that decides this for you is not 89,479 and it is not $604. It is your own outside_window count, and it is the one input a vendor pricing page cannot give you.

So: run it against your own inboxes and post the three numbers. Outbound, inside window, outside window. I am specifically curious whether the 67% outside-window share holds anywhere else, or whether it is an artifact of how my bots are scheduled. If that ratio is stable across use cases, it is a much better rule of thumb than anything on a vendor pricing page.

I Audited 50,669 n8n Runs. 85% of Failures Alerted Nobody.

אחיה כהן — Mon, 24 Aug 2026 05:40:54 +0000

One of my own workflows — the gate that decides whether a new WhatsApp contact gets the bot or gets me — failed 24 times between Thursday and Sunday. Seventy-one hours. I found out on Monday morning, by opening the executions list for an unrelated reason.

That is embarrassing for someone who sells automation. So instead of fixing the one workflow and moving on, I went and measured the actual state of error handling across all 225 workflows on my main production instance. Here is every execution my instance still retains — pruning is on, so this is a six-day window, 18–24 August:

Metric	Value
Executions (6 days)	50,669
Failed executions (`error` + `crashed`)	106
Failure rate	0.21%
Active workflows	60
Active workflows with an error workflow attached	26 (43%)
Failures that occurred in a workflow with no error handling	90 of 106 (85%)

n8n 2.35.5, queue mode, Postgres. A 0.21% failure rate is fine. The 85% is not.

Why silence is the default

Error handling in n8n is opt-in, per workflow. You build one workflow whose trigger is the Error Trigger node, then in every other workflow you open Settings → Error Workflow and point it at that one. There is no switch that applies it to everything you have already built, or to everything you build next.

That is a per-workflow checkbox on a list that grows every week. Nobody forgets on purpose. You just build workflow #61 at 11pm, it works, you activate it, and you move on. The failure mode of an opt-in safety net is that coverage decays quietly while the number in your head stays at "yeah, we have error handling."

Mine had decayed to 43%.

What the gap actually cost

I pulled the failures grouped by workflow, with a column for whether that workflow had an error handler attached:

Workflow	Guarded	Failures	Window
WhatsApp group lead detection (AI)	no	34	36.5 h
Personal-line bot gate	no	24	70.9 h
Client error-handling workflow	no	13	1.2 h
Campaign-failure watchdog	no	11	8.7 h
Website lead intake form	no	4	53.0 h
SMS dispatcher (outbox)	yes	8	1.2 h
Appointment sync + no-show recovery	yes	4	1.1 h
6 others	mixed	1–2 each	~0 h

Read the last column, not the failure counts. Every window longer than nine hours belongs to an unguarded workflow. The longest is 70.9 hours — a WhatsApp bot gate that failed 24 times over three days while I was working on other things. Nothing was on fire. No customer wrote in. The bot simply did not answer new contacts, and the only reason I found it is that I went looking.

The guarded workflows failed too — 16 times between them. Their longest window was 1.2 hours, because something told me.

That is the whole value proposition of the Error Trigger, and it is not "fewer failures." It is failures that end in hours instead of days.

The part that actually scared me

Three of my workflows contain an Error Trigger node. Those are the handlers — the things that send the Telegram alert. Here is how many active workflows depend on each one:

Handler	Workflows it guards	Guarded itself?
Client error handler	15	no
Generic error alert	6	no
Product owner alert	5	no

Every handler was unguarded. And the one covering 15 workflows is the same row from the table above: it failed 13 times inside that same six-day window.

For those 13 failures, 15 workflows had no alerting at all and no way to find out. The error handler cannot report its own errors, because the thing that reports errors is the error handler. Quis custodiet ipsos custodes, in YAML.

You cannot fix this by pointing handler A at handler B, either. That just moves the single point of silence one hop and adds a cycle you will forget about. The catcher has to be watched from outside n8n.

The pattern, in four parts

1. One handler, not one per project. Fan-in beats fan-out. A single Error Trigger workflow that formats {{ $json.workflow.name }}, {{ $json.execution.id }} and {{ $json.execution.error.message }} into one alert is easier to keep correct than six near-copies.

2. Audit coverage with a query, not with your memory. This is the whole audit — run it against your n8n Postgres:

SELECT w.name,
       (w.settings->>'errorWorkflow' IS NOT NULL
        AND w.settings->>'errorWorkflow' <> '') AS guarded
FROM workflow_entity w
WHERE w.active = true
ORDER BY guarded, w.name;

Everything with guarded = f is a workflow that can fail into the void. If you are on n8n Cloud, the same check runs off GET /api/v1/workflows?active=true and reading settings.errorWorkflow per item.

That query is now the first line of my weekly runbook, next to the backup check. A new f row gets treated the way a failing test does.

3. Make the handler loud about workflows, not just about errors. The first thing I changed in mine was putting the workflow name in the subject line instead of in the body. That sounds trivial. It is the difference between "an automation failed" (which you snooze) and "the lead intake failed" (which you do not).

4. Guard the guard from outside. The Error Trigger has a blind spot that no amount of coverage fixes: it fires when an execution fails. It does not fire when an execution never starts — a deactivated trigger, a dead cron, a webhook whose URL changed, a worker that is not consuming the queue. A workflow that stopped running entirely produces zero failed executions and therefore zero alerts. It looks exactly like a quiet week.

So the handler pings a dead man's switch on every run, and a plain cron on a different machine screams if the ping stops:

# in the error handler workflow, and in one heartbeat workflow per instance
curl -fsS -m 10 "https://your-monitor/ping/$SLUG" || true

# elsewhere — not on the n8n box
LAST=$(stat -f %m /var/lib/heartbeat/n8n1 2>/dev/null || echo 0)
if [ $(( $(date +%s) - LAST )) -gt 3600 ]; then
  notify "n8n1 heartbeat is $(( ($(date +%s) - LAST) / 60 ))m stale"
fi

The || true matters: a monitoring call that can fail the run it is monitoring is worse than no monitoring.

That is it. Four parts, none clever. The reason it is worth writing down is that I had parts 1 and 3 for as long as I have run this instance and still ate a three-day outage, because I never had part 2 and never imagined I needed part 4.

Where I landed

I am not going to pretend this is finished. As I write this, coverage is still 43% — what changed is that I now have the number, the query that produces it, and a list of 34 workflows in the order I am going to fix them, longest silent window first. The handler-watching-the-handler problem is the part I am fixing today, because it is one workflow and it covers fifteen.

The honest reason I am writing this before finishing it: the audit took twenty minutes and the three-day outage took three days. Those twenty minutes are the cheapest thing in the entire stack I maintain — cheaper than the WhatsApp bots I run in production for clients, and a rounding error against the numbers in this breakdown of what business automation actually costs. Run the query before you finish reading this post and you will probably learn something uncomfortable too.

One genuine question, because I do not think I have solved this part: how do you monitor the trigger that never fires? A heartbeat proves the instance is alive and proves that workflow ran. It does not prove that a webhook from a third party still arrives, or that a specific cron survived the last upgrade. Per-workflow "expected minimum run rate" alerting is the obvious answer and it also sounds like a maintenance burden that decays exactly like error-workflow coverage did.

If you run n8n in production: do you alert on absence of runs, and if so — how do you keep the thresholds from rotting?

DEV Community: אחיה כהן

My sudo Logs Hit 10 GB. Session Count Told Me Nothing.

The measurement

Why bytes and sessions diverge

The fix is an exemption, not a shutdown

What ten days later looks like

The part I'd actually change

The metric to steal

My Monitoring Cron Never Ran Once. crontab -l Showed It Fine.

The setup

The first lie: crontab -l

The second lie: a log full of runs

The tell: one line in syslog

The root cause: who opens the >>

The fix

What I actually changed

Your turn

I Audited 38 Production Containers. Zero Use depends_on restart.

The measurement

What depends_on actually promises

Where condition alone was not enough

The 31 with no depends_on

What I changed

The question

I Measured 27,257 MCP Connections. The p90 Session Waits 35s.

What I measured

The headline numbers

Cause #1: it's OAuth, not your server

Cause #2: transport, with a twist

Cause #3: nobody connects to one server

The 29.7% nobody talks about

The trap I almost published

What actually helps

One question

3 of 7 WhatsApp Lines Died in 9 Days. Docker Said 'Healthy'.

The health check was answering the wrong question

The failure that looks like nothing at all

You cannot always tell whose line just died

The watchdog that never ran

What I check now

The question I am actually asking

My safety guard was checked in 6 places. Nothing ever set it.

Ruling things out

The actual mechanism

The part that actually bothers me

Why it survived review, including mine

What would have caught it

What I have not done

I Rotated 4 Secrets in 6 Days. 3 of Them Broke Production.

Failure 1: the copy you forgot (2.5 days, zero alerts)

Failure 2: the derived value (14 hours of rejected calls)

Failure 3: the shadow override (858 × 401 in 38 hours)

The one that worked (4 destinations, verified in order)

The pattern

What I run before touching a credential now

The secret you already leaked

The question

The commit that fixed my only security advisory failed its security audit. The audit never ran.

Release one: shipped with the alerts open

Release two: red CI on the commit whose only job was security

What the gate actually measured

Why nothing was stopped

The fix

The question

I Measured What Self-Hosted Chatwoot Actually Uses. 348,703 Messages, 1.8 GB of RAM.

The install

What it uses

The line item nobody quotes

The cost that is not in anyone's table

So which one

I Run n8n in Both Modes. Queue Mode's Peak Load: 19 Jobs.

The numbers, queried this morning

What queue mode actually bought me

What queue mode charged me

The incident that queue mode couldn't touch

Where the line actually is

I Ran 89,479 WhatsApp Messages Through WAHA. Twilio: $604.

The traffic, measured rather than estimated

The query that produces the real bill

What that costs on Twilio

The first lie: `crontab -l`

The root cause: who opens the `>>`