At 08:33 this morning I ran GET /api/sessions against my own WhatsApp gateway and read the line I least wanted to read:
{
"name": "session_<redacted>",
"status": "FAILED",
"me": { "pushName": "<insurance agency>", "id": "<redacted>@c.us" },
"timestamps": { "activity": 1789052954786 }
}
That timestamp is 10 September, 18:09. The check ran on 14 September. The line had been dead for eighty-six hours, and every dashboard I own was green.
I run WhatsApp automation for Israeli small businesses. Each client's business number lives as a session inside a self-hosted WAHA container, which bridges their line into a CRM and a pile of n8n workflows. Seven sessions, seven businesses, one container.
Between 4 and 13 September, three of those seven lines went down. Here is what each one taught me, and what I check now.
The health check was answering the wrong question
My monitor checked that the waha container was running and responsive. It was. It stayed running, responsive and healthy through all three outages, because a container that is perfectly happy to serve you a list of dead sessions is, by every definition Docker cares about, working.
This is the ordinary liveness-versus-readiness mistake, except the readiness that mattered was not the service's. It was the readiness of seven independent things inside the service, each of which can fail on its own, for unrelated reasons, without the process noticing or caring.
One client's line failed on 4 September at 20:47. I found it on the 8th. Three and a half days, during which the container reported itself healthy roughly five hundred times.
The failure that looks like nothing at all
The second one is the reason I am writing this.
On 5 September at 11:04 the gateway logged a device removed stream error for a clinic's line. The linked device had been removed from the client's phone — deliberately, or while clearing out old sessions. I never found out which, and it does not matter.
What matters is what the clinic saw afterwards: nothing. Messages kept arriving on the owner's phone exactly as before. Customers got replies, because a human was reading the phone. From the business's side the line was completely, visibly fine.
Only the CRM went blind. Only the automations stopped firing. The reminders that go out before appointments, the routing, the logging, the follow-ups — all silently not happening, behind a line that looked perfectly alive to the only person in a position to notice.
If your failure mode is invisible to the user and invisible to your health check, you do not have a monitoring gap. You have no monitoring.
You cannot always tell whose line just died
When I sat down to write the alerting, I hit a problem I had not anticipated: a dead session may not be able to tell you who it belongs to.
While a session is WORKING, /api/sessions gives you me.id (the number) and me.pushName (the business name). When a session is torn down by an unlink, those fields are wiped. The identity disappears at precisely the moment you need it to address an alert.
I want to be careful here, because I wrote this down as an absolute rule in my own notes last week and it is not one. Checking the live API this morning while drafting this post, the currently-failed insurance line does still carry its me block, as you can see in the JSON above. So it depends on how the session died: a removed device wipes the identity, a connection-level failure does not.
Which is worse than either rule on its own, because it means you cannot depend on the field being there. The fix is the same either way: keep your own roster. My watcher rewrites a roster.json on every pass — but only from sessions currently in WORKING, because a single pass during an outage would otherwise overwrite good identity with nulls.
And you cannot recover the mapping from billing, either. I scanned all 141 customers in our accounting system looking for these business numbers. Zero matches. What sits in an invoice is the owner's personal mobile, which is a different number from the business line the bot runs on. The mapping is manual, or it does not exist.
The watchdog that never ran
On 8 September I built the thing that should have caught all of this: a watcher on a ten-minute cron that diffs session states, notifies the business owner when their own line drops, and sends them a pairing code to reconnect it themselves.
I tested it. Ten test scenarios, three real bugs caught before install. I watched it run. I marked the problem solved.
It did not run once from cron until 13 September.
*/10 * * * * sudo /usr/local/bin/waha-session-watch >> /opt/waha-watch/cron.log 2>&1
/opt/waha-watch is owned by root, mode 755. The cron user is waha. The redirect is evaluated by the shell before sudo runs, as the unprivileged user, so creating cron.log failed, so the shell exited, so the command never executed at all. Cron dutifully tried to mail me the error and found no MTA installed, so it dropped it:
(CRON) info (No MTA installed, discarding output)
Five days of no monitoring, hidden behind a crontab entry that reads perfectly.
What fooled me during the install was watch.log. It had runs in it — 21:40, 21:52, 21:59, 22:22. Non-round minutes, every one of them, because they were my own manual test runs. Scheduled runs land on round minutes. I looked at a log full of evidence that the tool worked and read it as evidence that the schedule worked.
The insurance line fell on the 10th, into that exact blind window. The first genuinely scheduled run, on the night of the 13th, found it immediately — three days late.
The fix was touch cron.log && chown waha:waha. That is the entire fix. The lesson costs more than the fix: that No MTA installed, discarding output line in syslog is never noise. It means your scheduled command printed something nobody has ever read.
What I check now
Four things changed:
Health-check the units, not the process. The container being up says nothing about the seven sessions inside it. Whatever your service multiplexes — tenants, sessions, connections, device links — that is the thing with a state worth alerting on.
Verify the first scheduled run, in two places. Not "is it in crontab -l", which was true and meaningless for five days. Check syslog for the CMD line, and check the tool's own log for an entry on a round minute. If a cron entry redirects to a file, create the file first, owned by the cron user.
Keep a persistent roster, written only from healthy state. Never let a failure pass overwrite identity you will need to send the alert.
Alert the person who can actually fix it. A dropped line is repaired by someone holding the phone, not by me. The watcher now messages the business owner directly with a pairing code, inside working hours, once per incident. The one-message-per-incident rule matters as much as the alert: the fastest way to get a monitoring channel ignored is to send the same thing every ten minutes.
It matters that this is WhatsApp specifically. Bezeq's 2025 Internet Report — the annual survey of Israeli internet use, published in December 2025 — notes in passing that reaching 90% WhatsApp usage in Israel took about a decade. For a small business here, that line is not a channel. It is the front door. Which is exactly why the customer-service bots running on those lines need a liveness signal independent of the phone in the owner's pocket, because the owner's pocket will never report the outage. It does not look like one from there.
Three lines, nine days, one container that never once said anything was wrong.
The question I am actually asking
I want to hear about the second category specifically — not the outage your monitoring missed, but the one where the user experience stayed completely normal while the system underneath was dead, so nobody had any reason to report it.
What was the failure, and what signal finally caught it? I am collecting these, because I suspect the honest answer for most of them is "a human noticed weeks later," and I would like to be proven wrong.
Top comments (0)