DEV Community: אחיה כהן

I Measured What Self-Hosted Chatwoot Actually Uses. 348,703 Messages, 1.8 GB of RAM.

אחיה כהן — Wed, 02 Sep 2026 16:49:33 +0000

Disclosure: I run a small automation shop, I set Chatwoot up for clients, and ACHIYAVS is my affiliate code — if you use it on Chatwoot Cloud you pay 5% less and I earn a commission. Self-hosting, which is most of this post, earns me nothing.

Every "Chatwoot pricing" article copies the same four numbers off the pricing page. None of them say what the free option actually consumes, because none of them are running it.

I am. So I measured mine.

The install

One box. Not a cluster, not a managed database, no autoscaling group.


Client accounts	8
Inboxes	20
Seats	34
Conversations, all time	10,321
Messages, all time	348,703
Messages, last 30 days	125,521
Collecting since	25 Nov 2025

What it uses

docker stats, steady state, mid-afternoon on a working day:

chatwoot-rails-1       827 MiB
chatwoot-postgres-1    595 MiB
chatwoot-sidekiq-1     368 MiB
chatwoot-redis-1         7 MiB
                     ---------
                      1.80 GiB

Postgres is 997 MB on disk. The whole box sits at 20 GB of 38 GB used and 2.4 GB of 3.7 GB RAM — and that box hosts other things too.

348,703 messages across 20 inboxes fit inside about 1.8 GB of RAM. A 4 GB VPS runs this without strain.

The structural point matters more than the number: that footprint tracks message volume, not seat count. Adding an agent to a self-hosted install costs nothing. On Cloud, every seat is another line on the bill, every month, for as long as you have it.

The line item nobody quotes

I went looking for what my install would cost on Cloud, and found the thing that actually decides the tier — and it is not the price.

Each Cloud tier caps how long your conversation history is kept. Thirty days on the free tier, six months on the cheapest paid one, longer above that.

My history goes back to 25 November 2025. That is 282 days. On the entry paid tier, roughly three months of client conversations would already be gone — not archived, gone.

A support inbox is a record. "You told me in March that this was included" is a question someone has to be able to answer. That constraint, not the sticker price, is what pushes most teams up a tier.

I put the full tier-by-tier breakdown, with the retention caps and the crossover math at different team sizes, on my own site: what Chatwoot actually costs. It is the version I wish I had found.

The cost that is not in anyone's table

Self-hosting is not free. It costs the thing hardest to price: your attention, at inconvenient hours.

In the months I have run this I have handled Postgres connection-pool exhaustion, a Redis eviction policy that quietly dropped background jobs, a log rotation config that corrupted an access log badly enough to break my own monitoring, and an upgrade that reverted files I had mounted into the container.

None of that was Chatwoot's fault. All of it was the price of the $0 licence.

If your team does not already have someone comfortable in a terminal at 11pm, the honest answer is that Cloud is cheaper for you than self-hosting — because the alternative is not "a server," it is "a server plus a person."

So which one

Self-host if you have ops capacity, more than a handful of seats, and you want history you control. The economics improve with every seat and never get worse.

Take Cloud if support is not your engineering problem and you would rather pay for that boundary. That is a legitimate purchase, not a defeat.

And if you take Cloud, the code is ACHIYAVS for 5% off — which is the disclosure at the top of this post, and the real number. Search for a Chatwoot code and you will find "70% off" and "$100 off" on half a dozen coupon sites. The published affiliate terms give the customer 5%. Anything larger is expired or was never real.

On 34 seats that discount is worth about $33 a month. It is not a reason to choose Cloud. It is just the correct number, which turns out to be surprisingly hard to find.

Figures measured on a production install, 2 September 2026.

If you self-host Chatwoot — what does your install actually use, and did the retention cap ever catch you out?

I Run n8n in Both Modes. Queue Mode's Peak Load: 19 Jobs.

אחיה כהן — Mon, 31 Aug 2026 05:42:13 +0000

I run two production n8n instances. Both sit on identical 2 vCPU / 4 GB VPSes. One runs queue mode with a dedicated worker, Redis, and Postgres. The other runs plain single-process mode.

Last week the queue-mode instance handled 76,100 executions. The regular one handled 322.

Here's the part that should change how you think about queue mode: at the single busiest moment of those 76,100 runs, exactly 19 executions were in flight at once. My worker runs with --concurrency=5. The box never broke a sweat.

Queue mode is the most recommended "scaling" step in every n8n thread, and most people who follow that advice are buying RAM and operational complexity they will never use. I have the numbers, and two production incidents, to show where the line actually is.

The numbers, queried this morning

Everything below comes from live SQL against both instances' Postgres, run on the day I wrote this (2026-08-31), plus docker stats snapshots.

	Instance A (queue mode)	Instance B (regular)
n8n version	2.36.7	2.36.7
Host	2 vCPU / 4 GB	2 vCPU / 4 GB
Executions, last 7 days	76,100	322
Active workflows	69	3
Success rate	99.65%	100%
Median run duration	0.20 s	n/a
p95 / p99 duration	1.59 s / 8.02 s	n/a
Longest single run	7,320 s (~2 h)	n/a
Peak concurrent executions	19	1
RAM: n8n processes	456 MiB main + 445 MiB worker	507 MiB
RAM: Redis	4 MiB	none

Two things jump out.

First, 88% of the retained executions on instance A finish in under one second. This is what real automation traffic looks like: webhooks arrive, a filter node rejects most of them, done. It is not a compute workload. It's a plumbing workload.

Second, the queue stack costs me roughly 900 MiB of n8n processes to do what the regular instance does in 507 MiB. Redis is a rounding error, but the worker is a whole second n8n. That's the price of admission, before you've gained anything.

What queue mode actually bought me

Not speed. A sub-second execution is sub-second in either mode. The queue adds a Redis hop; nobody notices it, but nobody gains from it either.

What it actually bought me, in order of how often it mattered:

1. Crash isolation. See that 7,320-second run in the table? That was an AI workflow hanging on a model call for two hours before dying with an error, and it wasn't alone: the three longest runs in my window are all AI workflows that hung and eventually errored. In regular mode, runs like that live inside the same process that serves your editor and your webhooks, and a memory-hungry one takes everything down with it. In queue mode each of them burned one worker slot out of five while everything else kept flowing. This is the real feature. It's an availability feature, not a performance feature.

2. Restart behavior. I can restart the worker (after an update, after a leak) without dropping incoming webhooks. The main process keeps accepting; the queue holds jobs until the worker returns.

3. A scaling path I've never used. If I ever need a second worker, it's one compose line. Peak of 19 in-flight against concurrency 5 says that day is far away. That in-flight count also includes parent workflows sitting idle waiting on sub-workflow calls, so the true CPU-busy number is lower still.

What queue mode charged me

The migration race. Every version upgrade, docker compose up -d starts main and worker together, and both immediately try to run database migrations on the same Postgres. n8n has no cross-instance migration lock. The main wins; the worker crashes mid-migration with a MigrationExecutor.executePendingMigrations stack trace and sits there unhealthy. First time it happened (2.30.6 → 2.32.6, during a routine fleet upgrade), the worker sat unhealthy while I worked out whether my database was half-migrated.

It wasn't. The fix is boring and permanent: wait for main to report healthy, then docker restart the worker. It rejoins cleanly because the migrations are already done. But nobody tells you this before you switch, and an unhealthy worker after an upgrade looks exactly like a disaster until you know it's choreography.

A second thing to monitor. A dead worker in queue mode fails quietly: webhooks still return 200, jobs still enqueue, and nothing executes. In regular mode, when n8n is down, everything visibly fails, which is ugly but honest. Queue mode converts loud failures into silent backlogs. You need a health check on the worker specifically, not just on the UI.

The incident that queue mode couldn't touch

The one time this instance actually hurt, no execution mode would have saved it.

One gateway workflow, a filter in front of a WhatsApp bot, was receiving every event on the line and rejecting most of them in its first node. 11,880 executions per day, 96% finishing under a second. Pure no-op traffic.

The damage wasn't CPU. The box was idle. The damage was writes: n8n persists every execution by default, and my pruning cap (EXECUTIONS_DATA_PRUNE_MAX_COUNT=50000) quietly became the effective retention window. At that rate, 50K rows is 2.5 days. Sixteen of my active workflows had zero saved executions left: a workflow failed three days earlier and there was no evidence it ever ran. Meanwhile execution_data grew to 92% of a 1.6 GB database and the nightly dumps inflated the backup directory to 9.8 GB.

The fix was two settings on one workflow: saveDataSuccessExecution: none, saveDataErrorExecution: all. You can see it in my daily counts: 18,238 saved executions on August 24, between 6,300 and 9,000 a day since. Same traffic, half the writes.

Queue mode has no opinion about any of this. Workers don't reduce writes; they just move where the writing happens. If your n8n feels heavy, check what it's storing before you scale what it's computing.

Where the line actually is

My rule after running both modes side by side:

Stay in regular mode while all of these are true: your runs are mostly short (check with the query below), a single stuck execution taking the UI down for a minute is survivable, and you're on one box anyway.

Switch to queue mode when any of these arrives: individual runs that go multi-minute or memory-heavy (my 2-hour hung AI call is the poster child), webhooks that must stay up while you restart things, or genuine horizontal scaling.

Notice what's not on the list: raw execution count. 76,100 a week fits through --concurrency=5 because almost all of it is sub-second. Volume is the wrong trigger. Duration and blast radius are the right ones.

Two queries to run against your own instance before you add containers:

-- What fraction of your runs are sub-second no-ops?
SELECT (EXTRACT(EPOCH FROM ("stoppedAt" - "startedAt")) < 1)::int AS under_1s,
       count(*)
FROM execution_entity
WHERE "stoppedAt" IS NOT NULL
GROUP BY 1;

-- Your true peak concurrency (sweep line over start/stop events)
WITH ev AS (
  SELECT "startedAt" AS t,  1 AS d FROM execution_entity
   WHERE "startedAt" IS NOT NULL AND "stoppedAt" IS NOT NULL
  UNION ALL
  SELECT "stoppedAt", -1 FROM execution_entity
   WHERE "startedAt" IS NOT NULL AND "stoppedAt" IS NOT NULL
)
SELECT max(c) FROM (SELECT sum(d) OVER (ORDER BY t, d DESC) AS c FROM ev) x;

If the first query says most of your load is sub-second, fix your filters before your architecture. If the second says your peak is under 10, queue mode is an availability decision, not a capacity one — make it for the crash isolation or don't make it at all.

These two instances run client-facing automation at the automation studio behind 69 production n8n workflows, so the "survivable downtime" bar is set by paying customers, not by my patience. That's why the busy instance got queue mode long before it needed a second worker.

What's your peak? Run the sweep query on your own instance and post the number. I am collecting evidence for a follow-up on how oversized most n8n deployments are. If you've crossed your worker concurrency in production, I especially want to hear what the workload was.

I Ran 89,479 WhatsApp Messages Through WAHA. Twilio: $604.

אחיה כהן — Thu, 27 Aug 2026 06:54:18 +0000

Last month my WhatsApp stack moved 89,479 messages. I got no invoice for any of them.

That is not a brag, it is the setup for an honest accounting. Because "self-hosting is cheaper" is the least interesting sentence in infrastructure, and it is usually said by someone who has never been paged at 7am by a bot that went quiet at 2am. I want to put a real number on both sides of that trade: the money Twilio would have charged, and the money self-hosting quietly takes back.

All the numbers below were pulled or fetched on August 27, 2026. The rate cards move quarterly, so check yours.

The traffic, measured rather than estimated

Five WhatsApp inboxes, bridged from WAHA into a self-hosted Chatwoot. Thirty days:

	messages
Total	89,479
Inbound (from users)	45,563
Outbound (from us)	43,916

Most benchmarks stop here, multiply by a per-message rate, and publish. That answer is wrong, because Meta does not charge per message. It charges per template sent outside an open customer service window. Multiplying my full 89,479 by a template rate overstates the Meta line by about 3x. Multiplying just the outbound half still overstates it by about 1.5x.

Since November 1, 2024 non-template messages are free. Since July 1, 2025 utility templates answering a user inside an open 24-hour window are also free. So the only line that costs money is the outbound message that goes out when nobody has written to you in the last day.

Which means the number you actually need is not "how many messages," it is "how many outbound messages had no inbound message from that contact in the preceding 24 hours."

The query that produces the real bill

Here it is against Chatwoot's schema. It uses a window function rather than a correlated NOT EXISTS, because on a messages table of any size the correlated version will happily eat your connection pool.

WITH src AS (
  SELECT m.conversation_id, m.created_at, m.message_type
  FROM messages m
  WHERE m.inbox_id IN (27, 23, 46, 50, 48)   -- your WhatsApp inboxes
    AND m.created_at > now() - interval '33 days'
    AND m.message_type IN (0, 1)             -- 0 = incoming, 1 = outgoing
), w AS (
  SELECT *,
    max(created_at) FILTER (WHERE message_type = 0) OVER (
      PARTITION BY conversation_id ORDER BY created_at
      ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
    ) AS last_in
  FROM src
)
SELECT
  count(*) FILTER (WHERE message_type = 1) AS outbound,
  count(*) FILTER (WHERE message_type = 1
        AND last_in IS NOT NULL
        AND created_at - last_in <= interval '24 hours') AS inside_window,
  count(*) FILTER (WHERE message_type = 1
        AND (last_in IS NULL
             OR created_at - last_in > interval '24 hours')) AS outside_window
FROM w
WHERE created_at > now() - interval '30 days';

Two details worth stealing: pull 33 days of history but count only the last 30, otherwise messages near the boundary look like they have no preceding inbound and you overstate your bill. And ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING is what keeps a message from counting itself as its own window opener.

My result:

 outbound | inside_window | outside_window
----------+---------------+----------------
    43916 |         14314 |          29602

So of 43,916 outbound messages, 29,602 (67%) would have been billable templates. The other third rode free inside an open conversation.

What that costs on Twilio

Twilio's WhatsApp pricing is two layers. Its own fee is $0.005 per message, inbound or outbound, and that one applies to all 89,479. On top sits Meta's template fee, passed straight through, which varies by the recipient's country calling code.

I'm in Israel. Meta's USD rate card effective July 1, 2026:

Template category	Israel	UK	Germany	India
Marketing	$0.0353	$0.0635	$0.1365	$0.0118
Utility	$0.0053	$0.0220	$0.0550	$0.0014
Authentication	$0.0053	$0.0220	$0.0550	$0.0014

Israel is a cheap market for utility and a mid market for marketing. Germany is over ten times Israel's utility rate. If you are benchmarking, your country's row is the whole story. A US-centric blog post about WhatsApp costs is nearly useless to you.

The month, priced out:

Twilio handling   89,479 × $0.005   = $447.40
Meta (utility)    29,602 × $0.0053  = $156.89
                                    ---------
                                      $604.29

Meta (marketing)  29,602 × $0.0353  = $1,044.95
                                    ---------
                                    $1,492.35

$604 if every billable template is a utility template. $1,492 if they are marketing. The gap between those two numbers is not a pricing question, it is a template-categorization question, and Meta decides the category, not you.

Scaled to the 10,000-messages-a-month shape people usually ask about, holding my ratios (49% outbound, 67% of that outside the window): $67.53 utility, $166.77 marketing.

The other side of the ledger changed in June, quietly

If you last looked at WAHA a year ago, you priced it wrong. WAHA used to sell a "Plus" license for the features anyone running it in production actually needs. In release 2026.6.1, announced June 21, 2026, every Plus feature moved into the free Core image and the tiers collapsed into a single optional $5/month Community subscription. The project's own words: "100% free and open source, with no limits on messages or time, and no license expiration."

So the software line is now genuinely $0.

The hardware line: WAHA is running on a 2 vCPU / 4 GB / 40 GB x86 VPS. It is not alone on that box: Redis and a second internal service share it. While carrying those 89,479 messages, the machine's load average was 0.05, 0.22, 0.23. This workload is not compute-bound, it is session-bound. Whatever your provider charges for the cheapest 4 GB instance, single digits a month, it does not move the comparison. Round the infra to $10 and self-hosting is still about 60× cheaper than the utility-rate scenario.

Which is exactly the point at which an honest post has to stop being a sales pitch.

What self-hosting actually charges you

It bills in incidents, not invoices. Here is last week's, in full.

On August 23 I rotated WAHA_API_KEY. The rotation updated /opt/waha/.env and the nginx snippet that injects the header. What it did not update was a second vhost, an internal tailnet-only copy created weeks earlier by duplicating the public one, where the key had been written as a literal instead of the $waha_api_key variable.

Result: every GET /api/files/… over the internal path started returning 401. Bots that download an image, OCR it, and act on the contents stopped being able to fetch anything. Of the three code nodes involved, one threw and killed its run outright; the other two caught the error and skipped silently, so those pipelines produced output — just output with no images in it. The public vhost was untouched, so every check that went through the domain came back green.

The failure surfaced the next morning, because nobody had sent a photo overnight. A one-line fix, roughly twelve hours of quiet wrongness, and a monitoring path that was structurally incapable of seeing it.

Twilio would have absorbed that entire class of problem. That is what the $604 buys. Not messages. The absence of a category of 2am.

And there is a risk that no amount of care removes. WAHA drives a real WhatsApp Web session. Its own homepage says it plainly: "WhatsApp does not allow bots or unofficial clients on their platform, so this shouldn't be considered totally safe." A banned number is not a support ticket, it is a dead asset. That is a business risk, not a technical one, and it does not belong in a cost table. It belongs in whatever conversation you have before you build.

The line I actually draw

After enough of these deployments, the rule I use with clients is not about volume at all:

The number is the business. A clinic's only phone line, a store's published contact. Official API, pay Twilio. The ban risk is unpriceable.
Internal, operational, or a secondary channel. Courier groups, staff notifications, an intake line that can be replaced in an afternoon. WAHA. The savings are real and the downside is survivable.
Marketing blasts at any volume. Official API, and check the marketing rate for your country before you promise anyone a budget. At $0.0353 a message in Israel and $0.1365 in Germany, the same campaign has wildly different economics depending on who receives it.

Volume is the wrong axis. Replaceability is the right one. This is roughly the split behind the pricing I quote Israeli businesses for WhatsApp automation: the channel decision comes first, and the bill follows from it.

One more thing worth knowing: Meta has further pricing changes landing October 1, 2026, covering service and utility messages. Whatever you calculate today has a shelf life of about five weeks.

Run the query

The number that decides this for you is not 89,479 and it is not $604. It is your own outside_window count, and it is the one input a vendor pricing page cannot give you.

So: run it against your own inboxes and post the three numbers. Outbound, inside window, outside window. I am specifically curious whether the 67% outside-window share holds anywhere else, or whether it is an artifact of how my bots are scheduled. If that ratio is stable across use cases, it is a much better rule of thumb than anything on a vendor pricing page.

I Audited 50,669 n8n Runs. 85% of Failures Alerted Nobody.

אחיה כהן — Mon, 24 Aug 2026 05:40:54 +0000

One of my own workflows — the gate that decides whether a new WhatsApp contact gets the bot or gets me — failed 24 times between Thursday and Sunday. Seventy-one hours. I found out on Monday morning, by opening the executions list for an unrelated reason.

That is embarrassing for someone who sells automation. So instead of fixing the one workflow and moving on, I went and measured the actual state of error handling across all 225 workflows on my main production instance. Here is every execution my instance still retains — pruning is on, so this is a six-day window, 18–24 August:

Metric	Value
Executions (6 days)	50,669
Failed executions (`error` + `crashed`)	106
Failure rate	0.21%
Active workflows	60
Active workflows with an error workflow attached	26 (43%)
Failures that occurred in a workflow with no error handling	90 of 106 (85%)

n8n 2.35.5, queue mode, Postgres. A 0.21% failure rate is fine. The 85% is not.

Why silence is the default

Error handling in n8n is opt-in, per workflow. You build one workflow whose trigger is the Error Trigger node, then in every other workflow you open Settings → Error Workflow and point it at that one. There is no switch that applies it to everything you have already built, or to everything you build next.

That is a per-workflow checkbox on a list that grows every week. Nobody forgets on purpose. You just build workflow #61 at 11pm, it works, you activate it, and you move on. The failure mode of an opt-in safety net is that coverage decays quietly while the number in your head stays at "yeah, we have error handling."

Mine had decayed to 43%.

What the gap actually cost

I pulled the failures grouped by workflow, with a column for whether that workflow had an error handler attached:

Workflow	Guarded	Failures	Window
WhatsApp group lead detection (AI)	no	34	36.5 h
Personal-line bot gate	no	24	70.9 h
Client error-handling workflow	no	13	1.2 h
Campaign-failure watchdog	no	11	8.7 h
Website lead intake form	no	4	53.0 h
SMS dispatcher (outbox)	yes	8	1.2 h
Appointment sync + no-show recovery	yes	4	1.1 h
6 others	mixed	1–2 each	~0 h

Read the last column, not the failure counts. Every window longer than nine hours belongs to an unguarded workflow. The longest is 70.9 hours — a WhatsApp bot gate that failed 24 times over three days while I was working on other things. Nothing was on fire. No customer wrote in. The bot simply did not answer new contacts, and the only reason I found it is that I went looking.

The guarded workflows failed too — 16 times between them. Their longest window was 1.2 hours, because something told me.

That is the whole value proposition of the Error Trigger, and it is not "fewer failures." It is failures that end in hours instead of days.

The part that actually scared me

Three of my workflows contain an Error Trigger node. Those are the handlers — the things that send the Telegram alert. Here is how many active workflows depend on each one:

Handler	Workflows it guards	Guarded itself?
Client error handler	15	no
Generic error alert	6	no
Product owner alert	5	no

Every handler was unguarded. And the one covering 15 workflows is the same row from the table above: it failed 13 times inside that same six-day window.

For those 13 failures, 15 workflows had no alerting at all and no way to find out. The error handler cannot report its own errors, because the thing that reports errors is the error handler. Quis custodiet ipsos custodes, in YAML.

You cannot fix this by pointing handler A at handler B, either. That just moves the single point of silence one hop and adds a cycle you will forget about. The catcher has to be watched from outside n8n.

The pattern, in four parts

1. One handler, not one per project. Fan-in beats fan-out. A single Error Trigger workflow that formats {{ $json.workflow.name }}, {{ $json.execution.id }} and {{ $json.execution.error.message }} into one alert is easier to keep correct than six near-copies.

2. Audit coverage with a query, not with your memory. This is the whole audit — run it against your n8n Postgres:

SELECT w.name,
       (w.settings->>'errorWorkflow' IS NOT NULL
        AND w.settings->>'errorWorkflow' <> '') AS guarded
FROM workflow_entity w
WHERE w.active = true
ORDER BY guarded, w.name;

Everything with guarded = f is a workflow that can fail into the void. If you are on n8n Cloud, the same check runs off GET /api/v1/workflows?active=true and reading settings.errorWorkflow per item.

That query is now the first line of my weekly runbook, next to the backup check. A new f row gets treated the way a failing test does.

3. Make the handler loud about workflows, not just about errors. The first thing I changed in mine was putting the workflow name in the subject line instead of in the body. That sounds trivial. It is the difference between "an automation failed" (which you snooze) and "the lead intake failed" (which you do not).

4. Guard the guard from outside. The Error Trigger has a blind spot that no amount of coverage fixes: it fires when an execution fails. It does not fire when an execution never starts — a deactivated trigger, a dead cron, a webhook whose URL changed, a worker that is not consuming the queue. A workflow that stopped running entirely produces zero failed executions and therefore zero alerts. It looks exactly like a quiet week.

So the handler pings a dead man's switch on every run, and a plain cron on a different machine screams if the ping stops:

# in the error handler workflow, and in one heartbeat workflow per instance
curl -fsS -m 10 "https://your-monitor/ping/$SLUG" || true

# elsewhere — not on the n8n box
LAST=$(stat -f %m /var/lib/heartbeat/n8n1 2>/dev/null || echo 0)
if [ $(( $(date +%s) - LAST )) -gt 3600 ]; then
  notify "n8n1 heartbeat is $(( ($(date +%s) - LAST) / 60 ))m stale"
fi

The || true matters: a monitoring call that can fail the run it is monitoring is worse than no monitoring.

That is it. Four parts, none clever. The reason it is worth writing down is that I had parts 1 and 3 for as long as I have run this instance and still ate a three-day outage, because I never had part 2 and never imagined I needed part 4.

Where I landed

I am not going to pretend this is finished. As I write this, coverage is still 43% — what changed is that I now have the number, the query that produces it, and a list of 34 workflows in the order I am going to fix them, longest silent window first. The handler-watching-the-handler problem is the part I am fixing today, because it is one workflow and it covers fifteen.

The honest reason I am writing this before finishing it: the audit took twenty minutes and the three-day outage took three days. Those twenty minutes are the cheapest thing in the entire stack I maintain — cheaper than the WhatsApp bots I run in production for clients, and a rounding error against the numbers in this breakdown of what business automation actually costs. Run the query before you finish reading this post and you will probably learn something uncomfortable too.

One genuine question, because I do not think I have solved this part: how do you monitor the trigger that never fires? A heartbeat proves the instance is alive and proves that workflow ran. It does not prove that a webhook from a third party still arrives, or that a specific cron survived the last upgrade. Per-workflow "expected minimum run rate" alerting is the obvious answer and it also sounds like a maintenance burden that decays exactly like error-workflow coverage did.

If you run n8n in production: do you alert on absence of runs, and if so — how do you keep the thresholds from rotting?

Self-Hosted Chatwoot: 5 Failures the Docs Don't Warn You About

אחיה כהן — Thu, 20 Aug 2026 06:10:34 +0000

I run self-hosted Chatwoot as the WhatsApp inbox for a dozen or so small Israeli businesses. Two servers, a few thousand conversations a week, a drip-sequence engine bolted on the side.

Chatwoot is good software. The self-hosting docs will get you to a running container. What they will not tell you is which failures actually happen at month six, when you have real customers and real volume. These five all bit me in production, and none of them looked like what they were.

1. Your disk fills from somewhere Postgres never sees

I got a disk alert at 86 percent and immediately went looking at the database. That was the wrong place.

DB (postgres):            680 MB
chatwoot_storage_data:     17 GB

Attachments live in ActiveStorage, on a Docker volume, not in Postgres. Every image, voice note, and PDF a customer sends is a file on disk, and none of it shows up when you check database size. If your monitoring watches the DB, it will report everything is fine right up until the container cannot write.

The growth curve is a function of how many accounts you host, not how busy any one of them is. Mine sat at roughly 0.05 GB a month until I onboarded seven new businesses over two months, and then it hit 16 GB a month.

Check the right volume:

docker system df -v | grep chatwoot_storage_data

2. Forty-four percent of my outbound storage was duplicate files

This is the part that surprised me. When I actually measured what was on that volume, almost half the outbound media was byte-identical copies of the same file.

One 14.5 MB video was stored 48 separate times. One image was stored 325 times.

Chatwoot creates a new blob and a new file on disk on every send, even when the bytes are identical. That is correct behavior for a chat app where every message owns its attachment. It becomes expensive the moment you have anything that fans one file out to many conversations. In my case it was not campaigns at all, it was the drip engine sending the same media to 48 separate conversations as ordinary outbound messages, each one a fresh readFile plus fd.append('attachments[]', ...).

Deduplicating is safe, and I verified this in the Rails source before touching anything. ActiveStorage::Blob#purge is guarded by a foreign key on active_storage_attachments.blob_id, so deleting one message will not take out a file that other messages still point at:

# ActiveStorage 7.1.5.2
def purge
  destroy
  delete if previously_persisted?
rescue ActiveRecord::InvalidForeignKey
end

One pass over existing blobs, matching on checksum and byte_size within the same account, reclaimed 3.20 GB across 7,160 blobs and took the disk from 86 percent to 71 percent. You want an index on active_storage_blobs (checksum, byte_size) before you try this, and a size floor so you are not doing lookups for every 4 KB thumbnail.

3. `POST /messages` returns 200 before anything has been sent

This one cost me an outage, and it is entirely my own fault for reading the status code as confirmation.

POST /api/v1/conversations/{id}/messages returns 200 immediately. All it has done is insert a row into public.messages with status = 0 and no source_id. The actual delivery to Meta happens later, in a Sidekiq job on the high queue. The source_id, which is the WhatsApp message ID, only gets written once Meta acknowledges.

So a tight send loop looks completely healthy from the client side while it quietly fills a queue that everything else also depends on. Four resend runs kicked off within ninety seconds, roughly thirteen messages a second, pushed 5,108 messages through that endpoint. The high queue grew to 3,786 jobs with eleven minutes of latency, and every inbound message from an actual paying customer sat behind them.

Nothing errored. The dashboard just showed a lot of pending clocks, which in Chatwoot means "waiting for delivery receipt" rather than "scheduled" — a distinction I have now confused twice.

Measure queue depth, not HTTP status:

docker exec chatwoot-sidekiq-1 bundle exec ruby -e '
  require "sidekiq/api"
  q = Sidekiq::Queue.new("high")
  puts "#{q.size} jobs (latency #{q.latency.round}s)"'

The fix that actually held was a backpressure check in my own sender: every 40 sends, count my messages still sitting at source_id IS NULL AND status = 0. Above 250, pause until it drops under 80, with a 120-second ceiling. That needs no access to Chatwoot's Redis and it measures the right thing, which is the pressure I created rather than global queue depth.

4. Deleting an inbox is a Rails-level cascade, and it is silent

During a WhatsApp Business Account migration I deleted an inbox. Here is what went with it:

894    conversations
6,160  messages          (471 of them lead replies)
1,776  contact_inboxes

Contacts survive, because they live at the account level. Everything else is gone. This is a dependent: :destroy cascade in the Rails models, not a database constraint, so nothing in Postgres warns you and there is no confirmation dialog proportionate to what is about to happen.

The part I did not anticipate: the real blocker afterwards was not the lost conversation history. It was contact_inboxes. Without those rows, nothing can open a conversation at all — my engine just started returning no WhatsApp contact_inbox for every contact. Conversation history is nice to have. contact_inboxes is load-bearing.

The consolation is that anything you keep in your own schema survives, since it has no foreign keys into Chatwoot's tables. My sequence enrollments came through untouched, so nobody's position in a drip sequence was lost.

5. Restoring from backup has three traps that all look like data loss

I had a backup. Restoring it still took most of a day, because of three things that each make it look like the restore failed when it has not.

A trigger overwrites your display_id. conversations_before_insert_row_tr calls nextval on a per-account sequence, BEFORE INSERT, so every conversation you inject with an explicit display_id silently gets a brand new one. Every foreign reference you were trying to preserve detaches. The way through is to insert, then UPDATE ... SET display_id from a staging table (UPDATE does not fire that trigger), then setval the sequence to the real maximum.

There are unique indexes that are not constraints. They do not appear in pg_constraint, so if you go looking for what you might collide with, you will not find them:

contact_inboxes (inbox_id, source_id)
conversations   (account_id, display_id)
conversations   (uuid)

ON CONFLICT (id) DO NOTHING sails straight into all three. Use ON CONFLICT DO NOTHING with no target.

One skipped row 500s the entire UI. A conversations.contact_inbox_id pointing at a contact_inbox you did not inject produces undefined method 'source_id' for nil, which surfaces as an infinite spinner across the whole dashboard rather than a broken single conversation. Remap every orphaned contact_inbox_id before you declare the restore done.

Worth knowing: contact_inboxes.source_id is just the phone number in E.164 without the leading + (so +972 50 000 0000 becomes 972500000000). It does not depend on which WABA you are on, which means you can rebuild these rows from contacts.phone_number even with no backup at all.

Bonus: "timeout exceeded when trying to connect" is not your database

The dashboard stopped loading with timeout exceeded when trying to connect. The same error appeared on my background ticks, which meant sending had stopped for every client. It looked exactly like Postgres falling over. Postgres was fine.

The cause was one OR inside one NOT EXISTS:

NOT EXISTS (
  SELECT 1 FROM ledger s
  WHERE s.campaign_id = m.campaign_id
    AND (s.message_id = m.id
         OR (m.source_id IS NOT NULL AND s.source_id = m.source_id))
)

With the OR in there, Postgres can only hash on campaign_id. The rest becomes a Join Filter evaluated across every pair in the bucket. On 23K messages against 23K ledger rows that is Rows Removed by Join Filter: 22,371,572 and 125 seconds, to return zero rows. The cost is quadratic in campaign size, so it only detonates for your largest customers.

The connection pool did the rest. It is max: 5 and shared across every client's API requests plus the background ticks. Four of these queries at once starved it, and everything else died on a 10-second connection timeout. A slow query in one tenant took down every tenant.

The fix is De Morgan, ¬(A∨B) ≡ ¬A ∧ ¬B — split into two NOT EXISTS, each with a complete equality condition to hash on:

NOT EXISTS (SELECT 1 FROM ledger s
            WHERE s.campaign_id = m.campaign_id AND s.message_id = m.id)
AND NOT EXISTS (SELECT 1 FROM ledger s
                WHERE s.campaign_id = m.campaign_id
                  AND m.source_id IS NOT NULL
                  AND s.source_id = m.source_id)

125,366 ms to 381 ms. I verified equivalence with a bidirectional EXCEPT against production across every account before shipping it, including the one account that actually had legacy rows the filter was there to catch.

The pattern

Four of these five presented as something other than what they were. A disk alert that was not the database. A 200 that had not sent. A timeout that was not the database either. A restore that looked like it had lost data it had not.

Self-hosting Chatwoot is genuinely worth it at this scale, and I would make the same call again. But budget your operational attention for the layer between the container and your own code, because that is where all of this lives. Most of what I have learned here came out of running the WhatsApp automation I build for Israeli businesses on top of it, which is to say it came out of breaking things in front of paying customers.

If you are still weighing self-hosting against Cloud, the operational attention above is the real line item — I put it next to the tier prices and the data-retention caps in what Chatwoot actually costs to run.

One I have not solved: has anyone found a clean way to get Chatwoot to tell you a message actually reached Meta, without polling source_id yourself? I would rather subscribe to something than poll a column, and I have not found the hook.

The Bing Side Door: 87% of ChatGPT's Citations Match Bing's Top 10

אחיה כהן — Tue, 18 Aug 2026 09:11:12 +0000

Two days ago I published a census of every AI crawler that hit my site over 34 days. The headline was that ChatGPT now fetches my pages for live users more often than Googlebot crawls them. This post is about the number in that dataset I couldn't stop thinking about — the one that isn't ChatGPT at all.

It's bingbot.

The ratio nobody is optimizing for

Fresh numbers from this morning's run (36 days of Caddy access logs, HTTP 200 responses only, for a small business site):

Bot	Hits	Per day
bingbot	4,488	123.3
ChatGPT-User (live retrieval)	1,394	38.3
Googlebot	1,073	29.5
GPTBot (training)	505	13.9
OAI-SearchBot	249	6.8

Bing's crawler reads my site 4.2 times more than Google's. For years I would have read that as wasted crawl budget from the search engine nobody uses — Bing's share of human search traffic barely registers in my analytics.

Then you put the second fact next to it: Seer Interactive ran 500+ citations through SearchGPT and found that 87% of them matched Bing's top-10 organic results for the same query. Only 56% matched Google's top 10.

ChatGPT's web search doesn't have its own index. When it goes looking for sources to cite, it queries Bing. Those 4,488 bingbot hits aren't for Bing's ten human users — they're the supply chain for every ChatGPT answer that might mention you. Bing stopped being a search engine and became an API for AI answers, and its crawler's appetite finally makes sense.

What the side door looks like from inside the logs

The chain is visible end to end in my own data. bingbot's most-crawled pages this window are my blog's long-form posts and guides. And the pages ChatGPT-User actually fetches when a human asks something? The same category: my Chatwoot-vs-Intercom comparison got 101 live ChatGPT retrievals in 36 days, and a Hebrew post about WhatsApp spam law got 226 retrievals from Claude-User alone. Comparison pages and definitive guides go in through Bing, come out as citations.

I run a WhatsApp automation studio in Israel, and this pipeline stopped being theoretical for me in mid-July, when a Google core update cut my Israeli search clicks nearly in half. The traffic that kept arriving through the collapse was exactly this AI-assistant channel — which, per Seer's data, is largely downstream of an index whose webmaster console I had never once opened.

One more uncomfortable detail from the earlier post: when I compared server logs to Google Analytics, GA4 saw about 11% of the visitors the logs saw. AI-referred and privacy-shielded traffic is mostly invisible to your dashboard. If you only look at GA4, this entire channel doesn't exist.

The practical checklist

If 87% of ChatGPT citations come from Bing's top 10, then "rank in Bing" is now a growth channel with almost no competition, because everyone else is still fighting over Google. What I've changed:

1. Stop treating bingbot as a nuisance. Plenty of robots.txt files and WAF rules throttle or block bingbot to "save crawl budget." That's now equivalent to blocking your ChatGPT visibility at the front door. Check yours before anything else.

2. Register in Bing Webmaster Tools. It takes minutes (you can import your verified site straight from Google Search Console) and it's the only place you'll see how Bing actually indexes you.

3. Ship IndexNow. Bing supports instant push indexing — a tiny API ping on every publish instead of waiting for a crawl. For a channel where citations follow the index, freshness is free ranking.

4. Feed it the content type the pipeline rewards. In my logs, what gets retrieved for live users is comparisons ("X vs Y"), pricing breakdowns, and complete guides — pages that answer a full question. Listicle fragments don't get cited.

5. Measure in your logs, not your analytics. Thirty seconds on any server:

# bingbot vs Googlebot, raw hit counts
grep -c 'bingbot' access.log
grep -c 'Googlebot' access.log

# which pages bingbot actually wants
grep 'bingbot' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head

(Adjust the $7 field for your log format; mine is Caddy JSON piped through a small parser.)

The uncomfortable conclusion

I spent years doing SEO for an audience of humans on Google. The team behind 50+ WhatsApp bot deployments that pays my bills gets found today by a different reader: a retrieval bot working for an AI that a buyer asked in natural language. That reader enters through Bing, and almost nobody is holding the door.

Update (19 Aug): I checked, and part of this post is wrong

A commenter, @bulti_global, pushed back on turning the 87% overlap into "citations follow the Bing index," and asked a sharp question: are there pages that rank in Bing's top 10 but never trigger a retrieval? He suggested splitting the funnel page by page instead of trusting the aggregate bot ratio.

Answering it forced me to open Bing Webmaster Tools — where I found an account I'd forgotten I had, holding 91 days of data. Two corrections follow from that.

First, I overstated my own ignorance. I wrote that this ran through "an index I had never once logged into." I had registered; I had just never looked.

Second, and worse for my thesis: at page level, Bing rank does not predict AI citation. Same site, same 91-day window (19 May – 17 Aug). Bing organic delivered 650 impressions and 4 clicks. AI citations over that window: 81.

Page	Bing impressions	Avg position	AI citations
whatsapp-bot-platforms-comparison-2026	169	4.48	62
whatsapp-automation-guide	90	4.00	0
whatsapp-bot-for-business	60	4.20	0
free-whatsapp-bot-guide	46	5.87	7
pricing	31	4.32	0
whatsapp-bot-clinics	29	3.28	0
whatsapp-bot-for-gyms	16	2.81	0

My best-ranking page in Bing — average position 2.81 — has never been cited. Neither have 3.28, 4.00 or 4.32. The page that takes 62 of the 81 citations ranks worse than four pages that were never cited once, and one page picked up 7 citations without appearing in my top-14 by impressions at all.

What separates the winner looks like format, not position: it is the only head-to-head "X vs Y vs Z" comparison table on the site. Item 4 of the checklist below survives. The implicit "rank in Bing and citations follow" does not — not at page level, not on this sample.

One caveat on the numbers, because it matters: Bing labels that report "Microsoft Copilots and Partners." It measures the Microsoft AI surface, not ChatGPT specifically. It is adjacent to the claim, not identical to it.

Registering still turned out to be the single highest-value thing in this post — just not for the reason I gave. It is where you find out that your best-ranking page is invisible to the answer engines.

Run the grep above on your own logs and tell me your bingbot-to-Googlebot ratio in the comments — I'm collecting data points for a follow-up post. Mine is 4.2:1. If yours is under 1:1, I'd especially like to know what kind of site it is.

I found code in my repo I'd never seen. All 82 tests passed. I quarantined it for three days anyway.

אחיה כהן — Mon, 17 Aug 2026 03:22:43 +0000

During a routine morning triage of my open-source project, git status showed a modified file I had no memory of touching: extension/background.js, last modified 24 hours earlier, sitting next to a fresh background.js.bak someone had thoughtfully left behind.

Nobody broke in. I run several AI coding sessions in parallel against the same machine, and one of them — working on a completely different task, automating a GoHighLevel workflow — had hit a limitation in my browser automation tool, fixed the tool itself, verified the fix, and then moved on with its actual job. It never committed. It never told anyone. It just left better code in my working tree and walked away.

The diff was good. That was the problem.

The change itself was a real feature. My query_all tool (it queries DOM elements across a page) stopped at the main frame: if the elements you wanted lived inside a cross-origin iframe, you got back a clean, confident, empty array. The uncommitted diff added an execAcrossFrames() helper that runs the query in every frame and merges the results, plus x/y/frame fields on each returned element.

I verified it the way you'd verify anything: syntax check passed, and the full test suite — all 82 tests — ran green with the change in place.

So: useful feature, my own repository, every signal green. Everything about the situation said commit it.

I didn't. I wrote it up in my project log, left the file untouched, and set an explicit deadline: if it's still sitting there uncommitted in three days, evaluate it properly — upstream it or revert it and file an issue. Not "leave it and see," which is how working trees rot. A quarantine with no release date is just a junk drawer.

Why quarantine green code?

Two reasons, and neither is paranoia.

First: authorship isn't verification. The session that wrote this code had context I didn't have. Maybe it was mid-iteration and the diff was half of a plan. Maybe the .bak file meant it intended to roll back. Committing someone's work-in-progress freezes it at a moment they didn't choose. The fact that the "someone" was technically me, in another window, changes nothing — I had none of that session's context. A diff you don't remember writing is a stranger's diff. The stranger being you is a detail.

Second: green tests measure what you thought to test. My suite passed because nothing in it asserted anything about cross-origin frames — the tests were blind to the change, not endorsing it. "All tests pass" and "no test looked" produce the same green checkmark.

Day 3: review it like a stranger's pull request

The deadline arrived, the diff hadn't moved, so I did what I'd do with an external PR from an unknown contributor: re-ran everything (still 82/82), then read the semantics line by line instead of trusting the vibe.

And there it was — the one bug no test could have caught, exactly where I'd felt vague unease on day 0. The code's comments claimed the new x/y coordinates were page-level. They aren't. Each element's coordinates are relative to its own frame's viewport. For main-frame elements that's the same thing; for an element inside a cross-origin iframe, it's iframe-relative — so a caller who took those numbers and clicked at that position on the page would click the wrong spot. Silently. Only on the exact pages this feature was built for.

The code was right; its claim about itself was wrong. That's a documentation bug today and a caller's logic bug tomorrow. Fix: correct the comment to state the real coordinate space, and document that callers must offset using the frame field the diff had already (presciently) added.

Then it graduated: comment fixed, changelog written, committed as a proper feature with attribution to the context it came from, the .bak file byte-compared against git history and deleted, and released in v2.16.0. The quarantine didn't slow the feature down. It shipped three days later than "immediately" — and shipped correct.

What I keep from this

A diff you don't remember is untrusted input, even in your own tree, even green. Multi-agent workflows make this a weekly event, not a freak accident.
Never commit a parallel session's WIP on discovery. You're freezing someone else's half-thought.
Quarantine needs a deadline. "I'll look at it later" is how repos accumulate mystery files. Name the day and the two exits: upstream or revert-and-file.
On review day, read the claims, not just the code. The only real bug was in a comment — a semantic promise no test asserted and no linter checks.

The uncomfortable part: five years ago, "unexplained code appeared in my working tree overnight" meant your laptop was compromised. Now it means Tuesday. The tooling caught up with writing code faster than our habits caught up with receiving it.

How do you handle this? If you run parallel AI sessions — or just share a repo with your past self — what's your protocol when you find a diff you don't remember writing? Commit-if-green, revert-on-sight, or something in between?

I Logged Every AI Crawler for 34 Days. ChatGPT Outreads Googlebot

אחיה כהן — Sun, 16 Aug 2026 09:40:07 +0000

In mid-July, my Google clicks in my home market (Israel) dropped by almost half. Buyer-intent queries that used to bring steady leads just evaporated from Search Console. While I was staring at GSC dashboards trying to figure out what broke, I finally did the thing I should have done months earlier: I stopped looking at dashboards and started reading raw server logs.

What I found there was a parallel universe. Google Search was sending me less than ever — but AI systems were reading my site constantly. Not "someday this will matter" constantly. Right-now constantly: an AI assistant was fetching a page of mine roughly every 26 minutes, around the clock, because a real human had just asked it a question.

So I built a small log analyzer and let it run. Here's what 34 days of complete Caddy logs from a small business site (about 70 real human visitors a day) actually look like.

The numbers

All counts are HTTP 200 responses only (more on why below), over 34 days:

Bot	Requests	Per day	What it is
bingbot	5,444	158.2	Bing's index — which feeds ChatGPT
ChatGPT-User	1,388	40.3	Live fetch while a human asks ChatGPT
Googlebot	1,233	35.8	Classic Google crawl
GPTBot	547	15.9	OpenAI training crawler
Claude-User	519	15.1	Live fetch while a human asks Claude
OAI-SearchBot	281	8.2	ChatGPT search indexing
Applebot	268	7.8	Apple (Siri / Apple Intelligence)
ClaudeBot	214	6.2	Anthropic training crawler
Amazonbot	136	4.0	Amazon (Alexa & co.)
PerplexityBot	103	3.0	Perplexity indexing

Three things in that table genuinely surprised me.

ChatGPT-User outreads Googlebot. 40.3 fetches a day versus 35.8. This isn't a crawler building an index for later — ChatGPT-User is the user-agent OpenAI sends when a human is mid-conversation and ChatGPT decides to pull a live page to answer them. On my site, that now happens more often than Googlebot visits. For a tiny business site in a niche market, I did not expect that.

Bing crawls 4.4x harder than Google. 158 requests a day versus 36. Nobody optimizes for Bing. Bing's referral traffic is a rounding error. But Seer Interactive found that 87% of SearchGPT's citations match Bing's top organic results (versus 56% for Google) — Bing's index is a structural input to what ChatGPT can retrieve and cite. Suddenly that 4.4:1 ratio reads very differently: the "irrelevant" search engine is the supply chain for the assistant that's actually fetching my pages 40 times a day.

Live retrieval is the biggest AI bucket that matters. Grouping by meaning rather than by bot name:

🔴 Live user retrieval (a human asked right now)   1,907  (~56/day)
🔎 AI-search indexing (OAI-SearchBot, Perplexity,
   bingbot, Applebot)                               6,096  (~177/day)
📚 Training crawls (GPTBot, ClaudeBot, Amazonbot)     897  (~26/day)
🌐 Classic Google                                   1,233  (~36/day)

1,907 live retrievals in 34 days. Each one of those is a moment where a real person asked an AI assistant something, and the assistant chose my site as a source. That's the leading indicator for citations — and it's the number I now track weekly.

What they actually read (this is the useful part)

Per-bot top pages turned out to be more interesting than the totals:

ChatGPT-User loves the homepage (115 fetches), a WhatsApp automation guide (108), and — third place — our WhatsApp bot pricing guide. People are asking ChatGPT what a WhatsApp bot costs, and ChatGPT is pulling my pricing page to answer them. That's a sales conversation I'm part of without knowing it.
Claude-User is obsessed with exactly one post: a spam-detection write-up got 233 of its 519 total fetches — 45% of everything Claude reads on my site is one article. One strong, specific technical post apparently becomes a go-to source for one assistant.
GPTBot (the training crawler) spends its time on /signin and /forgot-password. Twelve visits to my login page. Enjoy the training data, I guess.
Perplexity-User: zero. PerplexityBot indexes me (103 requests), but not a single live user-triggered retrieval in 34 days. Every assistant has its own taste in sources, and Perplexity's taste apparently doesn't include me. Yet.

Methodology, and the trap that almost ruined the data

The analyzer is ~60 lines of Python that runs over Caddy's JSON access logs. The core is just user-agent patterns with one ordering rule — the *-User patterns must be matched before the generic bot patterns, because the loop stops at first match:

BOTS = {
    "ChatGPT-User":    re.compile(r"ChatGPT-User", re.I),   # live retrieval
    "Claude-User":     re.compile(r"Claude-User", re.I),    # live retrieval
    "Perplexity-User": re.compile(r"Perplexity-User", re.I),
    "OAI-SearchBot":   re.compile(r"OAI-SearchBot", re.I),
    "GPTBot":          re.compile(r"GPTBot", re.I),          # training
    "PerplexityBot":   re.compile(r"Perplexity", re.I),
    # ... bingbot, Googlebot, ClaudeBot, Applebot, Amazonbot
}

The trap: count only HTTP 200s. My first version counted every request, and the numbers were inflated with garbage — security scanners spoof OpenAI user-agents (a scanner claiming to be GPTBot while probing for /wp-admin on a site that doesn't run WordPress) and land on 404s. Filtering to 200-only removed the impostors and left actual content fetches. If you run this on your own logs and skip that filter, your "AI traffic" will be part vulnerability scan.

Why your analytics can't see any of this

None of these 8,900 AI-related requests appear in GA4. No JavaScript execution, no consent banner click, no pageview event. And it's worse than that: on my site, GA4 undercounts humans by roughly 9x too — only ~11% of visitors accept the cookie banner (I measured this two independent ways over 11 days: ~780 real browser visitors in the logs and widget-loads, 89 in GA4).

So the fastest-growing reader of my site — AI assistants answering buyer questions in real time — is completely invisible unless you read server logs. If your hosting gives you raw access logs, that's where this story lives. If it doesn't, you can't see it at all, and you might be making content decisions while blind to a quarter of your real readership.

What I'm doing differently now

Watching *-User agents weekly, not rankings daily. Live retrievals are the citation pipeline.
Taking Bing seriously for the first time in my life. Bing Webmaster Tools went from "never opened it" to the top of this week's list — if 87% of ChatGPT citations flow through Bing's index, it's the cheapest GEO lever there is.
Writing more single-topic, specific pages. Claude's 233-fetch obsession with one spam-detection post convinced me: assistants latch onto pages that answer one question thoroughly, not pillar pages that answer twelve questions shallowly.

Now I want your denominator. Pull one month of access logs and count bingbot vs Googlebot hits — my ratio is 4.4:1. If you run a small site, especially a non-English one, I'd genuinely like to know whether that ratio holds outside my niche: drop your two numbers (and your site's language) in the comments.

I said my tool supported multiple instances. A stranger running 15 of them filed three bugs. All three were right.

אחיה כהן — Thu, 13 Aug 2026 08:20:51 +0000

I maintain safari-mcp, an MCP server that lets AI agents drive a real Safari session on macOS. Over a few releases this summer I shipped per-session tab isolation — the feature that lets several agents share one browser without stealing each other's tabs. I wrote about it here. Twice. I was reasonably proud of it.

Then a stranger showed up with a deployment I had never imagined — three macOS profiles, up to fifteen concurrent server instances — and filed three bug reports in the space of seven seconds.

All three were right.

The reports

What made them remarkable wasn't just the hit rate. It was the form. Each issue had a description of the deployment, a repro, and observed data from their own machines. No speculation, no "it feels flaky." They had read the code, formed a hypothesis about where it breaks at their scale, and then watched it break exactly there.

Bug 1: the memory guard that guarded nothing. The server has a WebKit memory monitor that sweeps idle tabs before Safari's content process balloons. It also has a gate: only the instance acting as "extension host" — in practice, whichever instance won the race for port 9224 — was allowed to sweep, and only its own tabs. In a fifteen-instance deployment, that means fourteen instances accumulate tabs forever and one instance politely cleans up after itself. The guard was effectively inert precisely in the topology where memory pressure is worst.

The embarrassing part: the gate was redundant. There was already a lock file guaranteeing one sweeper per cycle. Someone (me) had added a second safety mechanism on top of an existing one, and the second one broke the feature. The fix was a deletion.

Bug 2: the ownership file that ate its own entries. Instances record which tabs they own in a shared JSON file. The save path was read-at-startup, modify-in-memory, overwrite-on-save. Classic. With one instance, harmless. With fifteen, every save was a snapshot overwrite that silently dropped whatever the other fourteen had written in the meantime.

Note what this bug doesn't do: it doesn't crash, it doesn't touch the wrong tab, it doesn't log anything. Losing an ownership entry is fail-safe — a tab just stops being tracked. Which is exactly why it survived. Bugs that fail loudly get fixed in week one. Bugs that fail politely wait for someone to run fifteen instances and notice entries vanishing.

The fix was merge-on-write: union the disk state with local state, newest timestamp wins, with explicit removal deltas so deletions don't get resurrected by a stale peer. I considered a file lock and deliberately skipped it — the residual race window is sub-millisecond and fail-safe, and a lock adds a failure mode of its own. That tradeoff is written down in the code, next to the merge.

Bug 3: the poll that never backed off. When a profile's Safari window is closed, the server polls for it to reappear — a fixed 3-second interval spawning an osascript subprocess each time. Forever. Their logs showed the consequence: roughly 1,200 log lines and 1,200 subprocess spawns per hour, per idle instance. Multiply by fifteen.

The fix is the boring one you already know: self-scheduling backoff, 3s doubling to a 60s cap, reset on success. While fixing it I found a bonus bug the report hadn't mentioned — a trace log being written to __dirname, which for an npm package means inside node_modules. That file had been quietly growing inside an installed package directory the whole time.

What I actually learned

Your test topology is a claim about the world, and nobody peer-reviews it. I had tested multiple instances — on one profile, launched from one place, in small numbers. The reporter ran three profiles times five. Every one of the three bugs lived precisely in the gap between my topology and theirs. Not one was visible in mine. The tests were green the entire time; the suite passed 78/78 the morning the reports came in. Green tests verify the world you thought to simulate.

Fail-safe failures are the long-lived ones. All three bugs share a property: the system kept working. Memory got worse slowly. Entries vanished silently. Subprocesses spawned invisibly. If any of them had thrown an exception, they'd have died in code review. Design your failures to be loud, or accept that the quiet ones will be found by a stranger, years later, at a scale you didn't test.

Sometimes the fix is a deletion. The memory-guard bug existed because two safety mechanisms overlapped and the newer one strangled the feature. I didn't need to write cleverer coordination code — I needed to remove the redundant half. The diff that fixed the scariest-sounding of the three bugs was net negative lines.

A good bug report at foreign scale is free QA you could not have bought. I could not have afforded to stand up a fifteen-instance, three-profile deployment as a test rig for a feature I wasn't sure anyone used that hard. Someone built it because they needed it, then handed me the findings with repros. The only correct responses are fast verification, honest credit, and shipping the fix. All three fixes went out in one release the next day, each issue closed with a reply documenting not just what changed but which alternatives I rejected and why — including the lock I chose not to add.

That last part matters more than it looks. The reporter now knows the merge-on-write has a known, bounded, fail-safe race. The next person who hits something weird there won't start from zero — the reasoning is in the thread, not in my head.

The uncomfortable question

The feature worked. The story I told about it was, at my scale, true. It took someone else's production topology to reveal that "supports multiple instances" was really "supports the multiple-instance shapes I happened to try."

What's the biggest gap between your test topology and a user's real deployment that you've been bitten by — and did the bug fail loudly or politely?

An invisible character broke a security patch. Then it broke my review. Then it broke my review of the fix.

אחיה כהן — Sun, 09 Aug 2026 07:15:27 +0000

My open-source project, safari-mcp, got a security-hardening pull request from an automated scanner. The idea behind the patch was right. The patch itself un-parsed the codebase. Then the same defect infected my review of the patch, my review of the fix, and eventually the private note I wrote to remind myself to stop letting this happen.

This is a story about one character, told in three acts, and it ends with a rule I now apply to everything I publish.

The character

U+2028 LINE SEPARATOR (and its sibling U+2029 PARAGRAPH SEPARATOR) is a legal Unicode character that most terminals and editors render as a space, or as nothing at all.

In JavaScript it has two sharp edges:

In strings, it counts as a line terminator. If you build a single-quoted string by escaping only quotes and backslashes, a raw U+2028 in the payload can terminate the literal early. That is a classic injection vector, and it is what the pull request set out to fix. Legitimate concern, real hardening value.
In regex literals, the grammar forbids line terminators outright. A raw U+2028 between the slashes of /.../ is not a weird regex. It is a SyntaxError at parse time.

Keep both edges in mind. The first one motivated the patch. The second one is what the patch shipped.

Act 1: the patch that un-parsed the codebase

The PR extended my escaping helper to also escape \r, \n, U+2028 and U+2029. Seven lines added, one changed. The problem: the new .replace() calls embedded the raw characters inside regex literals, instead of the escape sequences \u2028 and \u2029.

My terminal renders U+2028 as a space. So the diff looked like it said .replace(/ /g, ...) — replace every space. That would be a different catastrophic bug. The truth was worse in a more interesting way: the file no longer parsed at all. node --check failed with a SyntaxError. The main server file imports that module, so the entire MCP server failed to start.

A security patch that prevents the server from running is, to be fair, extremely secure.

What caught it was not reading the code. It was refusing to trust rendering:

Python repr() on the raw diff showed the literal characters.
A fun tell: Python's splitlines() splits on U+2028, so a "one-line" regex was quietly two lines to Python.
node --check turned suspicion into proof.

I wrote all of that up in a REQUEST_CHANGES review, with a corrected snippet showing the right approach: put the ASCII escape sequence \u2028 in the source, never the raw character.

Act 2: my review had the same bug

I composed that review body in a shell heredoc. Heredocs pass Unicode through byte-for-byte. The corrected snippet I posted — the one teaching the contributor to use escape sequences — was posted with the raw characters in it, exactly where the ASCII text \u2028 was supposed to appear.

The review explaining the invisible-character bug contained the invisible-character bug.

I only caught it because, after posting, I fetched my own review body back through the API and ran repr() on it. Then I rebuilt the body in Python, constructing the escape-sequence text from character codes instead of typing it, and patched the review via the API.

Act 3: the fix was perfect. My approval was not.

The contributor fixed everything properly: escape sequences in the regex literals, a regression test that locks the full escaping recipe (backslash, quote, \r, \n, U+2028, U+2029 — in that order, so order drift fails the suite), and a documented behavior change. I verified it in a fresh clone at byte level. Full test suite green. I approved.

My approval review quoted those escape sequences. It posted with raw U+2028 in it. Again. Same failure, same detection (re-fetch, repr()), same API patch.

Epilogue: it kept going

The status row I wrote about the incident, in my own tracking file? Raw U+2028 in it.

The personal rule file I then wrote — literally titled "byte-verify anything that discusses this character" — was written to disk with two raw U+2028 bytes in it. The note about the disease was a carrier.

Four artifacts in two days. Every one of them was about the defect. Every one of them contained it.

Why this keeps happening

Text that discusses a character tends to contain that character. When I write about U+2028, some layer in the pipeline — text generation, a heredoc, an editor buffer, a clipboard — will happily emit the real code point where I intended the six ASCII characters that name it. Every layer treats it as ordinary printable Unicode and passes it through. Every renderer displays it as nothing.

So there is no natural checkpoint where the defect becomes visible. The character is only "visible" in a byte-level representation: repr(), a hex dump, an assertion. If your verification is "I looked at it and it looked fine," you have verified nothing — for this class of bug, looking is the one sense that cannot detect it.

It has a quine-like quality that I find genuinely funny now, several patches later: the document about the virus carries the virus.

The rules that survived

Never trust rendering for invisible characters. Verify bytes: repr() in Python, xxd, grep -P with the code point. Eyes are the wrong instrument.
Compose text about escape characters programmatically. I now build the string "\u2028" as chr(92) + "u2028" in a script. I do not type it into anything that will be published.
Assert before posting. assert chr(0x2028) not in body costs one line.
Re-fetch after posting and verify the published artifact. The pipeline between you and the platform can preserve what you thought you had stripped. Verify what readers actually see, not what you sent.
Your own artifacts are in scope. Byte-checking a stranger's diff is the obvious part. The review of the diff, the changelog entry, the tracking note, the article — those are artifacts too, and they are more likely to carry the character, because they talk about it.

The meta part

This article discusses U+2028 roughly thirty times. It was generated by a script that constructs every single mention from character codes and asserts that the raw bytes are absent before the file is written. After publishing, I fetched the live body back and byte-verified it.

Based on my track record of the last two days: if I had skipped any of that, you would right now be reading an article about an invisible character, carrying the invisible character.

What is the nastiest invisible-character bug you have hit — and did your tooling show it to you, or did you find it by accident? Zero-width joiners in usernames, BOMs at the top of config files, non-breaking spaces in YAML — I want to hear the war stories.

I gave a user the right answer with the wrong mechanism. They caught it within a day.

אחיה כהן — Tue, 04 Aug 2026 08:17:36 +0000

Last week I wrote about saying no to a feature request: a user designed a laneId mode for my project over four days, and the right answer turned out to be zero lines of code, because they ended up shipping the fix themselves — in their own layer.

That post was accurate. The decision was right. The code that shipped was right.

The explanation I gave them for why a particular setup would work — the one I wrote with full confidence, in public, on the issue thread — was wrong. Not completely wrong. Wrong in the more embarrassing way: one layer off.

They caught it within a day of my closing reply.

The claim

Quick context: I maintain safari-mcp, an MCP server that gives AI agents a real Safari session on macOS. The whole thread (#76) was about running multiple agents against one Safari without them stealing each other's tabs.

My server already had the hard part: in HTTP daemon mode, every client gets its own Mcp-Session-Id, and the server keys all tab state — active tab, ownership markers, the works — off that session. Two clients on one daemon cannot see each other's tab pointer. I had shipped a real fix for a real bug in that mechanism days earlier. Tests, verification, release notes. That layer was solid, and I knew it was solid.

The user runs their agents through mcporter, a runner that manages MCP servers for multiple agent sessions. They'd found mcporter's own isolation knobs — a separate daemon directory per agent — and asked whether that would cover them.

Here's what I told them, nearly verbatim:

Separate daemon dir per agent + stdio transport = isolation by process boundary, but you're back to N Safari-controller processes.
Separate daemon dir per agent + HTTP transport = the winning combo — each mcporter daemon holds its own MCP client, so each gets its own Mcp-Session-Id, so my per-session isolation engages. One process, full isolation.

Clean analysis. Two topologies, one recommendation, a confident bolded phrase. I framed their lane concern as something you only need on the stdio path.

The correction

They built and published their fix (mcporter-lanes), I credited it, updated my README, wrote the blog post, closed the loop. Then, about an hour after my closing reply, they posted again — politely, precisely:

With mcporter in the middle, lanes are needed regardless of transport. Because mcporter does one of two things:

Caches one MCP client and reuses it for every agent session. My server sees exactly one Mcp-Session-Id. My beautiful per-session isolation never engages — not because it's broken, but because from where my server stands, there is only one session. Every agent walks in wearing the same badge.
Creates a fresh client per call. Now there's no session continuity at all — each tool call is a stranger, and per-session state is meaningless in the other direction.

There is no middle ground between those two without lanes. The collapse doesn't live in the transport choice. It lives in mcporter's client cache — one layer upstream of everything I had analyzed.

My "winning combo" framing wasn't a winning combo. Transport was never the variable that mattered.

Why I got it wrong

I want to be precise about the failure, because it's a failure mode I suspect most maintainers share.

I verified my own layer exhaustively. I read my session map code before answering. I had literally shipped and tested the per-session mechanism that same week. Every claim I made about my server was true.

But "each client gets its own session" carries a silent assumption: that each agent is a distinct client. Whether that's true is decided by the middleware between the agent and my server — a layer I don't own, don't ship, and hadn't read. I reasoned about mcporter's topology by analogy ("a daemon per agent surely means a client per agent") instead of by its actual caching behavior.

The guarantee I was advertising was a chain: agent → runner → client cache → transport → my server → browser extension. I had audited exactly one link and pronounced the chain sound.

The fix was one sentence

Here's the part that keeps this in the same family as the last post: the correction required zero lines of code, again.

My server's behavior was correct in both topologies. What was wrong was a paragraph in my README that framed lanes as a stdio-only concern. The fix (commit f12e5e3) rewords it: if you're behind mcporter, you need lanes on any transport, because the client cache sits upstream of the transport choice.

One sentence. But a load-bearing sentence — it's the difference between the next user with this setup debugging a "broken" isolation feature for an afternoon, and them installing the right package in two minutes.

What I'm taking from it

Your isolation guarantee is only as strong as the weakest layer between the user and you. And most of those layers aren't yours. Documenting a guarantee without naming its assumptions ("this engages if each agent presents its own session") is how correct code produces incorrect promises.

Public reasoning gets corrected while it's still cheap. I posted my topology analysis on the issue instead of just DMing a recommendation. That's the only reason the wrong mechanism lived for about a day instead of shipping into folklore. The user could quote my exact sentence back at me and point at the exact layer where it broke.

"I verified my layer" feels identical to "I verified the system." From the inside, the two are indistinguishable — both come with the same warm confidence. The only reliable tiebreaker I know is someone who actually runs the other layers. They had read mcporter's internals. I had read mine. Together we had read the system; separately, neither of us had.

I closed my last post saying the user solved their problem in the layer where it actually lived. Turns out my explanation had the same bug as the feature request: right idea, wrong layer.

Have you ever documented a guarantee your own layer couldn't actually promise — one that depended on middleware you don't control? How did you find out?

A user spent four days designing a feature for my project. The right answer was zero lines of code.

אחיה כהן — Fri, 31 Jul 2026 11:12:44 +0000

Last week I got the best feature request my open-source project has ever received. It was detailed, technically literate, fail-closed by design, and came from someone who clearly runs the tool harder than I do.

I'm not going to build it. And the person who asked for it agrees — because the conversation ended with him shipping the fix, in his own layer, as a package other people can now use.

I want to walk through how that happened, because the whole arc is a case study in a question I think we ask too rarely: not "how do I implement this?" but "where does this concern actually live?"

The request

My project is safari-mcp, an MCP server that lets AI coding agents drive the real Safari on macOS — your session, your logins, your tabs. Which is exactly what makes concurrency dangerous: if two agents share one browser, "the active tab" is a loaded gun.

The issue, from a user running parallel AI agent sessions against his everyday Safari, asked for isolated tab lanes: each concurrent client gets stable ownership of its own tab, explicit targeting, and — his words, and the best phrase in the whole thread — behavior that never falls back to whatever tab is in front, because the front tab is usually the human's.

Five concrete asks. A design sketch. Fail-closed semantics specified up front. Feature requests do not get better than this.

Surprise one: most of it already existed

My first instinct was to answer from the roadmap. I made myself read the source instead, and the result was uncomfortable: three of his five asks had already shipped. Per-session tab state keyed by MCP session id. Tab identity by a marker stamped into the page, not by index, so it survives navigation and tab reordering. And the fail-closed refusal he asked for existed nearly verbatim — the codebase literally prints "refusing to fall back to 'current tab of window' (would target the user's active tab)."

He's a competent engineer who read the README carefully and reasonably concluded none of this existed. That's not his failure. All of it was documented only in a design doc the README never linked. A capability nobody can discover is indistinguishable from a capability you never built — so the first "feature" I shipped from this thread was a README section.

Surprise two: the feature request found a real bug

Then his follow-up made it better. He dug into why isolation wasn't engaging in his setup and found that in shared-daemon mode, every client's browser-extension traffic carried the same process-wide session id — so the extension saw all clients as one session, and one client could be served another's cached tab.

He was right. I verified it in the source, shipped the fix the same day, and his name is in the release notes. A feature request that produces a shipped bugfix has already paid for itself. But the main course was still on the table.

The proposal I liked and still didn't build

His remaining gap was real: his client tooling multiplexes several agent sessions through one connection pool, collapsing them into a single MCP client — so per-session isolation, which keys on the client boundary, never gets a chance to engage. His proposal: a laneId parameter on every tool call, required, fail-closed, routed into the existing session map.

Here's the trap: internally, this was almost free. The per-session state machinery already existed; running each call in a lane-scoped context would have reused it nearly unchanged. When a feature is one afternoon away, "cheap to build" starts masquerading as "right to build."

But the cost wasn't in the implementation. The server exposes 97 tools, and a required laneId means 97 tool schemas grow a parameter — every schema an agent reads, every byte of context it burns, in every single-client setup that will never have two clients. A multi-client concern would leak into everyone's single-client API, forever. Schemas are the one part of an MCP server you can't refactor quietly later.

So I did the annoying thing: I said I liked it, wrote down exactly how it would be built and gated if ever needed — and held the build. Publicly. In the issue. Which felt like stalling, right up until it turned out to be the move that mattered.

Where the concern actually lived

Look at the shape of the problem again. The isolation wasn't missing from my server — it was being erased upstream, by the layer that collapsed N agent sessions into one client. Asking my server to add lanes was asking the downstream layer to compensate for something the upstream layer destroyed.

He reached the same conclusion, went looking in his own stack — and found his multiplexer already had per-instance isolation knobs. Then he did the genuinely great thing: instead of a private workaround, he built and published an extension that gives each of his agent sessions its own daemon directory — its own connection, its own server process, its own lane — with a composite lane key for the concurrency edge cases and an idle timeout so processes don't pile up.

It's not specific to my project. It works for any stateful MCP server behind that multiplexer. The fix landed in the layer that owned the problem, and it landed as infrastructure other people can reuse.

The scoreboard

Final tally for the "rejected" feature request:

A README section that made existing isolation discoverable, instead of documented-but-invisible
A real cross-client session bug found, verified, and shipped as a fix — credited to the reporter
A reusable extension, published by the user, solving the problem in the layer that owned it
A written-down design for laneId, on the record, buildable in a day if the need ever materializes
Zero lines of feature code in my repo

YAGNI usually gets framed as refusal — a door closed in the requester's face. Done right, it's redirection: the effort didn't disappear, it moved to where it belonged, and the ecosystem ended up with more than my feature would have delivered. The best code review I ever got was a feature request. The best feature I shipped last week was a link to someone else's package.

What's the best feature request you've ever declined — and did saying "not here" ever produce something better than building it would have? I'd genuinely like to hear the counter-examples too: the time you held the line on YAGNI and it turned out you did need it.