DEV Community

Cover image for Self-Hosted Chatwoot: 5 Failures the Docs Don't Warn You About
אחיה כהן
אחיה כהן

Posted on

Self-Hosted Chatwoot: 5 Failures the Docs Don't Warn You About

I run self-hosted Chatwoot as the WhatsApp inbox for a dozen or so small Israeli businesses. Two servers, a few thousand conversations a week, a drip-sequence engine bolted on the side.

Chatwoot is good software. The self-hosting docs will get you to a running container. What they will not tell you is which failures actually happen at month six, when you have real customers and real volume. These five all bit me in production, and none of them looked like what they were.

1. Your disk fills from somewhere Postgres never sees

I got a disk alert at 86 percent and immediately went looking at the database. That was the wrong place.

DB (postgres):            680 MB
chatwoot_storage_data:     17 GB
Enter fullscreen mode Exit fullscreen mode

Attachments live in ActiveStorage, on a Docker volume, not in Postgres. Every image, voice note, and PDF a customer sends is a file on disk, and none of it shows up when you check database size. If your monitoring watches the DB, it will report everything is fine right up until the container cannot write.

The growth curve is a function of how many accounts you host, not how busy any one of them is. Mine sat at roughly 0.05 GB a month until I onboarded seven new businesses over two months, and then it hit 16 GB a month.

Check the right volume:

docker system df -v | grep chatwoot_storage_data
Enter fullscreen mode Exit fullscreen mode

2. Forty-four percent of my outbound storage was duplicate files

This is the part that surprised me. When I actually measured what was on that volume, almost half the outbound media was byte-identical copies of the same file.

One 14.5 MB video was stored 48 separate times. One image was stored 325 times.

Chatwoot creates a new blob and a new file on disk on every send, even when the bytes are identical. That is correct behavior for a chat app where every message owns its attachment. It becomes expensive the moment you have anything that fans one file out to many conversations. In my case it was not campaigns at all, it was the drip engine sending the same media to 48 separate conversations as ordinary outbound messages, each one a fresh readFile plus fd.append('attachments[]', ...).

Deduplicating is safe, and I verified this in the Rails source before touching anything. ActiveStorage::Blob#purge is guarded by a foreign key on active_storage_attachments.blob_id, so deleting one message will not take out a file that other messages still point at:

# ActiveStorage 7.1.5.2
def purge
  destroy
  delete if previously_persisted?
rescue ActiveRecord::InvalidForeignKey
end
Enter fullscreen mode Exit fullscreen mode

One pass over existing blobs, matching on checksum and byte_size within the same account, reclaimed 3.20 GB across 7,160 blobs and took the disk from 86 percent to 71 percent. You want an index on active_storage_blobs (checksum, byte_size) before you try this, and a size floor so you are not doing lookups for every 4 KB thumbnail.

3. POST /messages returns 200 before anything has been sent

This one cost me an outage, and it is entirely my own fault for reading the status code as confirmation.

POST /api/v1/conversations/{id}/messages returns 200 immediately. All it has done is insert a row into public.messages with status = 0 and no source_id. The actual delivery to Meta happens later, in a Sidekiq job on the high queue. The source_id, which is the WhatsApp message ID, only gets written once Meta acknowledges.

So a tight send loop looks completely healthy from the client side while it quietly fills a queue that everything else also depends on. Four resend runs kicked off within ninety seconds, roughly thirteen messages a second, pushed 5,108 messages through that endpoint. The high queue grew to 3,786 jobs with eleven minutes of latency, and every inbound message from an actual paying customer sat behind them.

Nothing errored. The dashboard just showed a lot of pending clocks, which in Chatwoot means "waiting for delivery receipt" rather than "scheduled" — a distinction I have now confused twice.

Measure queue depth, not HTTP status:

docker exec chatwoot-sidekiq-1 bundle exec ruby -e '
  require "sidekiq/api"
  q = Sidekiq::Queue.new("high")
  puts "#{q.size} jobs (latency #{q.latency.round}s)"'
Enter fullscreen mode Exit fullscreen mode

The fix that actually held was a backpressure check in my own sender: every 40 sends, count my messages still sitting at source_id IS NULL AND status = 0. Above 250, pause until it drops under 80, with a 120-second ceiling. That needs no access to Chatwoot's Redis and it measures the right thing, which is the pressure I created rather than global queue depth.

4. Deleting an inbox is a Rails-level cascade, and it is silent

During a WhatsApp Business Account migration I deleted an inbox. Here is what went with it:

894    conversations
6,160  messages          (471 of them lead replies)
1,776  contact_inboxes
Enter fullscreen mode Exit fullscreen mode

Contacts survive, because they live at the account level. Everything else is gone. This is a dependent: :destroy cascade in the Rails models, not a database constraint, so nothing in Postgres warns you and there is no confirmation dialog proportionate to what is about to happen.

The part I did not anticipate: the real blocker afterwards was not the lost conversation history. It was contact_inboxes. Without those rows, nothing can open a conversation at all — my engine just started returning no WhatsApp contact_inbox for every contact. Conversation history is nice to have. contact_inboxes is load-bearing.

The consolation is that anything you keep in your own schema survives, since it has no foreign keys into Chatwoot's tables. My sequence enrollments came through untouched, so nobody's position in a drip sequence was lost.

5. Restoring from backup has three traps that all look like data loss

I had a backup. Restoring it still took most of a day, because of three things that each make it look like the restore failed when it has not.

A trigger overwrites your display_id. conversations_before_insert_row_tr calls nextval on a per-account sequence, BEFORE INSERT, so every conversation you inject with an explicit display_id silently gets a brand new one. Every foreign reference you were trying to preserve detaches. The way through is to insert, then UPDATE ... SET display_id from a staging table (UPDATE does not fire that trigger), then setval the sequence to the real maximum.

There are unique indexes that are not constraints. They do not appear in pg_constraint, so if you go looking for what you might collide with, you will not find them:

contact_inboxes (inbox_id, source_id)
conversations   (account_id, display_id)
conversations   (uuid)
Enter fullscreen mode Exit fullscreen mode

ON CONFLICT (id) DO NOTHING sails straight into all three. Use ON CONFLICT DO NOTHING with no target.

One skipped row 500s the entire UI. A conversations.contact_inbox_id pointing at a contact_inbox you did not inject produces undefined method 'source_id' for nil, which surfaces as an infinite spinner across the whole dashboard rather than a broken single conversation. Remap every orphaned contact_inbox_id before you declare the restore done.

Worth knowing: contact_inboxes.source_id is just the phone number in E.164 without the leading + (so +972 50 000 0000 becomes 972500000000). It does not depend on which WABA you are on, which means you can rebuild these rows from contacts.phone_number even with no backup at all.

Bonus: "timeout exceeded when trying to connect" is not your database

The dashboard stopped loading with timeout exceeded when trying to connect. The same error appeared on my background ticks, which meant sending had stopped for every client. It looked exactly like Postgres falling over. Postgres was fine.

The cause was one OR inside one NOT EXISTS:

NOT EXISTS (
  SELECT 1 FROM ledger s
  WHERE s.campaign_id = m.campaign_id
    AND (s.message_id = m.id
         OR (m.source_id IS NOT NULL AND s.source_id = m.source_id))
)
Enter fullscreen mode Exit fullscreen mode

With the OR in there, Postgres can only hash on campaign_id. The rest becomes a Join Filter evaluated across every pair in the bucket. On 23K messages against 23K ledger rows that is Rows Removed by Join Filter: 22,371,572 and 125 seconds, to return zero rows. The cost is quadratic in campaign size, so it only detonates for your largest customers.

The connection pool did the rest. It is max: 5 and shared across every client's API requests plus the background ticks. Four of these queries at once starved it, and everything else died on a 10-second connection timeout. A slow query in one tenant took down every tenant.

The fix is De Morgan, ¬(A∨B) ≡ ¬A ∧ ¬B — split into two NOT EXISTS, each with a complete equality condition to hash on:

NOT EXISTS (SELECT 1 FROM ledger s
            WHERE s.campaign_id = m.campaign_id AND s.message_id = m.id)
AND NOT EXISTS (SELECT 1 FROM ledger s
                WHERE s.campaign_id = m.campaign_id
                  AND m.source_id IS NOT NULL
                  AND s.source_id = m.source_id)
Enter fullscreen mode Exit fullscreen mode

125,366 ms to 381 ms. I verified equivalence with a bidirectional EXCEPT against production across every account before shipping it, including the one account that actually had legacy rows the filter was there to catch.

The pattern

Four of these five presented as something other than what they were. A disk alert that was not the database. A 200 that had not sent. A timeout that was not the database either. A restore that looked like it had lost data it had not.

Self-hosting Chatwoot is genuinely worth it at this scale, and I would make the same call again. But budget your operational attention for the layer between the container and your own code, because that is where all of this lives. Most of what I have learned here came out of running the WhatsApp automation I build for Israeli businesses on top of it, which is to say it came out of breaking things in front of paying customers.

One I have not solved: has anyone found a clean way to get Chatwoot to tell you a message actually reached Meta, without polling source_id yourself? I would rather subscribe to something than poll a column, and I have not found the hook.

Top comments (0)