Two writers, one database: why your self-hosted app breaks the moment it runs twice

I keep running into the same failure in the self-hosted world, and it took me an embarrassingly long time to name it the first time round. The app works fine as one container. You add a second replica, or a second backend process, and everything goes strange: logins stop sticking, requests hang, or the logs fill up with database is locked. Nothing crashes loudly. It just quietly stops making sense.

Two separate mistakes tend to get made at once when you run a second copy of something that was built as one instance, and both of them look like somebody else's bug.

Your old session dies, but a fresh one works

This is the one that fools people. You log in, the page works, you click something and you get bounced back to the login screen. Open an incognito window, log in again, and it behaves (at least for a while).

What's happening is that the instance that answered your login is not the instance answering your next request. If the app auto-generates a signing secret on first start, and Open WebUI calls this WEBUI_SECRET_KEY, each copy invents its own. Instance A signs your session cookie, instance B tries to verify it with a different key, and the honest answer it gives you is 401. A fresh login works because it gets minted and used by whichever pod you landed on.

On Kubernetes this sometimes arrives as an Envoy message instead of a clean 401: upstream connect error or disconnect/reset before headers. That's transport-level noise sitting on top of a state-level problem.

Or the logs say database is locked

The other half of this is a file that only tolerates one writer. SQLite allows exactly one writer at a time. If a writer can't take the lock it returns SQLITE_BUSY and the app reports database is locked, unless a busy handler is configured to wait. Write-ahead logging lets readers and writers share on the same machine, but the wal-index lives in an mmapped -shm file, so WAL does not work across machines, and the SQLite docs say it plainly: the WAL implementation will not work on a network filesystem. Rollback journals are no better, they use fcntl() locks that are broken on plenty of NFS implementations.

So the tempting fix, "put the .db on a shared volume and scale the replicas", isn't a fix at all. It's a faster route to corrupt data.

First, count your writers

Before changing anything, prove how many copies are running and who owns the state file.

# processes and pods
pgrep -af "serve|gateway|uvicorn"
docker ps --format '{{.Names}}\t{{.Ports}}'
kubectl get pods -o wide

# who holds the file and the port
lsof /path/to/webui.db          # or: fuser -v /path/to/webui.db
ss -ltnp | grep 8080

Two rows where you expected one is the whole diagnosis. Also look for stale PID or lock files in the app's data directory. A process that died badly leaves its lock behind, and the next start either refuses to boot or quietly becomes a second writer.

The fix, in order

1. Get back to one writer first. On macOS actually quit the app (Cmd+Q, not just closing the window), or run the app's own gateway stop, or kubectl scale deployment open-webui --replicas=1. Recovery before redesign.
2. Give every replica the same secret. One openssl rand -base64 32 value, injected into all pods as WEBUI_SECRET_KEY (and OAUTH_SESSION_TOKEN_ENCRYPTION_KEY if you use OAuth). This is the step people skip, because everything boots fine without it.
3. Move the database out of the container: DATABASE_URL=postgresql://user:pass@db-host:5432/openwebui. One warning the docs are clear about, Open WebUI does not migrate your existing SQLite data into Postgres for you. Do this before you have production data, or plan the export.
4. Externalize coordination as well: REDIS_URL=redis://redis-host:6379/0, plus WEBSOCKET_MANAGER=redis and ENABLE_WEBSOCKET_SUPPORT=true. Without Redis, websocket handling and config sync stay in-process, and multi-instance users get 403s and intermittent auth weirdness that looks nothing like a database problem.
5. Check the side databases too. The default ChromaDB vector store is SQLite-backed and not fork-safe, so vector search falling over on two replicas is the same disease in a different organ. Swapping to VECTOR_DB=pgvector puts it in the Postgres you already have.
6. Orchestrator hygiene: keep UVICORN_WORKERS=1 per container, and let exactly one replica run migrations (ENABLE_DB_MIGRATIONS=false on the rest).
7. Verify instead of hoping. Compare kubectl exec <pod> -- printenv WEBUI_SECRET_KEY across both pods, then log in and roll the deployment while keeping that session alive. Watch grep -c "database is locked" in the logs stop climbing.

Things that don't fix it

A shared NFS or PVC volume holding the SQLite file, with two replicas on top, doesn't fix anything. The Open WebUI docs warn you'll get database is locked and data corruption, and the SQLite WAL docs explain why the shared-memory file can't cross hosts anyway.

Sticky sessions don't fix it either...... sessionAffinity: ClientIP only hides the problem until a rollout moves your pod, and then everyone is logged out at once.

Copying a live .db around with docker cp or plain cp is worse than it looks. A copy taken mid-transaction is a mix of old and new pages, and if there's a -wal or -journal file sitting next to it, that has to travel with it or your copy is unusable. Use VACUUM INTO, the backup API, or sqlite3_rsync on 3.47 and newer.

Raising the busy timeout buys you nothing. No pragma makes SQLite safe on network storage.

The honest caveats

The Kubernetes issue I started from is still open, and the maintainers there haven't confirmed a root cause (fwiw, treat that particular attribution as community reasoning rather than gospel). The SQLite documentation is firm about network filesystems but never says the word Docker, so the bind-mount case is inference on my part, not a quote. And if your app ships a single-instance lease or lock, use it. The lease is the real fix, everything above is a workaround.

I could be wrong about the specifics of your stack, but if you're seeing session weirdness that only shows up once a second copy is running, this is the first place I'd look. It cost me a couple of hours of staring at logs the first time.

Sources worth reading: SQLite WAL, SQLite FAQ, locking, how to corrupt, and Open WebUI's scaling and hardening pages.

Comments

[Raknaos]

Naming the auto-generated secret key as the real culprit is the part most writeups skip. We got bitten the same way: single container worked, second replica silently broke auth, incognito masked it because the cookie got minted and verified on the same pod.

Our rule now is any signing secret is injected from outside (env or vault), never generated on boot. Did you cover the database-is-locked half in a follow-up? The session-key story and the shared-sqlite write-contention story feel like two separate incidents to me. Good one.