A worksheet in prod stopped saving and caused the whole CockroachDB cluster to stop accepting writes.
The pod was healthy: 404 MiB of a 2 GiB limit, 655m of CPU out of 1500m, zero restarts, zero alerts. Everything looked perfectly healthy and normal, yet the database refused writes. 3 active queries across the whole cluster, 12% CPU, all three nodes up and running. No OOM, nothing had crashed but the services still couldn't write.
The document was 155 KiB. Its range in CockroachDB was 1GiB. How did this happen? That is the interesting part.
The software
The software is a collaborative editor. Teachers build worksheets, lesson plans, whiteboards, and teaching documents. A few people can be in the same document at the same time. Every document is a CRDT built on Loro. The browser keeps a replica, applies your edits locally, then ships them over WebSocket to a sync server. The server keeps the copy in memory, merges all the incoming copies, and writes the result into CockroachDB v25.
The important part is how the result is written. To persist a document, it exports the entire Loro doc as a snapshot and saves it to a BYTEA column, on a single row. Normally CRDT are stored as an append-only log of updates. We wanted a simpler structure, semantic search, and fast fetch so we decided to store the whole doc in a single row instead of an append-only log.
A wild goose chase
I got paged that the app isn't working. I hopped on even though it was 9 PM and started investigating.
First clue, object storage is down. The logs were full of it, hundreds of lines of thumbnail and image fetches timing out against storage.googleapis.com. That can't be right, Google storage surely wouldn't be down. I checked the status page and it was up. I thought maybe its a permission or CORS thing, so I ran a curl from a pod on the same node as the server, pulling one of the URLs that were timing out. HTTP 200, 1.1MB downloaded. So Google was in fact not down, and there were no network or permission issues either, meaning the issue was our clients. But that wouldn't block writes on our database, so I left this aside to investigate the database issue. I don't know why I got sidetracked and lost a bit of time to this.
New contender, I was able to see a few pool timed out while waiting for an open connection logs. Very convincing but that also ended up being wrong. I checked the number of queries and number of open connections, but it was only 3. The containers were CFS throttled, 44 to 48% against the limit but the nodes were running at 85 to 97%. Starving an async runtime of CPU and every timeout starts firing even when things are healthy. The pool wasn't exhausted, the tasks holding the connections weren't getting scheduled to return the connections. This was occurring infrequently, and wasn't the issue I was looking for.
Third time is the charm. I noticed the payload sizes varied largely and considered that as the clients sending incremental deltas, which meant the volume was real edits and there was no bug to find here. However, what I observed were different documents. For the same document, the payload size varied little to none. A CRDT snapshot of a document that keeps changing comes out a slightly different size every time, not such a large variance. The check that settles this was dividing the received bytes by what was stored. Full credits to Claude Code on this one because I hadn't even considered this could be an issue.
| document type | writes per resource | payload ÷ snapshot |
|---|---|---|
| worksheet | 995 | 0.81 |
| lesson plan | 57 | 0.94 |
| whiteboard | 5.9 | 0.87 |
| text document | 18 | 0.82 |
All pretty close to 1.0 i.e. every type was sending as many bytes as the entire saved document. This wasn't a delta. Worksheets weren't doing anything different from the other document types, they were just doing it 169 times more than the whiteboards.
The actual problem
It was in the logs the whole time, but buried under everything else:
split failed while applying backpressure to Put [/Table/111/60/"..."/0]
on range r725: could not find valid split key
Four conditions that make this happen.
CockroachDB uses MVCC, so a write never overwrites anything. Every write stores a fresh copy of the row under the same key at a new timestamp. The old copies stay exactly where they are until they are eventually garbage collected. The key on disk is the row plus the timestamp, so the same row written twice is two keys sitting next to each other on the same row. This is what lets a transaction read a consistent snapshot without locking. This is what allows AS OF SYSTEM TIME, follower reads, and incremental backups to function. Old versions can't be dropped immediately, in case they need to be read by something. They are later picked up by the GC queue once they are older than gc.ttlseconds, which was configured to 4 hours on our cluster. This means a row takes up its size multiplied by the number of times it was written in the GC window.
The whole row is one key CockroachDB stores a row as one key per column family, this table never defined any beyond the default, so the whole thing sits in one. One Loro doc, one key, no matter the snapshot size.
A split has to cut between two keys Ranges are kept under range_max_bytes by splitting. The split picks a key and cuts there, everything below goes to one range, everything above to the other. If every byte in the range is one key and the copies differ only by timestamps, there is no split boundary. So all the versions live together and the range keeps growing.
The client kept pushing every 2.2 seconds whether anything had changed or not. The new build deployed earlier had introduced this bug, so the same document was being pushed over and over and over as long as the tab was open.
This is what the range actually looked like:
keys 1
versions 6766
val_bytes 1024.02 MiB
live 0.151 MiB
One key. Nearly seven thousand copies of it. A gigabyte of stored versions sitting on top of 155 KiB of actual data.
The math: 0.47 writes/s for a 4 hour ie 14,400 second GC window. counts to 6768 copies, against measured 6766 copies. All of them are almost the same.
Why it can't self-heal
Splitting does not happen on the write path, which was surprising but it made sense. Each one of the 6766 writes was proposed, replicated, committed and visible for the next read. The split runs later on a timer, from a queue. This is based on the MVCC stats. The split isn't a local thing, it is a distributed transaction that updates the meta ranges for the whole cluster and describes where the keys live.
There is always a gap between this range is too big and the range got split, and normally the gap is closed within seconds by the queue. Backpressure is what stops the range from outrunning the queue. At 2x range_max_bytes, the KV layer stops letting writes in:
range_max_bytes 536,870,912 (512 MiB)
backpressure at 1,073,741,824
r725 1,073,844,534
The write isn't rejected immediately, it gets batched. The batch waits for the range to get split apart and come back under the limit again. The write fails when it runs out of time waiting for the split. That's why we saw a lot of persist timeout errors. The assumption is built on the backpressure: the split you are waiting for is going to happen soon. Ours was 100 KiB over the backpressure limit, so the split was not going to happen. This leaves GC as the only thing to save the day. GC also runs on a queue but it can't touch anything inside the TTL window, 4 hours in our case.
Raft never failed
This wasn't a distributed systems failure, even though it might look like one from the outside. No quorum loss, no election, no replication trouble. Raft was healthy the entire time and it is the reason the size limit actually exists.
A range isn't a storage bucket, it is a Raft group, three replicas by default with one holding the lease. Every write to that row is a Raft proposal. It requires quorum acceptance, and each replica gets its own copy. So the 6766 writes were each a round of distributed consensus. Each write shipped the full 155 KiB snapshot across the network and each replica built up the gigabyte backlog.
The size limit exists for recovery. If a replica falls behind and can't catch up from the log, it gets the full snapshot from the leader. The leader truncates the log so it can send the entire range as a snapshot over the network when required. If a node is decommissioned and a new replica takes its place, it recovers from the full snapshot. Transferring more than 1 GiB over the network certainly has its cost and the replica won't help the quorum until it catches up. I was a little surprised that the limit was as large as 1 GiB. Keeping the ranges small is what allows fast recovery and cheap reshuffling of replicas.
Initially, I thought this was perhaps a page split on the storage engine, as I had been writing a B+ tree recently. However, CockroachDB uses Pebble, which is an LSM tree. There are no pages here. Splitting a range is all about distribution across a key space, not storage layout.
A tab nobody closed
Two thousand consecutive pushes for the wedged document:
payload size min 130,009 median 130,009 max 130,009
distinct clients 1
inter-write gap p50 2.24s
All of the writes were the exact same. One client was sending the same document over and over every 2.2 seconds.
The simplest explanation was that someone left a tab open. On the client, the network gate was "did any command run during this dispatch?" instead of "did the document change?". A layout loop that measures rendered block heights kept producing command work, so the client kept re-exporting and re-sending the whole document. We didn't have any comparison for whether the bytes actually changed or not, and simply accepted every write from the client.
Five solutions
Raise range_max_bytes The first thing we considered was to raise the range limit and allow the split to occur. This simply removes the ceiling for every range but does not address the 6766 copies we had piled up. We also didn't know what kind of consequences this would have with Raft so we dropped the idea.
Lower gc.ttlseconds This is what we did since it didn't require a hotfix deployment. A hotfix would require code changes, code review, image build, and release. That can come later once the database is working again.
ALTER TABLE resources CONFIGURE ZONE USING gc.ttlseconds = 600;
We couldn't change the write rate without shipping code, and even then the frontend wouldn't really update until they refreshed the tab. We could add the duplicate write guard on the backend but it would take a while to deploy everything and still take a couple of hours for the GC to pick up the old copies. The easiest solution was to lower the GC window so the next run would clean up all of the old copies and free up the range. I checked the write operations immediately after applying and everything was back to normal.
Skip the coalescing persist queue There was already a PR open for this. 750ms debounce, 5-second ceiling. This would reduce the write rate to ~2880 versions per GC window. This would require a document to be at least 364 KiB to cause the same issue. There is some breathing room, but its better to fix it permanently. I asked the author to change the dispatch mechanism to "did anything actually change?" instead.
Deduplicate writes by hashing the snapshot The real fix that we shipped the next day. We hash the snapshot we receive and check against the currently stored value. If the snapshots are the same then the write is skipped. This would fix the issue even when the frontend didn't refresh the tab to pick up the dispatch mechanism fix. We added logs to watch the Dedup counter to make sure it worked correctly, and we saw the log volume gradually decrease over time.
Move to an append-only log This would require a big refactor and we didn't have a requirement for this yet.
We didn't receive an alert
None of our alerts caught this. The thresholds weren’t wrong, OOMKilled, MemoryHigh, CpuThrottled, Down, CrashLooping, none of that happened. The service and the database both looked completely healthy, just quietly failing writes.
CockroachDB actually exports a metric for this. queue_split_process_failure increases every time the split fails. Healthy clusters don't fail splits, so rate(queue_split_process_failure[15m]) > 0 tells you that there is trouble afoot in the database. I added an alert for this one the next day.
Closing
None of it was broken. MVCC kept the old versions as it was designed to, Raft kept the quorum, split queue and backpressure worked just fine. The client kept saving, over and over, from a tab that was left open. Four conditions that happened to meet at once brought down the database.
Top comments (1)
The “retained bytes = write rate × version size × retention window” equation is an excellent operational signal. I would alert on that projected value and on version-bytes/live-bytes ratio long before range backpressure, segmented by key so one hot document cannot hide inside table-level averages.
For the hash-skip fix, the comparison also needs concurrency semantics. If multiple sync servers can persist the same document, make the write conditional on the currently stored content/version digest. Otherwise two writers can both observe an old digest and still create redundant versions. Bind the digest to a canonical serialization format and schema/version so an encoder change is not mistaken for a content edit.
A content-addressed snapshot blob plus a small current-pointer row can reduce repeated large MVCC values further: identical snapshots reuse the blob, while the hot key updates only a digest/version pointer. The pointer still needs optimistic concurrency tied to the CRDT state/version.
Finally, lowering GC TTL is a recovery lever with a documented blast radius. I would turn the protected-timestamp and historical-read checks you performed into a preflight gate, then add a restore/CDC/AS-OF canary before keeping the shorter TTL. That makes the emergency setting a controlled compatibility change rather than just a storage knob.