DEV Community: Vatsal Patel

Why CockroachDB refused writes to a healthy 155 KiB row

Vatsal Patel — Tue, 11 Aug 2026 16:10:53 +0000

A worksheet in prod stopped saving.

The pod was healthy. 404 MiB of a 2 GiB limit, 655m of 1500m, no restarts. I didn't believe that, so I went and looked at the database too. Three active queries cluster-wide, 12% CPU, all three nodes live. Idle.

Nothing was exhausted, nothing had crashed, and the service still couldn't write.

The software

It's a collaborative editor. Teachers build worksheets, whiteboards and lesson plans, and several people can have the same document open at once. Every document is a CRDT, built on Loro. The browser holds a replica and applies edits to it locally, then pushes them over a WebSocket to a sync server. The server keeps its own copy of each open document in memory, merges whatever arrives into it, and writes the result to CockroachDB v25.x.

That last step is the one that matters here. Persisting a document means exporting the entire Loro doc as a snapshot and writing it into a single BYTEA column, on a single row. Not an append-only log of updates, which is the usual way to store a CRDT. The whole document, on every save.

The document that stopped saving was 155 KiB. Its range was 1 GiB.

A wild goose chase to find the root cause

The red herring: the same service had an unrelated CPU problem running that day, readiness probes flapping, the node pegged, hundreds of timeout errors in the logs. I went through all of it. Every bit real, none of it connected to this. Two separate problems on one service on the same day, and the louder one wasn't the one refusing writes.

A second false trail: I noticed payload sizes varied a lot from one document to the next and read that as clients sending incremental deltas, which would make the write volume real edits.

That was wrong. Varying payload size doesn't imply a delta. A CRDT snapshot of a changing document is a different size every time, the same way two zip files of slightly different inputs come out different sizes. The check that settles it is dividing the payload by the stored snapshot:

document type	writes per resource	payload ÷ snapshot
worksheet	995	0.81
lesson plan	57	0.94
whiteboard	5.9	0.87
text document	18	0.82

Near 1.0 means the client sent as many bytes as the entire stored document. Every document type was doing it. Worksheets weren't doing anything different in kind. They were doing it 169 times more often than whiteboards, and that was the whole difference between a wasteful system and a broken one.

How to wedge a CockroachDB range

The error was in the logs the whole time, buried at a much lower volume than the noise.

split failed while applying backpressure to Put [/Table/111/60/"..."/0]
on range r725: could not find valid split key

Four things had to be true at once for that, each one reasonable on its own.

CockroachDB is MVCC (Multiversion Concurrency Control), so a write never overwrites anything. Every write stores a new copy of the row under the same key at a new timestamp, and the previous copies stay exactly where they are. The key in the storage engine isn't the row; it's the row plus a timestamp.

That's what lets a transaction read a consistent view of the database without locking the rows it reads. A transaction reading at timestamp T sees the newest committed version at or below T of every key it touches. CockroachDB runs SERIALIZABLE by default and there is more machinery than that behind it, since reads leave marks in the timestamp cache that push later writers, and a read that meets an unresolved intent below its own timestamp has to wait on it. But keeping every committed version around is what the rest is built on top of. It's also what AS OF SYSTEM TIME, follower reads and incremental backups are built on. All three are reads at an older timestamp, and they only work if the data as of that timestamp is still on disk.

So old versions can't be dropped at write time. Something has to guarantee they're still there for anyone reading in the past. They get collected later by the MVCC GC queue, once they're older than gc.ttlseconds, which was four hours here. Which means the storage a row occupies isn't its size. It's its size multiplied by how many times you wrote it in the last four hours.

The whole row is one key. CockroachDB stores a row as one key per column family, and this table never defined any beyond the default, so every column sits in the same one. One document, one key, however large the snapshot gets.
A split has to cut between two keys. Ranges are kept under range_max_bytes by splitting, and a split picks a key and cuts the keyspace there: everything below goes to one range, everything above to the other. If every byte in a range belongs to one key and the copies differ only by timestamp, there's nowhere to put the boundary. They can't be separated anyway, because the range is what serves reads of that key at any timestamp, so all of them have to live together.
The client was pushing every 2.2 seconds, whether or not anything had changed.

Here's what the range actually looked like:

keys       1
versions   6,766
val_bytes  1024.02 MiB
live       0.151 MiB

One key. Nearly seven thousand copies of it. A gigabyte of stored versions against 155 KiB of actual row.

0.47 writes per second against a 14,400 second GC window predicts 6,768 versions. There were 6,766. The range was holding exactly one GC window of writes, which is where this stopped being a mystery and became arithmetic. I liked that part a lot.

Nothing was queued or deferred to get there, which is worth being explicit about. Every one of those writes applied immediately: proposed, replicated, committed, visible to the next read. The range grew because that is what a range does when you write to it. Splitting is not part of the write path.

Splitting happens on the split queue. Each store walks its replicas on a timer, reads their size straight off the MVCC stats it already maintains, and queues anything over range_max_bytes. That's deliberately asynchronous, because a split isn't a local operation. It's a distributed transaction that carves the keyspace in two, writes a new range descriptor, and updates the meta ranges that tell the rest of the cluster where keys live. You don't want that on the hot path of a Put.

So there's always a gap between "this range is too big" and "this range has been split", and under normal load, the queue closes it in seconds. Backpressure is what stops a range from outrunning the queue when it doesn't. At twice range_max_bytes, the KV layer stops letting writes into a range with a split pending:

range_max_bytes      536,870,912   (512 MiB)
backpressure at    1,073,741,824
r725               1,073,844,534

It doesn't reject them outright. It holds the batch, waiting for the range to come back under the threshold, and the write fails only when the request runs out of time. That distinction is why the failure surfaced to us as persist timeouts rather than as a clean error, and it's the whole design assumption: the split you're waiting on is going to happen.

100 KiB over the line. And the split was never going to happen.

I sampled the version count twice, 25 seconds apart, to be sure writes were genuinely frozen rather than merely slow. 6,766 both times. While wedged, the range produced about 390 log lines every 15 minutes, continuously, because failed persists retried with no backoff.

Which leaves GC as the only thing that could end it, and GC runs on a queue too, with the same asynchronous, scored shape as the split queue. Each replica tracks a statistic called gc_bytes_age, the volume of collectable garbage multiplied by how long it's been collectable, and the queue prioritises by that rather than by raw size. When it gets to a range it computes a threshold of now - gc.ttlseconds, drops every version older than that, and advances the range's own GC threshold so that later reads below it are refused rather than served wrong.

Two things follow from that shape. GC can never reach anything inside the TTL window, so during a wedge a range can only shed what has already aged past it. And because the queue is scored and periodic rather than continuous, recovery begins when the queue reaches the range, not when the first version becomes collectable.

The second of those is the part I can't fully account for. Getting back under the line needed almost nothing, since the range was sitting 100 KiB over a 1024 MiB threshold, and yet writes stayed refused for 75 to 90 minutes every time. Aging alone doesn't explain a gap that size, so what dominates it has to be when the GC queue got round to the range. I never pinned that down more precisely, and the incident was resolved before it mattered enough to.

The cycle itself is legible enough without it. Roughly two hours of rewriting to rebuild a gigabyte, then the wedge, then GC clears it and it starts over. Five times across two days, always the same row.

And gc.ttlseconds is a floor on retention rather than a target. Retained bytes are write rate times version size times that window, and nothing in the system pushes back on the product.

Raft never failed in any of this. No quorum loss, no elections, nothing. But it sits underneath every part of it, and it's the reason the size limit exists at all.

A range isn't a storage bucket. It's a Raft group: three replicas by default, one of them holding the lease. Every write to that row was a Raft proposal, which the leaseholder proposed, a quorum accepted, and each replica then applied to its own copy. So those 6,766 versions weren't 6,766 disk writes. They were 6,766 rounds of distributed consensus, each shipping a full 155 KiB snapshot across the network, and the gigabyte existed three times over, once per replica.

Size matters to Raft in two more places. A replica that falls far enough behind can't be caught up from the log, because the leader has already truncated the entries it would need, so it gets sent a Raft snapshot instead: the entire range, over the network. Same story when a node is decommissioned and its replicas are rebuilt elsewhere. A 1 GiB range is a 1 GiB transfer, and until it lands that replica isn't contributing to quorum. Keeping ranges small is what keeps rebalancing and recovery cheap enough to happen automatically.

The split is a Raft operation too, committing a new range descriptor through this same group. None of that got as far as running. It failed at the first step, choosing the key to cut at.

So the rule that trapped us exists to keep Raft groups small enough to move around, and we'd built one that could never be divided.

One note if you're coming from Postgres. This isn't a page split. The storage engine is Pebble, an LSM tree, so there are no pages and no fillfactor to tune. Splitting a range is a decision about distribution across a sorted keyspace, not about storage layout. The page-split intuition is the obvious one to reach for and it doesn't transfer.

Where the writes came from

Two thousand consecutive pushes for the wedged document:

payload size        min 130,009   median 130,009   max 130,009
distinct clients    1
inter-write gap     p50 2.24s

Not one byte of variance across any of them. One client, sending the whole document every 2.2 seconds, unchanged. The row's lifetime write counter was at 41,201, which at that cadence is about 25 hours of continuous pushing, and lines up with the first wedge the previous afternoon.

Someone left a tab open.

On the client, the checkpoint gate asked "did any command run during this dispatch?" instead of "did the document change?". A layout loop that measures rendered block heights and reports them back kept producing command work, so it kept re-exporting and re-sending the entire document. On the server, nothing compared the incoming bytes against what was already stored, so each one landed as a fresh 155 KiB version of an identical document.

Five ways out, in the order we considered them

1. Raise `range_max_bytes`

First thing suggested, first thing rejected. It moves the ceiling for every range in the table and does nothing about the accumulation. The ceiling here comes from storing one document per row, not from that number being too small.

2. Lower `gc.ttlseconds`

What we actually did, because it needed no deploy:

ALTER TABLE resources CONFIGURE ZONE USING gc.ttlseconds = 600;

Retained bytes are write rate times version size times retention window. We couldn't touch the write rate without shipping code, so we took the window from four hours to ten minutes. That's 155 KiB × 0.47/s × 600s, or about 43 MiB, against 1024 MiB before.

One range went from 728 MiB to 95 MiB in two minutes. Another went from 579 MiB to 168 MiB. gc_bytes_age on the first fell from 7.2e12 to 3.6e10.

That speed deserves a note, because deleting from an LSM frees nothing immediately. Pebble writes deletion markers and the space comes back at compaction, whenever that happens to be. But the size the split queue reads is the MVCC stats, not the disk footprint, and GC updates those the moment it runs. So the range stopped counting as oversized well before it stopped occupying the bytes, which is the only reason a one-line config change unwedged production in minutes.

This settles at a steady state rather than counting down to anything. Versions arrive and expire at the same rate, so the pile reaches a size and stays there.

I checked system.protected_ts_records first, and it's worth being precise about which direction that check runs in. A protected timestamp pins the GC threshold: while one is held, GC cannot collect anything newer than it, which is how a backup or a changefeed keeps its reads valid for as long as it takes to finish. So a record sitting on this table wouldn't have been the thing at risk. It would have defeated the fix. GC would have refused to drop below it, the range would never have drained, and prod would have stayed wedged with a config change applied and nothing to show for it. The table was empty.

What an empty table doesn't tell you is whether anything was reading historically without taking a protected timestamp. Those are the consumers this actually breaks, and after the change an AS OF SYSTEM TIME read further back than ten minutes on that table fails outright.

3. Ship the coalescing persist queue

There was already an open PR for one: 750 ms debounce, 5 second ceiling. A 5 second ceiling caps a continuously-edited document at 2,880 versions per GC window, which moves the wedge threshold to roughly 364 KiB.

The wedged document was 155 KiB on disk, so 364 KiB sounds like room. It isn't much, and there are two thresholds here rather than one. 364 KiB is where a document wedges. 182 KiB is the earlier one, where its range crosses range_max_bytes and starts attempting splits it can't finish, without being backpressured yet. Another worksheet had already reached 188 KB and was still climbing, which puts it past the first line and heading for the second.

So this buys headroom without removing the ceiling. It also doesn't compare content, so identical resends still get written, only less often.

I'm not actually sure it would have prevented this one. All of that assumes the 5 second ceiling binds, but at a 2.2 second arrival rate the 750 ms debounce expires between pushes, so each push probably still flushes on its own and the rate doesn't move at all. I haven't tested it.

4. Hash the snapshot, skip the write when it matches

The real fix, and the one that shipped. It takes almost all of these writes to zero no matter what any client does, and it holds for whatever version of the client happens to be running, which matters when your clients are browser tabs you can't force to reload.

5. Chunk the snapshot, or move to an append-only update log

The only option that removes the single-key property instead of buying room underneath it. Also the largest change by a wide margin, and not something you do on a Wednesday afternoon with prod wedging every few hours.

The monitoring

None of our alerts could have caught this, and it wasn't a threshold problem. OOMKilled, MemoryHigh, CpuThrottled, CrashLooping, Down. Every one of them stays quiet on a service that is perfectly healthy and simply not permitted to make progress. We had no signal for that shape of failure.

CockroachDB was already exporting exactly the right counter. queue_split_process_failure goes up on every failed split attempt, and healthy clusters don't fail splits, so rate(queue_split_process_failure[15m]) > 0 is about as clean a signal as you get. The node holding the lease for r725 was at 2,114. Nothing was scraping it.

The rule I then wrote on top of it failed twice over. Prometheus evaluated it and it went active, but Alertmanager dropped the notification on the floor: the default receiver is null behind an allow-list regex on alert names, and a new name that isn't in that regex gets discarded without a trace. So it fired and nobody heard it. Any new rule here needs an entry in that regex or its own route, which is not a thing you find out by testing the expression.

Once it was routed, it stayed active for about 25 minutes after the condition cleared. Not because counters only go up, which was my first guess: rate() does return to zero. It's that the 15 minute lookback keeps the rate positive until the window slides past the last failed split, and Alertmanager's resolve delay adds the rest. Shorten the window and you trade that against missing sparse failures.

The one I'd hand to someone else is the dedup counter we added afterwards, which counts writes skipped because the content hash matched what was stored. It started out near the entire write volume. That means a fall toward zero is the direction that should worry you, because it says writes have gone back to being genuinely distinct and the ceiling is live again. Every other metric on that dashboard alarms upward.

Three CockroachDB quirks worth knowing before you go looking, none of them well documented: crdb_internal.tables keys on table_id, not id. SHOW ZONE CONFIGURATION wants the real database name, and these tables live in defaultdb. And round() errors out if you mix decimal and float8, so cast first.

Moving 20,000+ customers to a new Stripe account without anyone noticing

Vatsal Patel — Fri, 10 Jul 2026 13:16:46 +0000

None of our 20,000+ US customers know this happened: last month, we moved them, their payment methods, subscriptions, credits, coupons, and promotion codes off our Australian Stripe account and onto a new US one. Nobody got double-charged. Nobody lost access. No emails went out, because from their side, nothing changed.

This is what it took to make that boring outcome happen.

Why we did it
The company operates in Australia and the US, but until recently, both regions ran on a single AU Stripe account that took USD payments from US customers and settled them out as AUD. That setup cost us roughly 2% per US transaction in international card surcharges and currency conversion fees on top of the domestic AU rate, plus FX losses on settlement. It also made tax season painful and obscured per-region revenue reporting. The brief from the CEO was simple: clean financials, lower fees, no customer impact. The shape of the solution - a separate US Stripe account with all US customers migrated over - was mine to figure out.

Scope
Two products with inverted shapes.

The tutoring product has 2500+ US customers, 1100+ actively paying, billed per lesson. Many of them carrying credits, coupons, and promo codes.

The schools product has 20,000+ US customers, but only a few dozen with active subscriptions and a few hundred with payment methods on file - most schools sit on the platform without a paid plan. So the smaller user base carried the bulk of the active billing risk, which is why I scoped the tutoring migration first. The schools product had a planned refactor and deployment coming up, so the Stripe migration was set to be carried out shortly after the launch.

What Stripe gives you, and what it doesn't
Stripe's self-serve PAN copy tool is the only way to move card data between accounts in a PCI-compliant way. This is what it actually covers:

Copies customer objects, preserving the customer ID
Copies attached payment methods (with new payment method IDs)
Hands you a CSV mapping old payment method IDs to new ones
That's it. It does not move credits. It does not move coupons or promo codes. It does not move subscriptions, invoices, or any metadata you care about beyond the customer record itself.

Everything else was scripts I wrote, all in Go. Each had a dry-run mode, printing what they would change and against which records, and only mutated state when explicitly invoked with a write flag. For migrations like this where the cost of a wrong run is high and the cost of an extra dry run is zero, that's the cheapest insurance you can buy.

The plan, and the timeline
From "we're doing this" to "ready to deploy" was 5 days of build. The longest single block of elapsed time was waiting a month for the new US Stripe business account to clear verification, which had nothing to do with us. The execution itself ran 3 hours, between midnight and 3 AM US time, with the payment button disabled on the US web app for the window - no one was trying to press it anyway.

The schools product would follow largely the same script, the difference being a much larger user base but none with credits.

The engineering work fell into a few buckets:

Customer + payment method migration. Run the PAN copy tool, ingest the CSV, run a script that updates the payment method IDs in our database against the preserved customer IDs.

Credit migration. A script that reads each customer's cash balance from the old account and recreates it on the new one. This is the part that bit us - more on that below.

Coupons and promo codes. We issue per-customer coupons with promo codes attached. Sales had no uniform convention for where they put usage limits - sometimes on the coupon, sometimes on the promo code, sometimes split across both. The migration script had to read both sides, reconcile what was actually still valid, and recreate each coupon-and-code pair on the new account with the correct remaining usages and the correct customer restriction. The "correct remaining usages" calculation was the trickiest single piece of logic in the migration, because the source of truth varied per customer.

In-flight invoices. Open and failed invoices on the AU account were the thorny data to move. These weren't static records - Stripe retries failed payments automatically over several days, and customers pay open invoices on their own time when they see the email. Voiding and recreating them at cutover would have meant cancelling invoices that were about to settle on their own, sending customers a fresh invoice with a new number, and creating support churn for billing relationships that didn't need any intervention. So we left them on the old account for a 48-hour settlement window, let natural retries and customer payments clear what they were going to clear, and then ran a script at midnight US two days after the main cutover that voided whatever was still open and recreated it on the new account. The set that needed manual intervention ended up being much smaller than the set we started with.

Webhook routing. This is where the architecture got interesting. Our backend runs as two regional clusters, sharded by region. With one Stripe account, we could route webhooks directly to the AU cluster and let it forward what it needed. With two Stripe accounts, neither cluster owns "the truth" anymore.

We solved it with a single Cloud Function as the webhook entry point for both Stripe accounts. The function verifies the signature against both accounts' signing secrets, looks up the customer's region in our metadata, and forwards the webhook to the correct cluster. We already have direct REST endpoints on each backend that can receive webhooks natively, and the longer-term plan is to point each Stripe account directly at the cluster that owns its customers. I deliberately chose not to do that yet. During the settlement tail, AU-account webhooks will keep firing about US customers as their old AU-account invoices clear - and those events need to land in the US cluster, not the AU one, because that's where the customer now lives. The cloud function is the one place that knows how to route by who the customer is rather than which Stripe account sent the webhook. Direct REST endpoints assume each cluster owns its inbound events; that assumption is broken until the AU account stops emitting events about migrated customers. The function comes out once all in-flight invoices on the old account have closed.

This is the kind of decision that's easy to get wrong by reflex. The "right" architecture is direct webhooks per region. The right next step was the cloud function, because the cleaner architecture would have caused real customer-facing breakage during the tail of the migration. Worth being explicit about.

Archiving. Stripe doesn't let you delete or archive customers on the old account. So I wrote a script that suffixed every migrated customer's name with (ARCHIVED). Sales and finance still see them in the dashboard, but nobody on either team is going to accidentally start operating on a (ARCHIVED) record - a small piece of code that did more for the migration than most of the actual logic.

The Execution
A few hours before the cutover window, I ran every script in dry-run mode against production data one more time, verified the outputs, and eyeballed the planned changes one more time. Nothing surprising came back, which is what you want from a final pre-cutover rehearsal - boring is the goal.

Midnight start. Disable the payment button on the US web app. Run the PAN copy tool.

Stripe's documentation says the PAN copy tool can take up to 3 days. Other writeups I'd read suggested up to 2 hours for under 2,000 customers was normal. We had budgeted 3 hours and had contingency plans for it taking longer.

It finished in 10 minutes.

That was the single biggest unknown in the whole plan, and it evaporated immediately. The remaining 2 hours and 50 minutes were spent running the metadata scripts, deploying the cloud function and the backend changes, flipping the live keys, and QA-ing the result.

We did the schools migration about 3 weeks later, following the same script. Running every script in dry-run mode against production data, and then running them for real once everything looked good. This time, however, we had 20,000+ customers to migrate over the self-copy PAN tool. We had budgeted 3 hours for it based on the estimate from the previous run, accounting for delays. It finished in just under 1.5 hours.

We ran a script that gave us the subscription details, such as subscription status, trial status, renewal date, coupons to apply, and everything else that we had to get right. We migrated the subscriptions to the US account, setting them up on trial until their next renewal date to prevent charging them again - this was important since a majority of subscribers are on the annual plan, some of whom had just activated their subscription.

What went wrong
One thing, during the tutoring execution, and it was instructive.

An account manager flagged it. She noticed that recently applied credits were missing on one of her accounts on the new US Stripe account. Stripe lets a customer hold cash balances in multiple currencies, and we read them through the customer API. For five customers, that API returned the AUD balance instead of the USD balance, even though those customers had real USD credit, and our migration script trusted the API and copied across a zero. Our verification script used the same API, so the bug went unnoticed at cutover and only surfaced because a human knew what the right number should have been. Why the API returned the zero AUD balance instead of the non-zero USD one, we still aren't sure.

The fix was small: pull balances using the cash balance API against both AUD and USD currency balances explicitly, then run a correction script on the affected customers. Five customers were affected, all of whom had their correct credit restored before any of them touched the product.

I'd call this a near-miss rather than an incident. No one was billed incorrectly, and no support ticket was opened. But it's the part of the migration I'd most like back. The lesson is specific: when an API returns a structured value, validating that the returned value matches a separate source of truth is worth the extra script run. We had the validation; we just ran it after the migration instead of as part of it.

No formal post-mortem. The fix was scoped and applied within a couple of hours, well before the customers even noticed the problem.

Verification
Two scripts ran post-cutover:

Customer + payment method audit. For every migrated customer, confirm the customer exists on the new account, has the expected number of payment methods, and that those payment methods are attached and the IDs matched with our database.
Credit audit. For every customer with a non-zero balance on the old account, confirm the same balance exists on the new account.
Subscription verification is its own piece of work. Scripts ensured that all of the active subscriptions were copied correctly, none of the users had been double-charged, and all of the original subscriptions had been cancelled on the AU account.

Team
I led the migration end-to-end: the technical plan, the research into the PAN copy tool's behavior and limits, the backend code changes, all of the scripts, the cutover sequencing, and the verification. Another senior engineer paired on the frontend changes and the cloud function - particularly the cluster-forwarding logic, which they owned. Finance and sales were kept in the loop on the timeline, but didn't have execution responsibilities during the window.

What I'd do differently
Two things, looking back. I'd validate balances against a second source as part of the migration script rather than after it - the cash balance API quirk was the only real surprise across both migrations, and it was the one thing my plan didn't have a pre-cutover check for. I'd write a short design doc on the coupon and promo-code logic before coding it, since the inconsistency in how sales had set usage limits was discovered pretty late into the building phase, so we settled for unit tests instead. The common thread is that both are pre-cutover discipline I traded for during-cutover speed - small upfront investments that would have made the cutover itself even quieter.

Takeaways
The migration was, by the metric that mattered, invisible. 22,500+ customers across two products, a meaningful book of credits and active billing relationships, a webhook architecture change, and a few dozen active subscriptions. Zero customer-visible incidents, five customers requiring a same-day correction that no one noticed.

If I had to compress the lessons:

The shape of vendor tooling determines the shape of your work. Stripe's PAN copy tool moves cards and customer IDs; everything else - credits, coupons, promo codes, the actual business state - is on you. Knowing exactly where the vendor's responsibility ends is the first task of a migration like this, not the last.
Choose the boring intermediate architecture when the in-flight state spans the cutover. A cloud function fan-out is uglier than direct webhooks per region. It's also the only thing that doesn't break the tail of customers whose invoices straddle the migration.
Build for the people who'll touch the data after you. The (ARCHIVED) prefix mattered more than most of the actual code. A migration ends when finance and sales can use the new system safely, not when the scripts finish running.