DEV Community: Vatsal Patel

Canada is about to break your scheduler

Vatsal Patel — Sun, 20 Sep 2026 12:59:47 +0000

At 2:00 AM on 1 November 2026, DST ends for Canada. However, DST will not end in British Columbia, Alberta, the Northwest Territories, and Manitoba starting from this shift.

Each of the above announced at separate times that they will stop moving their clocks.

Where	Zone	Decided	Days before 1 November	First tz release with it
British Columbia	`America/Vancouver`	announced 2 March, in effect 9 March	244	2026b, 22 April
Alberta	`America/Edmonton`	in effect 18 June	136	2026c, 8 July
Northwest Territories	`America/Inuvik`	in effect 21 August	72	2026d, 11 September
Manitoba	`America/Winnipeg`	announced 17 September	45	none

The politicians call it permanent daylight time. As usual, this means a whole big mess for all computer systems. In the tz database, this is a new standard time: Vancouver becomes MST, Edmonton and Inuvik become CST, and Manitoba stays on UTC-5 all year.

The tz project asks the government for a year's notice, as the updates can take a long time to propagate. Manitoba gave six weeks. As I write this, 3 of the 4 are included in the newest release 2026d. The project has said it will need a new release that will reflect all of them in the near future.

Now, anyone who has worked with timezone and date systems can attest to this, but dealing with timezones and DST is always a nightmare. Changes like this always break things. I assumed time changes like these were rare, so I went through the history to check how uncommon it actually is. The easiest way to check was to go through the release list of tz. Turns out, it is not as rare as I thought. There have been 31 releases since 2020a, and 26 of them changed future timestamps. Palestine is responsible for seven of these, Morocco is four, Fiji and Greenland at three each, and then a long list of places that show up only once. Some of them almost didn't give a warning, like Mexico's end of DST observation in 2022f, two days before it was to take effect. Lebanon decided to delay the DST start in March 2023 by a month, and then called it off 5 days later. I checked the tz releases against the official announcements to see if there was a delay, but tz released the same day as the official announcement most of the time.

Why I looked

I don't remember where I first saw the news, but I read something along the lines of "Parts of Canada to not observe DST" so I clicked on it out of curiosity. I run a scheduler and crons in Go so any DST or time-related changes naturally mean something will break. We naturally store the timestamps in UTC and each customer gets a time zone field. I checked, and we don't have any customers in the affected regions so this doesn't quite affect me this time, but the same could not be said for everyone else who works on schedulers and crons across the globe. I was curious as to how this would be fixed, was it a simple case of "just update your tzdata"? But turns out it's a lot messier than I thought.

Where tzdata comes from

Go uses the tzdata built into the OS, which is something different for every system. Python's zoneinfo does the same, and so does .NET on Linux and Ruby's tzinfo, unless you install the data gem. Anything in C gets the same files through glibc.

On my Mac, /usr/share/zoneinfo is symlinked to /var/db/timezone/zoneinfo and Apple keeps it updated, cat /usr/share/zoneinfo/+VERSION says 2026c on 20th September. On a normal Linux box, tzdata comes from the distro's tzdata package and it updates anytime someone runs a package update.

The second way is the runtime carrying its own copy and ignoring the OS. Node gets its timezones from ICU, which is compiled into the node binary, and it does not care about /usr/share/zoneinfo at all. This means installing a newer Node runtime rather than updating your base image or OS tzdata.

Containers are where this gets messy. The image has whatever /usr/share/zoneinfo its base image had on the day it was built. Some slimmer versions like Alpine don't ship any tzdata. I know this since I had built cron images that errored out with unknown time zone America/New_York. I had to then figure out what went wrong and fix it by importing _ "time/tzdata".

Databases do their own thing. Postgres ships its own copy of the tz database in its source tree. What tz database it uses depends on how it was built. The configure flag, --with-system-tzdata=/usr/share/zoneinfo makes it read the OS files instead. Postgres built without the above flag will update the tz ruleset from minor releases. The minor release from 13 August follows 2026c, which has British Columbia and Alberta. The next minor release will be on 12 November, which is after the DST change.

MySQL is a weird one. Named zones like America/Vancouver don't exist as files. They are stored in a table in the mysql schema. The table is populated by running mysql_tzinfo_to_sql /usr/share/zoneinfo. It's a copy of whatever the OS had at the time of running. The official Docker Image does the loading for you in the entrypoint, but only when initialising a new database.

The shift

The whole problem is one question: what UTC time is 9 AM in Vancouver on 1 December?

This fits in two lines of Go.

loc, _ := time.LoadLocation("America/Vancouver")
dec := time.Date(2026, 12, 1, 9, 0, 0, 0, loc)

The answer depends on which tzdata LoadLocation found. I wrapped those two lines in a small program that prints a 9 AM job for all four zones, on 15 October and on 1 December, and ran it twice. First against the system zoneinfo, which on this Mac is 2026c:

a 09:00 local job, before and after the change
America/Vancouver  15 Oct -> 16:00Z PDT  -7    1 Dec -> 16:00Z MST  -7
America/Edmonton   15 Oct -> 15:00Z MDT  -6    1 Dec -> 15:00Z CST  -6
America/Inuvik     15 Oct -> 15:00Z MDT  -6    1 Dec -> 16:00Z MST  -7
America/Winnipeg   15 Oct -> 14:00Z CDT  -5    1 Dec -> 15:00Z CST  -6

Then against the copy that go1.25.4 ships with, which is 2025b, by running it with ZONEINFO=$(go env GOROOT)/lib/time/zoneinfo.zip:

a 09:00 local job, before and after the change
America/Vancouver  15 Oct -> 16:00Z PDT  -7    1 Dec -> 17:00Z PST  -8
America/Edmonton   15 Oct -> 15:00Z MDT  -6    1 Dec -> 16:00Z MST  -7
America/Inuvik     15 Oct -> 15:00Z MDT  -6    1 Dec -> 16:00Z MST  -7
America/Winnipeg   15 Oct -> 14:00Z CDT  -5    1 Dec -> 15:00Z CST  -6

Look at Vancouver on 1 December. It's 16:00Z on the new data and 17:00Z on the old, so with old data the 9 AM job goes off when the clock in Vancouver says 10. Edmonton is the same. Inuvik and Winnipeg come out identical in both runs, and both are wrong, because 2026c doesn't know about the Northwest Territories, and no release includes Manitoba yet.

The 15 October column is the same in both. The difference comes after 1 November, which is the date when the new rules start to apply.

A newer Go doesn't fix it

My first guess was that a newer Go version would fix it, since Go embeds its own tz database when you import time/tzdata but that wouldn't work. Turns out, the embedded copy is simply a fallback. This is zoneinfo_unix.go:

var platformZoneSources = []string{
    "/usr/share/zoneinfo/",
    "/usr/share/lib/zoneinfo/",
    "/usr/lib/locale/TZ/",
    "/etc/zoneinfo",
}

$ZONEINFO is checked first, then those four directories, then the embedded database, then $GOROOT/lib/time/zoneinfo.zip. So if the image has a /usr/share/zoneinfo, that's what you get, however old it is and however new the binary is. An image keeps the tzdata it was built with. The embedded copy only matters in something like FROM scratch where there are no system files at all.

And the embedded copy is old anyway. go1.25.4 embeds 2025b. Releases don't necessarily pick up new tzdata. The newest release as I write this is go1.27.1, from 28 August, and it embeds 2026c: British Columbia and Alberta, no Northwest Territories, no Manitoba. go1.26.8 embeds 2025c, which has none of the four. The annoying part is that Go won't tell you what it loaded, there's no version string at runtime. However, process.versions.tz exists in Node and tzdata.IANA_VERSION exists in Python.

How our scheduler works

We store the first occurrence of a recurring event as a UTC timestamp. We find this to be easier than converting it to cron or any other format. Calculating the next occurrence is a simple add 7 days to it. We first check whether a timezone observes DST or not, by checking IsDST() on 1st January and 1st July. If DST is observed, we take the first occurrence UTC timestamp and the current occurrence timestamp, convert them to user's timezone and calculate the UTC offset difference. We subtract the difference from the current one. Since everything is measured against the first occurrence, it doesn't matter whether the first occurrence fell in DST or in standard time.

func CalculateDSTOffset(timezone *time.Location, firstOccurrence, currentOccurrence time.Time) (dstOffset time.Duration) {
    _, firstOccurrenceZone := firstOccurrence.In(timezone).Zone()
    _, currentOccurrenceZone := currentOccurrence.In(timezone).Zone()
    if firstOccurrenceZone != currentOccurrenceZone {
        return time.Duration(currentOccurrenceZone-firstOccurrenceZone) * time.Second
    }
    return 0
}

dstOffset := dst.CalculateDSTOffset(userTimezone, firstOccurrence, currentOccurrence)
currentOccurrence = currentOccurrence.Add(-dstOffset)

Previously, we had an engineer running a script that went through all of our recurrences and shifted them for DST. This was very annoying and easy to get wrong. I have heard horror stories of a 36+ hour workday when things went wrong. This was before I joined the company. I am very proud of this one, automating away the entire DST cycle.

The fix

We don't have any customers in any of the four regions, but here is how I would fix it if we did.

First thing to do would be to add unit tests to the above implementation, and unsurprisingly, the outcome depends on the tz release being used.

scheduler sees 2026c
  America/Vancouver  created 12 Jan, stored 17:00Z   15 Oct -> 09:00     1 Dec -> 09:00
  America/Vancouver  created 6 Jul,  stored 16:00Z   15 Oct -> 09:00     1 Dec -> 09:00

scheduler sees 2025b
  America/Vancouver  created 12 Jan, stored 17:00Z   15 Oct -> 09:00     1 Dec -> 10:00 !
  America/Vancouver  created 6 Jul,  stored 16:00Z   15 Oct -> 09:00     1 Dec -> 10:00 !

On the old tz, we are an hour late from November 1st. However, with the above solution, the code itself works out correctly. We check IsDST() before applying the offset but my implementation has the check based on 2025 dates. The check passes since Vancouver still had DST in 2025. I ran zdump -v -c 2026,2028 America/Vancouver | grep "Nov 1" to see the transition and sure enough, the gmtoff stays the same. This means CalculateDSTOffset will return 0 and no adjustment will be applied.

America/Vancouver  Sun Nov  1 08:59:59 2026 UT = Sun Nov  1 01:59:59 2026 PDT isdst=1 gmtoff=-25200
America/Vancouver  Sun Nov  1 09:00:00 2026 UT = Sun Nov  1 02:00:00 2026 MST isdst=0 gmtoff=-25200

The fix for us is rebuilding all of our images with the latest tz rules, where head -1 /usr/share/zoneinfo/tzdata.zi says 2026d. Then do it again, because Manitoba isn't in any release yet and whatever gets rebuilt today is still wrong for Winnipeg.

Some of you using Node or MySQL may not have it so easy.

Closing

I really wish DST didn't exist, and it seems my wish is coming true. At least with 4 Canadian time zones. Here is to hoping the rest of the world will follow, but ideally with advance notice of at least a year.

Building a multi-region routing system with Cloudflare Workers

Vatsal Patel — Wed, 02 Sep 2026 15:30:39 +0000

We serve customers primarily in Australia, but we are now expanding to the USA. The timeline for launch is less than 2 months. This is now a race against time to design a multi-region routing system that fits all of our needs. Here is the story.

Background

Almost all of our customers were based in Oceania. We run our Kubernetes Cluster on GCP in Australia. Go microservices, federated GraphQL, gRPC services. 2 products - Tutoring and Schools. All designed for Australia.

Then we expanded to the USA, which meant a new Kubernetes Cluster in US Central. The latency for serving US customers from Australia is an extra 200ms-300ms depending on network conditions - unacceptable. This would mean sharding the data by region, or does it? There are definitely ways to keep a unified dataset even across regions - though we did not need to do so. More on this later.

What are the requirements

If the only requirements were "Americans get served from America", we wouldn't be here discussing this, would we?

Logged in users are served from their own region, wherever they happen to be in the world.
Logged out users are routed geographically, as we have no other information to infer their actual region.
Account Managers and Admins should be able to access both regions from one button, with a single account.
Teaching materials opened via links from the Schools product must be shareable across both regions.

Geography takes care of the logged out user, but nothing else. Using geography for a logged in user can be actively wrong. They might be travelling or simply using a VPN.

Then comes the Admin; we have a lot of admin operations regarding curricula, which will be entirely separate for both clusters. Account Managers need to be able to see and modify information on both clusters. One admin should be able to access both clusters with a single account. We considered showing data of both clusters on one screen, but ruled it out as it may become too ambiguous or confusing, not worth the technical hassle of getting pagination to work cross-region. Next best option: switch the cluster with the flip of a switch.

Sharing is the hardest case to cover. We may have a teacher create some teaching material and share it on social media, where the person clicking on it may be from the other region. While curricula are not quite the same, a lot of the topics and subtopics do tend to be very similar. We observed this during the curriculum integration and decided this was a key requirement we needed to support, annoying as it may be.

To shard, or not to shard

Sharding would be the easier and obvious solution here, but it presents some problems. How do you know which region a user belongs to if your Auth service is sharded? Both regions will simply reject or infinitely redirect the requests to each other in case of a malformed or non-existent user ID. We need some central service or registry that can check for the existence of a user ID and tell us which cluster they belong to. This can be sharded or unsharded, but not sharding is simpler. We use Firebase for Authentication, so we decided not to shard our Auth service.

Not sharding is now difficult; you either end up with a cross-region database, where you either give up consistency or accept a high latency. For our other databases, we decided sharding would be the best approach. Running cross-region Neo4j or CRDB was not a particularly good idea. I have read enough on consensus to know it is possible, but simplicity is better than overfitting a solution to our problem here.

Our options

I brought a long list of options to the first meeting, most of them already crossed out.

A service mesh across both clusters This is the one our CTO wanted to consider the most, Istio or Linkerd. Connect both clusters, but that doesn't really solve the routing problem; the US requests still end up in Australia if we keep a single database. If we shard the databases, we now have a service discovery problem where we have to figure out which database instance to use for a given user.

Connect the two clusters Using Submariner. This also adds the same service discovery problem as above. P.S. I got really annoyed by seeing the Rolex Submariner whenever I googled "submariner".

Multi-region databases CRDB has real multi-region support using REGIONAL BY ROW tables, locality-aware placements, and follower reads. This solves our problem, but we weren't sure what the latency would be like when the 2 regions are on opposite sides of the planet. We also run Neo4j which doesn't have first class multi-regions support so we ended up benching this idea.

A single service that works out user's region and redirects them We could have written a new service for this, using Firebase Auth as its database, but then we have to accommodate the admin requests, admin overrides, the resource sharing and other use cases. We ended up doing something like this, but just in a different way.

Cloudflare Load Balancer and Geo Steering We already use Cloudflare for our Domains, so this looks like a natural solution. It wouldn't be the simplest thing to configure given all of our requirements but it certainly would be possible. This option was amongst the finalists, but the next option won as it worked better for us.

Cloudflare Workers I ended up recommending this one, as it is a natural addition to our networking stack and gives us a lot of control over how we route traffic. We would use Workers as an API gateway of sorts, essentially. This also makes it really easy to express our requirements in code and extend them easily in the future. Lots of libraries are available as well, so we could interact with Firebase Auth and any other common service with an npm package, or write our own code if something doesn't already have a supported package. Additionally, workers are really cheap. 10 million requests included per month + $0.30 per million additional requests. CPU time is fairly generous and quite sufficient since our API gateway is meant to be a very small hop.

Putting it all together

With our chosen solution, we got to work putting it all together.

We still had some decisions to make, what would take precedence when routing requests to a cluster, so we built a hierarchy.

First of all is the subdomains; these are directly connected to the cluster's IP as DNS A records and don't go through the workers at all, very convenient for internal direct routing uses. These are actually not part of the hierarchy, but are completely outside. Workers can use these for DNS override.

Next up is URL path routes. Mainly used for admin overrides and for indicating the country when sharing resources and other social media links. The worker strips the region bits from the URL path before forwarding it to the correct cluster, so no service had to be changed or reconfigured to support this. The admin site gets a selector at the top, which defaults to AU and uses the URL path. Switch it to US and it will use the US scheme, making sure the requests end up in the correct region without switching accounts.

Up next is cookies; we started saving user's country in a separate __country cookie. Started off with only au and us as the supported values, with the hope that there will be tons more. We also store the country on our JWT token as a backup in case the country cookie isn't included.

Geographical location is last. It is derived from request.cf.continent. We tried geographical distance first, but some of our overseas employees landed on AU and others on US, all of them sitting in the same country. So we settled on the continent

function getOriginByGeolocation(request: Request, env: Env): string {
    switch (request.cf?.continent ?? 'OC') {
        case 'EU':
        case 'NA':
        case 'SA':
            return env.US_HOST;
        case 'AN':
        case 'AS':
        case 'AF':
        case 'OC':
            return env.AU_HOST;
        default:
            return env.AU_HOST;
    }
}

Why is Antarctica listed there? Because why not.

Proxying instead of redirecting

We have figured out the cluster, but the job is not yet done. The obvious thing is to return a 302 and let the browser take care of things from there. We can't do that. We had built host matching in our cluster with IngressRoutes so a 302 to our AU host would not match the actual endpoint we want it to hit. Luckily, Cloudflare has a resolveOverride field, which directs the request to an alternate origin server by overriding the standard DNS lookup, while keeping the original Host header intact. This was the primary reason workers worked so well for us. We could set all of this up without a single change to our existing services or Kubernetes networking layer.

This was about 15 lines:

let newReq = new Request(request);
newReq.headers.delete(USER_ID_HEADER);

const jwt = await verifyJWT(request, env);
if (jwt !== null) {
    newReq.headers.set(USER_ID_HEADER, jwt.uid);
}

const { origin, url, cookies } = await getCorrectCountryOrigin(newReq, jwt, env);
const reqInfo = new Request(url, newReq);

let res = await fetch(reqInfo, {
    cf: { resolveOverride: origin },
});

res = new Response(res.body, res);
cookies.forEach((cookie) => res.headers.append('Set-Cookie', cookie));
return res;

There are two copies being made; both are intentional. An incoming Request in the Workers runtime has immutable headers, so a copy is required. new Request(request) and new Response(res.body, res) allow modifying the headers. A second copy is required because we may have changed the URL to remove the region prefix from the path. resolveOverride ensures we send it to the correct cluster. Proxying instead of redirecting also saves an entire round trip on the first request, and also prevents the client from being stuck on a single cluster. If they click on 2 shared links, one for AU and one for US. The earlier one will work but the latter one will break. Proxying prevents this.

We later benchmarked Workers with timing headers and logs, and found these to be extremely performant. Roughly 2-10ms overhead depending on which routing case ends up being true, including a full JWT verify. Much lower than what any of the other solutions likely would have ended up being. We really loved the performance and reliability. Cloudflare hasn't had any major outages affecting us in the 2 years, nor have there been any weird bugs. We have of course pushed some bugs every now and then, but we can only blame ourselves for that.

Where it is now

We had 2 months to launch in the US. This worker setup took about 2 weeks. From requirements to having it running on dev and proxying requests. Setting up the new Kubernetes cluster in the USA, loading up the curricula, adjusting our Firebase and auth services to support regions, all of these took up the rest of the 2 month window. We made it just in time.

That was roughly 2 years ago. The hierarchy hasn't changed since but we have built a lot of stuff around it. More signals above the cookie, a second domain, and the worker becoming the authentication layer for some of our use cases.

Why CockroachDB refused writes to a healthy 155 KiB row

Vatsal Patel — Tue, 11 Aug 2026 16:10:53 +0000

A worksheet in prod stopped saving and caused the whole CockroachDB cluster to stop accepting writes.

The pod was healthy: 404 MiB of a 2 GiB limit, 655m of CPU out of 1500m, zero restarts, zero alerts. Everything looked perfectly healthy and normal, yet the database refused writes. 3 active queries across the whole cluster, 12% CPU, all three nodes up and running. No OOM, nothing had crashed but the services still couldn't write.

The document was 155 KiB. Its range in CockroachDB was 1GiB. How did this happen? That is the interesting part.

The software

The software is a collaborative editor. Teachers build worksheets, lesson plans, whiteboards, and teaching documents. A few people can be in the same document at the same time. Every document is a CRDT built on Loro. The browser keeps a replica, applies your edits locally, then ships them over WebSocket to a sync server. The server keeps the copy in memory, merges all the incoming copies, and writes the result into CockroachDB v25.

The important part is how the result is written. To persist a document, it exports the entire Loro doc as a snapshot and saves it to a BYTEA column, on a single row. Normally CRDT are stored as an append-only log of updates. We wanted a simpler structure, semantic search, and fast fetch so we decided to store the whole doc in a single row instead of an append-only log.

A wild goose chase

I got paged that the app isn't working. I hopped on even though it was 9 PM and started investigating.

First clue, object storage is down. The logs were full of it, hundreds of lines of thumbnail and image fetches timing out against storage.googleapis.com. That can't be right, Google storage surely wouldn't be down. I checked the status page and it was up. I thought maybe its a permission or CORS thing, so I ran a curl from a pod on the same node as the server, pulling one of the URLs that were timing out. HTTP 200, 1.1MB downloaded. So Google was in fact not down, and there were no network or permission issues either, meaning the issue was our clients. But that wouldn't block writes on our database, so I left this aside to investigate the database issue. I don't know why I got sidetracked and lost a bit of time to this.

New contender, I was able to see a few pool timed out while waiting for an open connection logs. Very convincing but that also ended up being wrong. I checked the number of queries and number of open connections, but it was only 3. The containers were CFS throttled, 44 to 48% against the limit but the nodes were running at 85 to 97%. Starving an async runtime of CPU and every timeout starts firing even when things are healthy. The pool wasn't exhausted, the tasks holding the connections weren't getting scheduled to return the connections. This was occurring infrequently, and wasn't the issue I was looking for.

Third time is the charm. I noticed the payload sizes varied largely and considered that as the clients sending incremental deltas, which meant the volume was real edits and there was no bug to find here. However, what I observed were different documents. For the same document, the payload size varied little to none. A CRDT snapshot of a document that keeps changing comes out a slightly different size every time, not such a large variance. The check that settles this was dividing the received bytes by what was stored. Full credits to Claude Code on this one because I hadn't even considered this could be an issue.

document type	writes per resource	payload ÷ snapshot
worksheet	995	0.81
lesson plan	57	0.94
whiteboard	5.9	0.87
text document	18	0.82

All pretty close to 1.0 i.e. every type was sending as many bytes as the entire saved document. This wasn't a delta. Worksheets weren't doing anything different from the other document types, they were just doing it 169 times more than the whiteboards.

The actual problem

It was in the logs the whole time, but buried under everything else:

split failed while applying backpressure to Put [/Table/111/60/"..."/0]
on range r725: could not find valid split key

Four conditions that make this happen.

CockroachDB uses MVCC, so a write never overwrites anything. Every write stores a fresh copy of the row under the same key at a new timestamp. The old copies stay exactly where they are until they are eventually garbage collected. The key on disk is the row plus the timestamp, so the same row written twice is two keys sitting next to each other on the same row. This is what lets a transaction read a consistent snapshot without locking. This is what allows AS OF SYSTEM TIME, follower reads, and incremental backups to function. Old versions can't be dropped immediately, in case they need to be read by something. They are later picked up by the GC queue once they are older than gc.ttlseconds, which was configured to 4 hours on our cluster. This means a row takes up its size multiplied by the number of times it was written in the GC window.

The whole row is one key CockroachDB stores a row as one key per column family, this table never defined any beyond the default, so the whole thing sits in one. One Loro doc, one key, no matter the snapshot size.

A split has to cut between two keys Ranges are kept under range_max_bytes by splitting. The split picks a key and cuts there, everything below goes to one range, everything above to the other. If every byte in the range is one key and the copies differ only by timestamps, there is no split boundary. So all the versions live together and the range keeps growing.

The client kept pushing every 2.2 seconds whether anything had changed or not. The new build deployed earlier had introduced this bug, so the same document was being pushed over and over and over as long as the tab was open.

This is what the range actually looked like:

keys       1
versions   6766
val_bytes  1024.02 MiB
live       0.151 MiB

One key. Nearly seven thousand copies of it. A gigabyte of stored versions sitting on top of 155 KiB of actual data.

The math: 0.47 writes/s for a 4 hour ie 14,400 second GC window. counts to 6768 copies, against measured 6766 copies. All of them are almost the same.

Why it can't self-heal

Splitting does not happen on the write path, which was surprising but it made sense. Each one of the 6766 writes was proposed, replicated, committed and visible for the next read. The split runs later on a timer, from a queue. This is based on the MVCC stats. The split isn't a local thing, it is a distributed transaction that updates the meta ranges for the whole cluster and describes where the keys live.

There is always a gap between this range is too big and the range got split, and normally the gap is closed within seconds by the queue. Backpressure is what stops the range from outrunning the queue. At 2x range_max_bytes, the KV layer stops letting writes in:

range_max_bytes      536,870,912   (512 MiB)
backpressure at    1,073,741,824
r725               1,073,844,534

The write isn't rejected immediately, it gets batched. The batch waits for the range to get split apart and come back under the limit again. The write fails when it runs out of time waiting for the split. That's why we saw a lot of persist timeout errors. The assumption is built on the backpressure: the split you are waiting for is going to happen soon. Ours was 100 KiB over the backpressure limit, so the split was not going to happen. This leaves GC as the only thing to save the day. GC also runs on a queue but it can't touch anything inside the TTL window, 4 hours in our case.

Raft never failed

This wasn't a distributed systems failure, even though it might look like one from the outside. No quorum loss, no election, no replication trouble. Raft was healthy the entire time and it is the reason the size limit actually exists.

A range isn't a storage bucket, it is a Raft group, three replicas by default with one holding the lease. Every write to that row is a Raft proposal. It requires quorum acceptance, and each replica gets its own copy. So the 6766 writes were each a round of distributed consensus. Each write shipped the full 155 KiB snapshot across the network and each replica built up the gigabyte backlog.

The size limit exists for recovery. If a replica falls behind and can't catch up from the log, it gets the full snapshot from the leader. The leader truncates the log so it can send the entire range as a snapshot over the network when required. If a node is decommissioned and a new replica takes its place, it recovers from the full snapshot. Transferring more than 1 GiB over the network certainly has its cost and the replica won't help the quorum until it catches up. I was a little surprised that the limit was as large as 1 GiB. Keeping the ranges small is what allows fast recovery and cheap reshuffling of replicas.

Initially, I thought this was perhaps a page split on the storage engine, as I had been writing a B+ tree recently. However, CockroachDB uses Pebble, which is an LSM tree. There are no pages here. Splitting a range is all about distribution across a key space, not storage layout.

A tab nobody closed

Two thousand consecutive pushes for the wedged document:

payload size        min 130,009   median 130,009   max 130,009
distinct clients    1
inter-write gap     p50 2.24s

All of the writes were the exact same. One client was sending the same document over and over every 2.2 seconds.

The simplest explanation was that someone left a tab open. On the client, the network gate was "did any command run during this dispatch?" instead of "did the document change?". A layout loop that measures rendered block heights kept producing command work, so the client kept re-exporting and re-sending the whole document. We didn't have any comparison for whether the bytes actually changed or not, and simply accepted every write from the client.

Five solutions

Raise range_max_bytes The first thing we considered was to raise the range limit and allow the split to occur. This simply removes the ceiling for every range but does not address the 6766 copies we had piled up. We also didn't know what kind of consequences this would have with Raft so we dropped the idea.

Lower gc.ttlseconds This is what we did since it didn't require a hotfix deployment. A hotfix would require code changes, code review, image build, and release. That can come later once the database is working again.

ALTER TABLE resources CONFIGURE ZONE USING gc.ttlseconds = 600;

We couldn't change the write rate without shipping code, and even then the frontend wouldn't really update until they refreshed the tab. We could add the duplicate write guard on the backend but it would take a while to deploy everything and still take a couple of hours for the GC to pick up the old copies. The easiest solution was to lower the GC window so the next run would clean up all of the old copies and free up the range. I checked the write operations immediately after applying and everything was back to normal.

Skip the coalescing persist queue There was already a PR open for this. 750ms debounce, 5-second ceiling. This would reduce the write rate to ~2880 versions per GC window. This would require a document to be at least 364 KiB to cause the same issue. There is some breathing room, but its better to fix it permanently. I asked the author to change the dispatch mechanism to "did anything actually change?" instead.

Deduplicate writes by hashing the snapshot The real fix that we shipped the next day. We hash the snapshot we receive and check against the currently stored value. If the snapshots are the same then the write is skipped. This would fix the issue even when the frontend didn't refresh the tab to pick up the dispatch mechanism fix. We added logs to watch the Dedup counter to make sure it worked correctly, and we saw the log volume gradually decrease over time.

Move to an append-only log This would require a big refactor and we didn't have a requirement for this yet.

We didn't receive an alert

None of our alerts caught this. The thresholds weren’t wrong, OOMKilled, MemoryHigh, CpuThrottled, Down, CrashLooping, none of that happened. The service and the database both looked completely healthy, just quietly failing writes.

CockroachDB actually exports a metric for this. queue_split_process_failure increases every time the split fails. Healthy clusters don't fail splits, so rate(queue_split_process_failure[15m]) > 0 tells you that there is trouble afoot in the database. I added an alert for this one the next day.

Closing

None of it was broken. MVCC kept the old versions as it was designed to, Raft kept the quorum, split queue and backpressure worked just fine. The client kept saving, over and over, from a tab that was left open. Four conditions that happened to meet at once brought down the database.

Moving 20,000+ customers to a new Stripe account without anyone noticing

Vatsal Patel — Fri, 10 Jul 2026 13:16:46 +0000

We had one Stripe account, in Australia, taking USD off American customers and settling it out as AUD. Roughly 2% of every US transaction went to the international card surcharge and the conversion, before whatever FX did between authorization and settlement. Tax season was worse than it needed to be, and per-region revenue was something you assembled by hand rather than read off a dashboard. The CEO wanted clean financials and lower fees with nothing visible to customers. How we do this, was for me to figure out.

So: a second Stripe account in the US, and everything American moves onto it.

Two products. Tutoring had 2,500 US customers, most of them actively paying, billed per lesson. Many with credits and coupons. Schools had 20,000 US customers and a few dozen subscriptions. Almost all of the risk was in the smaller number. Tutoring went first.

A customer, for the purposes of this, is a Stripe customer object, one or more saved cards, sometimes a cash balance in USD, sometimes a coupon with a promo code attached, sometimes a subscription, and sometimes an invoice that hasn't settled yet.

Six things. Stripe gives you a self serve PAN copy tool that moves two of them.

What the tool does

Card data can only move between Stripe accounts through their self-serve PAN copy tool. It's the only PCI-compliant path, so there's nothing to decide here, you must use it.

It copies the customer objects and preserves the customer IDs. That's the part that makes the whole migration tractable, and it's worth being specific about why. We store the Stripe customer ID on our own customer row, and everything else we hold against Stripe hangs off it. If the tool had reissued customer IDs, every reference in our database would have needed remapping, in the right order, with a lookup table that had to survive the whole cutover. Instead cus_... on the new account is the same string it was on the old one, and the join we already had keeps working.

It copies the attached payment methods, under new IDs, and hands you a CSV mapping every old payment method ID to its new one.

That's the feature. It does not move credits. It does not move coupons or promo codes. It does not move subscriptions or invoices, and it does not move any metadata beyond the customer record itself.

The scripts

Everything was written in Go, one script per object type, all the same shape: default to a dry run that prints what it would change and against which records, take a flag to actually write. Nothing in the set mutates anything without the flag. They're rate limited as well, with enough headroom left under Stripe's limit for production traffic to carry on unaffected, because the migration is sharing that API with the live product the entire time it runs.

The payment method remap is the CSV. For each row, find our record by the customer ID, swap the old payment method ID for the new one. It's the easiest script in the migration precisely because of the ID preservation above.

Credits are a read from the old account and a write to the new one. Shortest script of the lot. This one gave me some pain, but let's save the interesting parts for later.

Coupons

We issue per-customer coupons with a promo code attached to each one. Moving a coupon means recreating it on the new account with the same discount, the same customer restriction, and however many uses that customer has left.

This one is where things go slightly wrong. Stripe lets you put a usage limit on the coupon or on the promo code, and sales had used both, at different times, for reasons that made sense to whoever was doing it. Sometimes the limit is on the coupon. Sometimes on the code. Sometimes there's one on each and they disagree, and then you have to decide which one the customer would consider correct, which is not a technical question.

There's no rule I could write down. Per customer, the number of uses left is wherever the person who set it up happened to put it, so the script reads both sides and works out which one is real. After a lot of pain and a lot of unit tests, I was confident this was now correct.

The invoices we didn't touch

Open and failed invoices are still moving while you migrate. Stripe retries a failed payment on its own for several days. A customer pays an open invoice whenever they get round to reading the email, or whenever Stripe feels like charging their card. Both of those have days left to run and a migration is an instant.

The obvious move is to void them all at cutover and recreate them on the new account, and I didn't want to. You'd be cancelling invoices that were an hour from settling by themselves, then sending someone a second invoice with a different number for money they already owe, and each one of those becomes a conversation with support about a billing relationship that was fine before we came along.

So we left them. 48 hours on the old account, let the retries and the human payments clear whatever they were going to clear, then a script two days later at midnight that voided what was still open and recreated it on the new side. What was left needing a person was a much shorter list than what we started with.

Which meant the old account was still alive for two days, and still sending us webhooks about customers who didn't live there any more.

Webhooks

The backend is two regional clusters, sharded by region. One Stripe account made this easy. Everything went to AU and AU forwarded whatever wasn't its own. With two accounts neither cluster owns the truth, because the account an event came from no longer tells you which cluster the customer is in.

The design I want is each Stripe account pointed straight at the cluster that owns its customers. Both backends already have endpoints that take Stripe webhooks natively, signature verification and all. Nothing in the middle, nothing extra to operate, maybe a day of work.

We shipped a Cloud Function instead. One entry point in front of both accounts, verifying the incoming signature against both accounts' signing secrets, looking up which region the customer belongs to, forwarding to that cluster. Another senior engineer wrote the forwarding half of it while I was on the scripts.

The reason is those 48 hours. The AU account is still emitting events about customers who now live in the US cluster, every time one of those old invoices settles or fails again. An endpoint per region assumes the account that sent the event owns the customer it's about, and for two days that isn't true. The function is the only thing in the system that routes on the customer rather than on the sender.

It comes out when the last invoice on the old account closes. I'd rather run the ugly one with a date on it than the clean one that drops events during the week it matters.

Two smaller ones

Stripe won't let you archive a customer, and deleting them meant losing their history and their data, so the old account keeps all records that look exactly as real as the live ones, forever. A script went through and put (ARCHIVED) on the end of every name. Finance and sales work out of that dashboard all day and nobody is going to operate on a row with that on it.

Subscriptions were a schools problem. Most of them are annual and some had been paid days earlier. Create the subscription the normal way on the new account and Stripe raises an invoice for another year straight away, so a script pulled status, trial state, renewal date and applicable coupons for every active subscription, recreated each one on a trial ending at its existing renewal date, then cancelled the original on the AU side. The trial does nothing except keep Stripe from billing until the date the customer has already paid through.

Midnight

A few hours before the window I ran everything dry against production one more time and read the output. Nothing new in it.

Payment button off on the US app, which nobody was pressing at that hour anyway. Start the copy.

Stripe's docs say it can take up to three days. The writeups I could find said a couple of hours for under 2,000 customers. We'd budgeted three, with something written down for what to do if it ran past that, and really the whole window existed for this one step.

Ten minutes.

Then two hours and fifty minutes of metadata scripts, the cloud function, the backend deploy, live keys and QA, most of which I'd rehearsed enough times that afternoon that I was mostly reading output and checking it said what it had said the last four times.

Schools went three weeks later. Same scripts, same order, eight times the customers, an hour and a half instead of ten minutes. No failures, nothing rate limited. That's the whole story of the second one.

Two audits ran afterwards, both times. For every migrated customer: exists on the new account, has the expected number of payment methods, they're attached, and the IDs match what's in our database. For every customer with a non-zero balance on the old account: the same balance exists on the new one. On the schools side a third checked that every active subscription had been copied, that nobody had been charged twice, and that the original on the AU account was cancelled.

The payment methods were never a problem, for what it's worth. Clean in both audits, both times.

Wrong balance

Reading a customer's balance off the customer object is the obvious way to read a customer's balance. It's what the field is for. Stripe lets a customer hold balances in more than one currency, and the customer object gives you a balance, and we took it.

For five customers it gave us the AUD one, which was zero. The USD balance was sitting right there, funded, but we wrote a zero to their balance.

The credit audit read the same field.

So it passed. Two scripts asking one API the same question agree with each other whatever the answer is, and everything we could see said the migration was fine, and all of it was coming from the same place. The audit checked every customer with a balance, on both accounts, and reconciled them. It just asked the question the same way twice.

An account manager caught it, about ninety minutes after we finished. She'd added credits to one of her accounts the week before, went looking for them on the new account, and knew what the number should have been.

I still don't know why the API did that, and I didn't chase it very far. The fix was to stop reading balances that way, query the cash balance API for AUD and USD explicitly, and run a correction over the five. All of them were right again the same day, before any of them logged in.

Those five accounts were the only thing that went wrong in either migration. Nobody was billed wrong, no ticket was opened, and we never wrote it up. What I'd change is the audit. Check the number against something that didn't come out of the same call, and run it before the cutover rather than after.

Five days of building. A month of waiting for Stripe to verify the new US business account before any of it could run, which there was nothing to do about.