DEV Community: Vatsal Patel

Building a multi-region routing system with Cloudflare Workers

Vatsal Patel — Wed, 02 Sep 2026 15:30:39 +0000

We serve customers primarily in Australia, but we are now expanding to the USA. The timeline for launch is less than 2 months. This is now a race against time to design a multi-region routing system that fits all of our needs. Here is the story.

Background

Almost all of our customers were based in Oceania. We run our Kubernetes Cluster on GCP in Australia. Go microservices, federated GraphQL, gRPC services. 2 products - Tutoring and Schools. All designed for Australia.

Then we expanded to the USA, which meant a new Kubernetes Cluster in US Central. The latency for serving US customers from Australia is an extra 200ms-300ms depending on network conditions - unacceptable. This would mean sharding the data by region, or does it? There are definitely ways to keep a unified dataset even across regions - though we did not need to do so. More on this later.

What are the requirements

If the only requirements were "Americans get served from America", we wouldn't be here discussing this, would we?

Logged in users are served from their own region, wherever they happen to be in the world.
Logged out users are routed geographically, as we have no other information to infer their actual region.
Account Managers and Admins should be able to access both regions from one button, with a single account.
Teaching materials opened via links from the Schools product must be shareable across both regions.

Geography takes care of the logged out user, but nothing else. Using geography for a logged in user can be actively wrong. They might be travelling or simply using a VPN.

Then comes the Admin; we have a lot of admin operations regarding curricula, which will be entirely separate for both clusters. Account Managers need to be able to see and modify information on both clusters. One admin should be able to access both clusters with a single account. We considered showing data of both clusters on one screen, but ruled it out as it may become too ambiguous or confusing, not worth the technical hassle of getting pagination to work cross-region. Next best option: switch the cluster with the flip of a switch.

Sharing is the hardest case to cover. We may have a teacher create some teaching material and share it on social media, where the person clicking on it may be from the other region. While curricula are not quite the same, a lot of the topics and subtopics do tend to be very similar. We observed this during the curriculum integration and decided this was a key requirement we needed to support, annoying as it may be.

To shard, or not to shard

Sharding would be the easier and obvious solution here, but it presents some problems. How do you know which region a user belongs to if your Auth service is sharded? Both regions will simply reject or infinitely redirect the requests to each other in case of a malformed or non-existent user ID. We need some central service or registry that can check for the existence of a user ID and tell us which cluster they belong to. This can be sharded or unsharded, but not sharding is simpler. We use Firebase for Authentication, so we decided not to shard our Auth service.

Not sharding is now difficult; you either end up with a cross-region database, where you either give up consistency or accept a high latency. For our other databases, we decided sharding would be the best approach. Running cross-region Neo4j or CRDB was not a particularly good idea. I have read enough on consensus to know it is possible, but simplicity is better than overfitting a solution to our problem here.

Our options

I brought a long list of options to the first meeting, most of them already crossed out.

A service mesh across both clusters This is the one our CTO wanted to consider the most, Istio or Linkerd. Connect both clusters, but that doesn't really solve the routing problem; the US requests still end up in Australia if we keep a single database. If we shard the databases, we now have a service discovery problem where we have to figure out which database instance to use for a given user.

Connect the two clusters Using Submariner. This also adds the same service discovery problem as above. P.S. I got really annoyed by seeing the Rolex Submariner whenever I googled "submariner".

Multi-region databases CRDB has real multi-region support using REGIONAL BY ROW tables, locality-aware placements, and follower reads. This solves our problem, but we weren't sure what the latency would be like when the 2 regions are on opposite sides of the planet. We also run Neo4j which doesn't have first class multi-regions support so we ended up benching this idea.

A single service that works out user's region and redirects them We could have written a new service for this, using Firebase Auth as its database, but then we have to accommodate the admin requests, admin overrides, the resource sharing and other use cases. We ended up doing something like this, but just in a different way.

Cloudflare Load Balancer and Geo Steering We already use Cloudflare for our Domains, so this looks like a natural solution. It wouldn't be the simplest thing to configure given all of our requirements but it certainly would be possible. This option was amongst the finalists, but the next option won as it worked better for us.

Cloudflare Workers I ended up recommending this one, as it is a natural addition to our networking stack and gives us a lot of control over how we route traffic. We would use Workers as an API gateway of sorts, essentially. This also makes it really easy to express our requirements in code and extend them easily in the future. Lots of libraries are available as well, so we could interact with Firebase Auth and any other common service with an npm package, or write our own code if something doesn't already have a supported package. Additionally, workers are really cheap. 10 million requests included per month + $0.30 per million additional requests. CPU time is fairly generous and quite sufficient since our API gateway is meant to be a very small hop.

Putting it all together

With our chosen solution, we got to work putting it all together.

We still had some decisions to make, what would take precedence when routing requests to a cluster, so we built a hierarchy.

First of all is the subdomains; these are directly connected to the cluster's IP as DNS A records and don't go through the workers at all, very convenient for internal direct routing uses. These are actually not part of the hierarchy, but are completely outside. Workers can use these for DNS override.

Next up is URL path routes. Mainly used for admin overrides and for indicating the country when sharing resources and other social media links. The worker strips the region bits from the URL path before forwarding it to the correct cluster, so no service had to be changed or reconfigured to support this. The admin site gets a selector at the top, which defaults to AU and uses the URL path. Switch it to US and it will use the US scheme, making sure the requests end up in the correct region without switching accounts.

Up next is cookies; we started saving user's country in a separate __country cookie. Started off with only au and us as the supported values, with the hope that there will be tons more. We also store the country on our JWT token as a backup in case the country cookie isn't included.

Geographical location is last. It is derived from request.cf.continent. We tried geographical distance first, but some of our overseas employees landed on AU and others on US, all of them sitting in the same country. So we settled on the continent

function getOriginByGeolocation(request: Request, env: Env): string {
    switch (request.cf?.continent ?? 'OC') {
        case 'EU':
        case 'NA':
        case 'SA':
            return env.US_HOST;
        case 'AN':
        case 'AS':
        case 'AF':
        case 'OC':
            return env.AU_HOST;
        default:
            return env.AU_HOST;
    }
}

Why is Antarctica listed there? Because why not.

Proxying instead of redirecting

We have figured out the cluster, but the job is not yet done. The obvious thing is to return a 302 and let the browser take care of things from there. We can't do that. We had built host matching in our cluster with IngressRoutes so a 302 to our AU host would not match the actual endpoint we want it to hit. Luckily, Cloudflare has a resolveOverride field, which directs the request to an alternate origin server by overriding the standard DNS lookup, while keeping the original Host header intact. This was the primary reason workers worked so well for us. We could set all of this up without a single change to our existing services or Kubernetes networking layer.

This was about 15 lines:

let newReq = new Request(request);
newReq.headers.delete(USER_ID_HEADER);

const jwt = await verifyJWT(request, env);
if (jwt !== null) {
    newReq.headers.set(USER_ID_HEADER, jwt.uid);
}

const { origin, url, cookies } = await getCorrectCountryOrigin(newReq, jwt, env);
const reqInfo = new Request(url, newReq);

let res = await fetch(reqInfo, {
    cf: { resolveOverride: origin },
});

res = new Response(res.body, res);
cookies.forEach((cookie) => res.headers.append('Set-Cookie', cookie));
return res;

There are two copies being made; both are intentional. An incoming Request in the Workers runtime has immutable headers, so a copy is required. new Request(request) and new Response(res.body, res) allow modifying the headers. A second copy is required because we may have changed the URL to remove the region prefix from the path. resolveOverride ensures we send it to the correct cluster. Proxying instead of redirecting also saves an entire round trip on the first request, and also prevents the client from being stuck on a single cluster. If they click on 2 shared links, one for AU and one for US. The earlier one will work but the latter one will break. Proxying prevents this.

We later benchmarked Workers with timing headers and logs, and found these to be extremely performant. Roughly 2-10ms overhead depending on which routing case ends up being true, including a full JWT verify. Much lower than what any of the other solutions likely would have ended up being. We really loved the performance and reliability. Cloudflare hasn't had any major outages affecting us in the 2 years, nor have there been any weird bugs. We have of course pushed some bugs every now and then, but we can only blame ourselves for that.

Where it is now

We had 2 months to launch in the US. This worker setup took about 2 weeks. From requirements to having it running on dev and proxying requests. Setting up the new Kubernetes cluster in the USA, loading up the curricula, adjusting our Firebase and auth services to support regions, all of these took up the rest of the 2 month window. We made it just in time.

That was roughly 2 years ago. The hierarchy hasn't changed since but we have built a lot of stuff around it. More signals above the cookie, a second domain, and the worker becoming the authentication layer for some of our use cases.

Why CockroachDB refused writes to a healthy 155 KiB row

Vatsal Patel — Tue, 11 Aug 2026 16:10:53 +0000

A worksheet in prod stopped saving.

The pod was healthy. 404 MiB of a 2 GiB limit, 655m of 1500m, no restarts. I didn't believe that, so I went and looked at the database too. Three active queries cluster-wide, 12% CPU, all three nodes live. Idle.

Nothing was exhausted, nothing had crashed, and the service still couldn't write.

The software

It's a collaborative editor. Teachers build worksheets, whiteboards and lesson plans, and several people can have the same document open at once. Every document is a CRDT, built on Loro. The browser holds a replica and applies edits to it locally, then pushes them over a WebSocket to a sync server. The server keeps its own copy of each open document in memory, merges whatever arrives into it, and writes the result to CockroachDB v25.x.

That last step is the one that matters here. Persisting a document means exporting the entire Loro doc as a snapshot and writing it into a single BYTEA column, on a single row. Not an append-only log of updates, which is the usual way to store a CRDT. The whole document, on every save.

The document that stopped saving was 155 KiB. Its range was 1 GiB.

A wild goose chase to find the root cause

The red herring: the same service had an unrelated CPU problem running that day, readiness probes flapping, the node pegged, hundreds of timeout errors in the logs. I went through all of it. Every bit real, none of it connected to this. Two separate problems on one service on the same day, and the louder one wasn't the one refusing writes.

A second false trail: I noticed payload sizes varied a lot from one document to the next and read that as clients sending incremental deltas, which would make the write volume real edits.

That was wrong. Varying payload size doesn't imply a delta. A CRDT snapshot of a changing document is a different size every time, the same way two zip files of slightly different inputs come out different sizes. The check that settles it is dividing the payload by the stored snapshot:

document type	writes per resource	payload ÷ snapshot
worksheet	995	0.81
lesson plan	57	0.94
whiteboard	5.9	0.87
text document	18	0.82

Near 1.0 means the client sent as many bytes as the entire stored document. Every document type was doing it. Worksheets weren't doing anything different in kind. They were doing it 169 times more often than whiteboards, and that was the whole difference between a wasteful system and a broken one.

How to wedge a CockroachDB range

The error was in the logs the whole time, buried at a much lower volume than the noise.

split failed while applying backpressure to Put [/Table/111/60/"..."/0]
on range r725: could not find valid split key

Four things had to be true at once for that, each one reasonable on its own.

CockroachDB is MVCC (Multiversion Concurrency Control), so a write never overwrites anything. Every write stores a new copy of the row under the same key at a new timestamp, and the previous copies stay exactly where they are. The key in the storage engine isn't the row; it's the row plus a timestamp.

That's what lets a transaction read a consistent view of the database without locking the rows it reads. A transaction reading at timestamp T sees the newest committed version at or below T of every key it touches. CockroachDB runs SERIALIZABLE by default and there is more machinery than that behind it, since reads leave marks in the timestamp cache that push later writers, and a read that meets an unresolved intent below its own timestamp has to wait on it. But keeping every committed version around is what the rest is built on top of. It's also what AS OF SYSTEM TIME, follower reads and incremental backups are built on. All three are reads at an older timestamp, and they only work if the data as of that timestamp is still on disk.

So old versions can't be dropped at write time. Something has to guarantee they're still there for anyone reading in the past. They get collected later by the MVCC GC queue, once they're older than gc.ttlseconds, which was four hours here. Which means the storage a row occupies isn't its size. It's its size multiplied by how many times you wrote it in the last four hours.

The whole row is one key. CockroachDB stores a row as one key per column family, and this table never defined any beyond the default, so every column sits in the same one. One document, one key, however large the snapshot gets.
A split has to cut between two keys. Ranges are kept under range_max_bytes by splitting, and a split picks a key and cuts the keyspace there: everything below goes to one range, everything above to the other. If every byte in a range belongs to one key and the copies differ only by timestamp, there's nowhere to put the boundary. They can't be separated anyway, because the range is what serves reads of that key at any timestamp, so all of them have to live together.
The client was pushing every 2.2 seconds, whether or not anything had changed.

Here's what the range actually looked like:

keys       1
versions   6,766
val_bytes  1024.02 MiB
live       0.151 MiB

One key. Nearly seven thousand copies of it. A gigabyte of stored versions against 155 KiB of actual row.

0.47 writes per second against a 14,400 second GC window predicts 6,768 versions. There were 6,766. The range was holding exactly one GC window of writes, which is where this stopped being a mystery and became arithmetic. I liked that part a lot.

Nothing was queued or deferred to get there, which is worth being explicit about. Every one of those writes applied immediately: proposed, replicated, committed, visible to the next read. The range grew because that is what a range does when you write to it. Splitting is not part of the write path.

Splitting happens on the split queue. Each store walks its replicas on a timer, reads their size straight off the MVCC stats it already maintains, and queues anything over range_max_bytes. That's deliberately asynchronous, because a split isn't a local operation. It's a distributed transaction that carves the keyspace in two, writes a new range descriptor, and updates the meta ranges that tell the rest of the cluster where keys live. You don't want that on the hot path of a Put.

So there's always a gap between "this range is too big" and "this range has been split", and under normal load, the queue closes it in seconds. Backpressure is what stops a range from outrunning the queue when it doesn't. At twice range_max_bytes, the KV layer stops letting writes into a range with a split pending:

range_max_bytes      536,870,912   (512 MiB)
backpressure at    1,073,741,824
r725               1,073,844,534

It doesn't reject them outright. It holds the batch, waiting for the range to come back under the threshold, and the write fails only when the request runs out of time. That distinction is why the failure surfaced to us as persist timeouts rather than as a clean error, and it's the whole design assumption: the split you're waiting on is going to happen.

100 KiB over the line. And the split was never going to happen.

I sampled the version count twice, 25 seconds apart, to be sure writes were genuinely frozen rather than merely slow. 6,766 both times. While wedged, the range produced about 390 log lines every 15 minutes, continuously, because failed persists retried with no backoff.

Which leaves GC as the only thing that could end it, and GC runs on a queue too, with the same asynchronous, scored shape as the split queue. Each replica tracks a statistic called gc_bytes_age, the volume of collectable garbage multiplied by how long it's been collectable, and the queue prioritises by that rather than by raw size. When it gets to a range it computes a threshold of now - gc.ttlseconds, drops every version older than that, and advances the range's own GC threshold so that later reads below it are refused rather than served wrong.

Two things follow from that shape. GC can never reach anything inside the TTL window, so during a wedge a range can only shed what has already aged past it. And because the queue is scored and periodic rather than continuous, recovery begins when the queue reaches the range, not when the first version becomes collectable.

The second of those is the part I can't fully account for. Getting back under the line needed almost nothing, since the range was sitting 100 KiB over a 1024 MiB threshold, and yet writes stayed refused for 75 to 90 minutes every time. Aging alone doesn't explain a gap that size, so what dominates it has to be when the GC queue got round to the range. I never pinned that down more precisely, and the incident was resolved before it mattered enough to.

The cycle itself is legible enough without it. Roughly two hours of rewriting to rebuild a gigabyte, then the wedge, then GC clears it and it starts over. Five times across two days, always the same row.

And gc.ttlseconds is a floor on retention rather than a target. Retained bytes are write rate times version size times that window, and nothing in the system pushes back on the product.

Raft never failed in any of this. No quorum loss, no elections, nothing. But it sits underneath every part of it, and it's the reason the size limit exists at all.

A range isn't a storage bucket. It's a Raft group: three replicas by default, one of them holding the lease. Every write to that row was a Raft proposal, which the leaseholder proposed, a quorum accepted, and each replica then applied to its own copy. So those 6,766 versions weren't 6,766 disk writes. They were 6,766 rounds of distributed consensus, each shipping a full 155 KiB snapshot across the network, and the gigabyte existed three times over, once per replica.

Size matters to Raft in two more places. A replica that falls far enough behind can't be caught up from the log, because the leader has already truncated the entries it would need, so it gets sent a Raft snapshot instead: the entire range, over the network. Same story when a node is decommissioned and its replicas are rebuilt elsewhere. A 1 GiB range is a 1 GiB transfer, and until it lands that replica isn't contributing to quorum. Keeping ranges small is what keeps rebalancing and recovery cheap enough to happen automatically.

The split is a Raft operation too, committing a new range descriptor through this same group. None of that got as far as running. It failed at the first step, choosing the key to cut at.

So the rule that trapped us exists to keep Raft groups small enough to move around, and we'd built one that could never be divided.

One note if you're coming from Postgres. This isn't a page split. The storage engine is Pebble, an LSM tree, so there are no pages and no fillfactor to tune. Splitting a range is a decision about distribution across a sorted keyspace, not about storage layout. The page-split intuition is the obvious one to reach for and it doesn't transfer.

Where the writes came from

Two thousand consecutive pushes for the wedged document:

payload size        min 130,009   median 130,009   max 130,009
distinct clients    1
inter-write gap     p50 2.24s

Not one byte of variance across any of them. One client, sending the whole document every 2.2 seconds, unchanged. The row's lifetime write counter was at 41,201, which at that cadence is about 25 hours of continuous pushing, and lines up with the first wedge the previous afternoon.

Someone left a tab open.

On the client, the checkpoint gate asked "did any command run during this dispatch?" instead of "did the document change?". A layout loop that measures rendered block heights and reports them back kept producing command work, so it kept re-exporting and re-sending the entire document. On the server, nothing compared the incoming bytes against what was already stored, so each one landed as a fresh 155 KiB version of an identical document.

Five ways out, in the order we considered them

1. Raise `range_max_bytes`

First thing suggested, first thing rejected. It moves the ceiling for every range in the table and does nothing about the accumulation. The ceiling here comes from storing one document per row, not from that number being too small.

2. Lower `gc.ttlseconds`

What we actually did, because it needed no deploy:

ALTER TABLE resources CONFIGURE ZONE USING gc.ttlseconds = 600;

Retained bytes are write rate times version size times retention window. We couldn't touch the write rate without shipping code, so we took the window from four hours to ten minutes. That's 155 KiB × 0.47/s × 600s, or about 43 MiB, against 1024 MiB before.

One range went from 728 MiB to 95 MiB in two minutes. Another went from 579 MiB to 168 MiB. gc_bytes_age on the first fell from 7.2e12 to 3.6e10.

That speed deserves a note, because deleting from an LSM frees nothing immediately. Pebble writes deletion markers and the space comes back at compaction, whenever that happens to be. But the size the split queue reads is the MVCC stats, not the disk footprint, and GC updates those the moment it runs. So the range stopped counting as oversized well before it stopped occupying the bytes, which is the only reason a one-line config change unwedged production in minutes.

This settles at a steady state rather than counting down to anything. Versions arrive and expire at the same rate, so the pile reaches a size and stays there.

I checked system.protected_ts_records first, and it's worth being precise about which direction that check runs in. A protected timestamp pins the GC threshold: while one is held, GC cannot collect anything newer than it, which is how a backup or a changefeed keeps its reads valid for as long as it takes to finish. So a record sitting on this table wouldn't have been the thing at risk. It would have defeated the fix. GC would have refused to drop below it, the range would never have drained, and prod would have stayed wedged with a config change applied and nothing to show for it. The table was empty.

What an empty table doesn't tell you is whether anything was reading historically without taking a protected timestamp. Those are the consumers this actually breaks, and after the change an AS OF SYSTEM TIME read further back than ten minutes on that table fails outright.

3. Ship the coalescing persist queue

There was already an open PR for one: 750 ms debounce, 5 second ceiling. A 5 second ceiling caps a continuously-edited document at 2,880 versions per GC window, which moves the wedge threshold to roughly 364 KiB.

The wedged document was 155 KiB on disk, so 364 KiB sounds like room. It isn't much, and there are two thresholds here rather than one. 364 KiB is where a document wedges. 182 KiB is the earlier one, where its range crosses range_max_bytes and starts attempting splits it can't finish, without being backpressured yet. Another worksheet had already reached 188 KB and was still climbing, which puts it past the first line and heading for the second.

So this buys headroom without removing the ceiling. It also doesn't compare content, so identical resends still get written, only less often.

I'm not actually sure it would have prevented this one. All of that assumes the 5 second ceiling binds, but at a 2.2 second arrival rate the 750 ms debounce expires between pushes, so each push probably still flushes on its own and the rate doesn't move at all. I haven't tested it.

4. Hash the snapshot, skip the write when it matches

The real fix, and the one that shipped. It takes almost all of these writes to zero no matter what any client does, and it holds for whatever version of the client happens to be running, which matters when your clients are browser tabs you can't force to reload.

5. Chunk the snapshot, or move to an append-only update log

The only option that removes the single-key property instead of buying room underneath it. Also the largest change by a wide margin, and not something you do on a Wednesday afternoon with prod wedging every few hours.

The monitoring

None of our alerts could have caught this, and it wasn't a threshold problem. OOMKilled, MemoryHigh, CpuThrottled, CrashLooping, Down. Every one of them stays quiet on a service that is perfectly healthy and simply not permitted to make progress. We had no signal for that shape of failure.

CockroachDB was already exporting exactly the right counter. queue_split_process_failure goes up on every failed split attempt, and healthy clusters don't fail splits, so rate(queue_split_process_failure[15m]) > 0 is about as clean a signal as you get. The node holding the lease for r725 was at 2,114. Nothing was scraping it.

The rule I then wrote on top of it failed twice over. Prometheus evaluated it and it went active, but Alertmanager dropped the notification on the floor: the default receiver is null behind an allow-list regex on alert names, and a new name that isn't in that regex gets discarded without a trace. So it fired and nobody heard it. Any new rule here needs an entry in that regex or its own route, which is not a thing you find out by testing the expression.

Once it was routed, it stayed active for about 25 minutes after the condition cleared. Not because counters only go up, which was my first guess: rate() does return to zero. It's that the 15 minute lookback keeps the rate positive until the window slides past the last failed split, and Alertmanager's resolve delay adds the rest. Shorten the window and you trade that against missing sparse failures.

The one I'd hand to someone else is the dedup counter we added afterwards, which counts writes skipped because the content hash matched what was stored. It started out near the entire write volume. That means a fall toward zero is the direction that should worry you, because it says writes have gone back to being genuinely distinct and the ceiling is live again. Every other metric on that dashboard alarms upward.

Three CockroachDB quirks worth knowing before you go looking, none of them well documented: crdb_internal.tables keys on table_id, not id. SHOW ZONE CONFIGURATION wants the real database name, and these tables live in defaultdb. And round() errors out if you mix decimal and float8, so cast first.

Moving 20,000+ customers to a new Stripe account without anyone noticing

Vatsal Patel — Fri, 10 Jul 2026 13:16:46 +0000

We had one Stripe account, in Australia, taking USD off American customers and settling it out as AUD. Roughly 2% of every US transaction went to the international card surcharge and the conversion, before whatever FX did between authorization and settlement. Tax season was worse than it needed to be, and per-region revenue was something you assembled by hand rather than read off a dashboard. The CEO wanted clean financials and lower fees with nothing visible to customers. How we do this, was for me to figure out.

So: a second Stripe account in the US, and everything American moves onto it.

Two products. Tutoring had 2,500 US customers, most of them actively paying, billed per lesson. Many with credits and coupons. Schools had 20,000 US customers and a few dozen subscriptions. Almost all of the risk was in the smaller number. Tutoring went first.

A customer, for the purposes of this, is a Stripe customer object, one or more saved cards, sometimes a cash balance in USD, sometimes a coupon with a promo code attached, sometimes a subscription, and sometimes an invoice that hasn't settled yet.

Six things. Stripe gives you a self serve PAN copy tool that moves two of them.

What the tool does

Card data can only move between Stripe accounts through their self-serve PAN copy tool. It's the only PCI-compliant path, so there's nothing to decide here, you must use it.

It copies the customer objects and preserves the customer IDs. That's the part that makes the whole migration tractable, and it's worth being specific about why. We store the Stripe customer ID on our own customer row, and everything else we hold against Stripe hangs off it. If the tool had reissued customer IDs, every reference in our database would have needed remapping, in the right order, with a lookup table that had to survive the whole cutover. Instead cus_... on the new account is the same string it was on the old one, and the join we already had keeps working.

It copies the attached payment methods, under new IDs, and hands you a CSV mapping every old payment method ID to its new one.

That's the feature. It does not move credits. It does not move coupons or promo codes. It does not move subscriptions or invoices, and it does not move any metadata beyond the customer record itself.

The scripts

Everything was written in Go, one script per object type, all the same shape: default to a dry run that prints what it would change and against which records, take a flag to actually write. Nothing in the set mutates anything without the flag. They're rate limited as well, with enough headroom left under Stripe's limit for production traffic to carry on unaffected, because the migration is sharing that API with the live product the entire time it runs.

The payment method remap is the CSV. For each row, find our record by the customer ID, swap the old payment method ID for the new one. It's the easiest script in the migration precisely because of the ID preservation above.

Credits are a read from the old account and a write to the new one. Shortest script of the lot. This one gave me some pain, but let's save the interesting parts for later.

Coupons

We issue per-customer coupons with a promo code attached to each one. Moving a coupon means recreating it on the new account with the same discount, the same customer restriction, and however many uses that customer has left.

This one is where things go slightly wrong. Stripe lets you put a usage limit on the coupon or on the promo code, and sales had used both, at different times, for reasons that made sense to whoever was doing it. Sometimes the limit is on the coupon. Sometimes on the code. Sometimes there's one on each and they disagree, and then you have to decide which one the customer would consider correct, which is not a technical question.

There's no rule I could write down. Per customer, the number of uses left is wherever the person who set it up happened to put it, so the script reads both sides and works out which one is real. After a lot of pain and a lot of unit tests, I was confident this was now correct.

The invoices we didn't touch

Open and failed invoices are still moving while you migrate. Stripe retries a failed payment on its own for several days. A customer pays an open invoice whenever they get round to reading the email, or whenever Stripe feels like charging their card. Both of those have days left to run and a migration is an instant.

The obvious move is to void them all at cutover and recreate them on the new account, and I didn't want to. You'd be cancelling invoices that were an hour from settling by themselves, then sending someone a second invoice with a different number for money they already owe, and each one of those becomes a conversation with support about a billing relationship that was fine before we came along.

So we left them. 48 hours on the old account, let the retries and the human payments clear whatever they were going to clear, then a script two days later at midnight that voided what was still open and recreated it on the new side. What was left needing a person was a much shorter list than what we started with.

Which meant the old account was still alive for two days, and still sending us webhooks about customers who didn't live there any more.

Webhooks

The backend is two regional clusters, sharded by region. One Stripe account made this easy. Everything went to AU and AU forwarded whatever wasn't its own. With two accounts neither cluster owns the truth, because the account an event came from no longer tells you which cluster the customer is in.

The design I want is each Stripe account pointed straight at the cluster that owns its customers. Both backends already have endpoints that take Stripe webhooks natively, signature verification and all. Nothing in the middle, nothing extra to operate, maybe a day of work.

We shipped a Cloud Function instead. One entry point in front of both accounts, verifying the incoming signature against both accounts' signing secrets, looking up which region the customer belongs to, forwarding to that cluster. Another senior engineer wrote the forwarding half of it while I was on the scripts.

The reason is those 48 hours. The AU account is still emitting events about customers who now live in the US cluster, every time one of those old invoices settles or fails again. An endpoint per region assumes the account that sent the event owns the customer it's about, and for two days that isn't true. The function is the only thing in the system that routes on the customer rather than on the sender.

It comes out when the last invoice on the old account closes. I'd rather run the ugly one with a date on it than the clean one that drops events during the week it matters.

Two smaller ones

Stripe won't let you archive a customer, and deleting them meant losing their history and their data, so the old account keeps all records that look exactly as real as the live ones, forever. A script went through and put (ARCHIVED) on the end of every name. Finance and sales work out of that dashboard all day and nobody is going to operate on a row with that on it.

Subscriptions were a schools problem. Most of them are annual and some had been paid days earlier. Create the subscription the normal way on the new account and Stripe raises an invoice for another year straight away, so a script pulled status, trial state, renewal date and applicable coupons for every active subscription, recreated each one on a trial ending at its existing renewal date, then cancelled the original on the AU side. The trial does nothing except keep Stripe from billing until the date the customer has already paid through.

Midnight

A few hours before the window I ran everything dry against production one more time and read the output. Nothing new in it.

Payment button off on the US app, which nobody was pressing at that hour anyway. Start the copy.

Stripe's docs say it can take up to three days. The writeups I could find said a couple of hours for under 2,000 customers. We'd budgeted three, with something written down for what to do if it ran past that, and really the whole window existed for this one step.

Ten minutes.

Then two hours and fifty minutes of metadata scripts, the cloud function, the backend deploy, live keys and QA, most of which I'd rehearsed enough times that afternoon that I was mostly reading output and checking it said what it had said the last four times.

Schools went three weeks later. Same scripts, same order, eight times the customers, an hour and a half instead of ten minutes. No failures, nothing rate limited. That's the whole story of the second one.

Two audits ran afterwards, both times. For every migrated customer: exists on the new account, has the expected number of payment methods, they're attached, and the IDs match what's in our database. For every customer with a non-zero balance on the old account: the same balance exists on the new one. On the schools side a third checked that every active subscription had been copied, that nobody had been charged twice, and that the original on the AU account was cancelled.

The payment methods were never a problem, for what it's worth. Clean in both audits, both times.

Wrong balance

Reading a customer's balance off the customer object is the obvious way to read a customer's balance. It's what the field is for. Stripe lets a customer hold balances in more than one currency, and the customer object gives you a balance, and we took it.

For five customers it gave us the AUD one, which was zero. The USD balance was sitting right there, funded, but we wrote a zero to their balance.

The credit audit read the same field.

So it passed. Two scripts asking one API the same question agree with each other whatever the answer is, and everything we could see said the migration was fine, and all of it was coming from the same place. The audit checked every customer with a balance, on both accounts, and reconciled them. It just asked the question the same way twice.

An account manager caught it, about ninety minutes after we finished. She'd added credits to one of her accounts the week before, went looking for them on the new account, and knew what the number should have been.

I still don't know why the API did that, and I didn't chase it very far. The fix was to stop reading balances that way, query the cash balance API for AUD and USD explicitly, and run a correction over the five. All of them were right again the same day, before any of them logged in.

Those five accounts were the only thing that went wrong in either migration. Nobody was billed wrong, no ticket was opened, and we never wrote it up. What I'd change is the audit. Check the number against something that didn't come out of the same call, and run it before the cutover rather than after.

Five days of building. A month of waiting for Stripe to verify the new US business account before any of it could run, which there was nothing to do about.