Building a multi-region routing system with Cloudflare Workers

We serve customers primarily in Australia, but we are now expanding to the USA. The timeline for launch is less than 2 months. This is now a race against time to design a multi-region routing system that fits all of our needs. Here is the story.

Background

Almost all of our customers were based in Oceania. We run our Kubernetes Cluster on GCP in Australia. Go microservices, federated GraphQL, gRPC services. 2 products - Tutoring and Schools. All designed for Australia.

Then we expanded to the USA, which meant a new Kubernetes Cluster in US Central. The latency for serving US customers from Australia is an extra 200ms-300ms depending on network conditions - unacceptable. This would mean sharding the data by region, or does it? There are definitely ways to keep a unified dataset even across regions - though we did not need to do so. More on this later.

What are the requirements

If the only requirements were "Americans get served from America", we wouldn't be here discussing this, would we?

1. Logged in users are served from their own region, wherever they happen to be in the world.
2. Logged out users are routed geographically, as we have no other information to infer their actual region.
3. Account Managers and Admins should be able to access both regions from one button, with a single account.
4. Teaching materials opened via links from the Schools product must be shareable across both regions.

Geography takes care of the logged out user, but nothing else. Using geography for a logged in user can be actively wrong. They might be travelling or simply using a VPN.

Then comes the Admin; we have a lot of admin operations regarding curricula, which will be entirely separate for both clusters. Account Managers need to be able to see and modify information on both clusters. One admin should be able to access both clusters with a single account. We considered showing data of both clusters on one screen, but ruled it out as it may become too ambiguous or confusing, not worth the technical hassle of getting pagination to work cross-region. Next best option: switch the cluster with the flip of a switch.

Sharing is the hardest case to cover. We may have a teacher create some teaching material and share it on social media, where the person clicking on it may be from the other region. While curricula are not quite the same, a lot of the topics and subtopics do tend to be very similar. We observed this during the curriculum integration and decided this was a key requirement we needed to support, annoying as it may be.

To shard, or not to shard

Sharding would be the easier and obvious solution here, but it presents some problems. How do you know which region a user belongs to if your Auth service is sharded? Both regions will simply reject or infinitely redirect the requests to each other in case of a malformed or non-existent user ID. We need some central service or registry that can check for the existence of a user ID and tell us which cluster they belong to. This can be sharded or unsharded, but not sharding is simpler. We use Firebase for Authentication, so we decided not to shard our Auth service.

Not sharding is now difficult; you either end up with a cross-region database, where you either give up consistency or accept a high latency. For our other databases, we decided sharding would be the best approach. Running cross-region Neo4j or CRDB was not a particularly good idea. I have read enough on consensus to know it is possible, but simplicity is better than overfitting a solution to our problem here.

Our options

I brought a long list of options to the first meeting, most of them already crossed out.

A service mesh across both clusters This is the one our CTO wanted to consider the most, Istio or Linkerd. Connect both clusters, but that doesn't really solve the routing problem; the US requests still end up in Australia if we keep a single database. If we shard the databases, we now have a service discovery problem where we have to figure out which database instance to use for a given user.

Connect the two clusters Using Submariner. This also adds the same service discovery problem as above. P.S. I got really annoyed by seeing the Rolex Submariner whenever I googled "submariner".

Multi-region databases CRDB has real multi-region support using REGIONAL BY ROW tables, locality-aware placements, and follower reads. This solves our problem, but we weren't sure what the latency would be like when the 2 regions are on opposite sides of the planet. We also run Neo4j which doesn't have first class multi-regions support so we ended up benching this idea.

A single service that works out user's region and redirects them We could have written a new service for this, using Firebase Auth as its database, but then we have to accommodate the admin requests, admin overrides, the resource sharing and other use cases. We ended up doing something like this, but just in a different way.

Cloudflare Load Balancer and Geo Steering We already use Cloudflare for our Domains, so this looks like a natural solution. It wouldn't be the simplest thing to configure given all of our requirements but it certainly would be possible. This option was amongst the finalists, but the next option won as it worked better for us.

Cloudflare Workers I ended up recommending this one, as it is a natural addition to our networking stack and gives us a lot of control over how we route traffic. We would use Workers as an API gateway of sorts, essentially. This also makes it really easy to express our requirements in code and extend them easily in the future. Lots of libraries are available as well, so we could interact with Firebase Auth and any other common service with an npm package, or write our own code if something doesn't already have a supported package. Additionally, workers are really cheap. 10 million requests included per month + $0.30 per million additional requests. CPU time is fairly generous and quite sufficient since our API gateway is meant to be a very small hop.

Putting it all together

With our chosen solution, we got to work putting it all together.

We still had some decisions to make, what would take precedence when routing requests to a cluster, so we built a hierarchy.

First of all is the subdomains; these are directly connected to the cluster's IP as DNS A records and don't go through the workers at all, very convenient for internal direct routing uses. These are actually not part of the hierarchy, but are completely outside. Workers can use these for DNS override.

Next up is URL path routes. Mainly used for admin overrides and for indicating the country when sharing resources and other social media links. The worker strips the region bits from the URL path before forwarding it to the correct cluster, so no service had to be changed or reconfigured to support this. The admin site gets a selector at the top, which defaults to AU and uses the URL path. Switch it to US and it will use the US scheme, making sure the requests end up in the correct region without switching accounts.

Up next is cookies; we started saving user's country in a separate __country cookie. Started off with only au and us as the supported values, with the hope that there will be tons more. We also store the country on our JWT token as a backup in case the country cookie isn't included.

Geographical location is last. It is derived from request.cf.continent. We tried geographical distance first, but some of our overseas employees landed on AU and others on US, all of them sitting in the same country. So we settled on the continent

function getOriginByGeolocation(request: Request, env: Env): string {
    switch (request.cf?.continent ?? 'OC') {
        case 'EU':
        case 'NA':
        case 'SA':
            return env.US_HOST;
        case 'AN':
        case 'AS':
        case 'AF':
        case 'OC':
            return env.AU_HOST;
        default:
            return env.AU_HOST;
    }
}

Why is Antarctica listed there? Because why not.

Proxying instead of redirecting

We have figured out the cluster, but the job is not yet done. The obvious thing is to return a 302 and let the browser take care of things from there. We can't do that. We had built host matching in our cluster with IngressRoutes so a 302 to our AU host would not match the actual endpoint we want it to hit. Luckily, Cloudflare has a resolveOverride field, which directs the request to an alternate origin server by overriding the standard DNS lookup, while keeping the original Host header intact. This was the primary reason workers worked so well for us. We could set all of this up without a single change to our existing services or Kubernetes networking layer.

This was about 15 lines:

let newReq = new Request(request);
newReq.headers.delete(USER_ID_HEADER);

const jwt = await verifyJWT(request, env);
if (jwt !== null) {
    newReq.headers.set(USER_ID_HEADER, jwt.uid);
}

const { origin, url, cookies } = await getCorrectCountryOrigin(newReq, jwt, env);
const reqInfo = new Request(url, newReq);

let res = await fetch(reqInfo, {
    cf: { resolveOverride: origin },
});

res = new Response(res.body, res);
cookies.forEach((cookie) => res.headers.append('Set-Cookie', cookie));
return res;

There are two copies being made; both are intentional. An incoming Request in the Workers runtime has immutable headers, so a copy is required. new Request(request) and new Response(res.body, res) allow modifying the headers. A second copy is required because we may have changed the URL to remove the region prefix from the path. resolveOverride ensures we send it to the correct cluster. Proxying instead of redirecting also saves an entire round trip on the first request, and also prevents the client from being stuck on a single cluster. If they click on 2 shared links, one for AU and one for US. The earlier one will work but the latter one will break. Proxying prevents this.

We later benchmarked Workers with timing headers and logs, and found these to be extremely performant. Roughly 2-10ms overhead depending on which routing case ends up being true, including a full JWT verify. Much lower than what any of the other solutions likely would have ended up being. We really loved the performance and reliability. Cloudflare hasn't had any major outages affecting us in the 2 years, nor have there been any weird bugs. We have of course pushed some bugs every now and then, but we can only blame ourselves for that.

Where it is now

We had 2 months to launch in the US. This worker setup took about 2 weeks. From requirements to having it running on dev and proxying requests. Setting up the new Kubernetes cluster in the USA, loading up the curricula, adjusting our Firebase and auth services to support regions, all of these took up the rest of the 2 month window. We made it just in time.

That was roughly 2 years ago. The hierarchy hasn't changed since but we have built a lot of stuff around it. More signals above the cookie, a second domain, and the worker becoming the authentication layer for some of our use cases.