A ticket lands on your desk: one of your load balancing services is using six gigabytes of memory and nobody knows why. You could throw more RAM at it. Or you could do what a team at Cloudflare just did, which is sit down with a first-semester probability formula, and use it to delete 100 terabytes of memory across their network.
That second option is real, and it is documented in a Cloudflare engineering post published this month titled "Saving another 100TB of RAM with math (and Rust)". The post climbed the Hacker News front page with over 450 points, and it earned that attention. It is one of the clearest public write-ups of a memory optimization I have read, and the best part is that the two core moves are not Cloudflare-specific at all. One is a data layout fix. The other is a formula that tells you how many hashes you actually need, which is almost always fewer than your defaults.
Full disclosure before we go further: I have not operated Cloudflare-scale infrastructure, and I have not run the new pingora-ketama v2 ring in production. Everything below is sourced from the Cloudflare post and the linked source code, all cited inline. What I have done is work through their math and checked it against my own numbers, and I will show you how to do the same for whatever hash ring you are running.
The problem: 6GB of hash rings nobody asked for
Cloudflare's internal load balancer, Pingora Backend Router, routes cacheable requests to servers by hashing URLs onto a consistent hash ring. If you have ever wondered how a CDN figures out "which of my 100 servers in this data center should hold this file", consistent hashing is the usual answer.
How consistent hashing works in one paragraph. Take a hash function with a 32-bit output. Hash each server (say, by its IP address) onto a number line from 0 to about 4.3 billion. Hash each request the same way. Each request belongs to the first server to the left of it on the line. When a server dies, only the requests that hashed to its segment move. Everything else stays put. That stability is the whole point, and it is why the technique shows up in load balancers, distributed caches, and database sharding everywhere.
The catch is statistics. With one hash point per server, the segments are random slices of the number line, and random slices are wildly uneven. Cloudflare gives the formula: with N servers, the expected segment share is 1/N, and the coefficient of variation (the error margin relative to that target) is sqrt((N-1)/(N+1)). At N=100, that is about 99 percent. In plain terms, some servers could be handling roughly twice their fair share of requests while others sit nearly idle.
The industry fix makes the memory problem worse. Everyone, including NGINX and Pingora's defaults, solved the imbalance by giving each server not one hash point but 160 of them. The same formula says 160 points drops the error margin from about 99 percent to about 8 percent. Requests land fairly. But now your ring holds 160 entries per server, and Cloudflare stacked more complexity on top: weighted servers (more hashes for bigger disks) and, because compliance rules and feature flags mean only some servers can serve some requests, entire duplicate rings for every combination of features. A handful of features means dozens of rings. That is where the 6GB per process came from.
Fix one: your 32-bit index is 16 bits of padding
The first fix came from an engineer named Zaidoon, and it is the kind of thing you can check in your own codebase today. Each entry in the ring stores a hash and a server index:
struct Point {
hash: u32,
index: u32,
}
The hash needs 32 bits. The index does not. Cloudflare is never going to route across more than about 65,000 servers in one ring, so the index fits in 16 bits:
struct PointV2 {
hash: u32,
index: u16,
}
Here is the trap. In Rust, this change saves nothing on its own. Struct alignment rules require the total size to be a multiple of the largest field, which is the 32-bit hash, so the struct stays at 8 bytes with 2 bytes of padding. The same is true of C and C++, and Java has comparable object layout behavior with field padding.
The workaround is to store the fields as a raw byte array and read them back with getters:
struct Point([u8; 6]);
impl Point {
fn hash(&self) -> u32 {
u32::from_ne_bytes(self.0[0..4].try_into().unwrap())
}
fn index(&self) -> u16 {
u16::from_ne_bytes(self.0[4..6].try_into().unwrap())
}
}
Six bytes instead of eight. That single change cut the memory used by the hash rings by 25 percent, and Cloudflare notes the packed version compiles to the same machine code as the naive struct. No runtime cost. Just 2 bytes per entry that were pure waste, multiplied by millions of entries across thousands of servers.
Before you scroll past this as a Rust curiosity: if you are storing ID pairs, coordinate tuples, or any fixed-shape records at scale, check what your language's alignment rules are doing to your footprint. sizeof in C, struct.calcsize in Python, or a heap dump in the JVM will tell you in minutes whether you are paying a padding tax.
Fix two: the formula that says you are 90 percent over-provisioned
The second fix is the more interesting one, because it removes the "just add more hashes" reflex. Cloudflare's author derived the exact formula (not an approximation) for the error margin when each server has k hash points:
- Expected share per server: 1/N
- Coefficient of variation: sqrt((N-1)/(N*k+1))
Watch what that second formula does as k grows. Going from 1 hash to 10 hashes per server is a massive improvement. Going from 10,000 to 100,000 buys you almost nothing. Each additional order of magnitude in hash count yields a smaller and smaller reduction in error. It is diminishing returns, but with an exact curve you can read numbers off of.
Cloudflare's own numbers made their over-provisioning obvious. Their weighted setup worked out to roughly 100,000 hash points for a large server. The formula showed that the last 90,000 of those points were buying a 0.7 percent error improvement. Then there is a second, sneaky problem: with 32-bit hashes, generating hundreds of thousands of points starts running into collisions, the same birthday-paradox math that breaks naive password hashing. Collisions silently drop some of the benefit you thought you were buying. Simulations confirmed that for data centers with 2,048 servers, the real error rate starts climbing again somewhere between 10,000 and 100,000 hashes per server.
So they cut the hash count by 90 percent. Combined with the 6-byte struct, the total was 100TB of RAM returned to the network, on top of the 100TB their DNS team reclaimed the month before with a similar exercise.
The migration lesson matters as much as the math
The part of the post I would flag for anyone running a stateful system: they did not flip the new ring on globally. Changing the hash ring changes where requests go, and a global cutover would have invalidated nearly all cached content at once, turning a memory win into an origin-server meltdown.
Instead:
- Both rings ran side by side, and a stable per-request decision picked which ring handled each request, which also gave a clean rollback path with no redeploy.
- The rollout was scoped by data center first, small validation sites, then progressively larger groups, so the blast radius of any cache churn stayed contained.
- They watched backend-selection traces, ring-version counters, connection errors, process memory, startup time, cache behavior, and origin traffic until the migration hit 100 percent, then deleted the old-ring code.
That is the standard playbook for any change that reshuffles work across servers, whether the trigger is a hashing change, a shard re-split, or a new load balancer. Stability per request, scoped rollout, measurable exit criteria.
The checklist: run this on your own stack
If you take one thing from this article, take this audit. It costs an afternoon:
- Find your hash count. Whatever consistent hashing you run, find the points-per-node setting. 160 is the NGINX and Ketama default and a likely candidate. If nobody chose that number deliberately, it is a candidate for the formula.
- Compute your error curve. For your server count N, evaluate sqrt((N-1)/(N*k+1)) at your current k and at k/10. If the error difference is under a percent, you are storing 10x the ring you need. A spreadsheet is enough; no statistics degree required.
- Check your record width. Look at the struct or object holding hash-plus-index. If the index field is wider than log2 of your realistic maximum node count, you are paying alignment padding for nothing. Pack it or store raw bytes.
- Check for ring duplication. Feature flags, tenant isolation, or compliance routing that multiply whole rings are where footprints explode combinatorially. If you have 2-to-the-N rings for N features, that is your 6GB.
- Never cutover globally. Dual-run, roll out per shard or per region, watch origin traffic like a hawk, and keep a rollback that does not need a redeploy.
What this story is really about
The reason this post resonated is not the 100TB number. It is the method. Two engineers looked at a default that had been running for years, asked "what is this costing us, and what would the math say if we actually computed it", and got a six-figure-in-dollars answer from one formula and 2 bytes of padding. No new framework, no hardware purchase, no rewrite. The changes shipped in the open source pingora-ketama crate as an opt-in v2 ring, so you can read the actual diff rather than trusting my summary.
Defaults are decisions someone else made for your workload. Most of the time they are fine. Occasionally they are silently costing you 100 terabytes. The only way to know which is to run the numbers yourself.
I write about backend engineering, performance, and the systems behind the tools we all use. Subscribe, it is free, and it tells me this kind of deep dive is worth doing.
Now the question back to you: have you ever audited a "set and forget" default in your infrastructure and found it badly mismatched to your actual workload? What did it cost you, or what did fixing it save? I am collecting stories for a follow-up piece, and the comments are the best place to share.
Sources
- Cloudflare: Saving another 100TB of RAM with math (and Rust)
- Cloudflare DNS memory optimization, the earlier 100TB: dns-cache-memory-optimization-1111
- pingora-ketama crate with the v2 ring changes
- The NGINX hash module with the hardcoded 160: ngx_http_upstream_hash_module.c
Top comments (0)