You Added Ten Nodes and Throughput Didn't Move: The Hot Shard Problem

The problem

A team I was helping had a straightforward scaling story: their write-heavy service was maxing out a single Postgres instance, so they sharded it — twelve shards, keyed by tenant_id. Capacity should have gone up roughly 12x. Instead, p99 write latency barely moved, and CPU on exactly one shard sat pinned at 95% while the other eleven idled in the 20-30% range.

They added four more shards. Same shard stayed pinned. The other fifteen got a little more idle.

This is the hot shard problem, and it shows up anywhere you partition data by a key: database sharding, Kafka topic partitions, consistent-hash caches, even queue routing. The team's instinct — "we're capacity constrained, add nodes" — is the correct response to a uniform load problem and the wrong response to this one, and the graphs won't tell you which one you have unless you know to look per-partition instead of in aggregate.

Why it happens

Sharding assumes your partition key distributes load evenly across the keyspace. It almost never does, because real-world entities aren't uniform. In that team's case, tenant_id looked fine in a COUNT(DISTINCT tenant_id) query — thousands of tenants, nicely spread across twelve hash buckets. But load isn't proportional to tenant count, it's proportional to tenant activity, and they had one enterprise customer generating 40% of total write volume. Whatever shard that customer's hash landed on became the ceiling for the entire system, permanently, regardless of how many shards you add around it.

The aggregate dashboards actively hid this. "Average CPU across shards: 34%" looks healthy. Nobody was graphing CPU per shard, because nobody expected the distribution to matter until the p99 numbers stopped responding to capacity added elsewhere.

The same failure mode shows up in Kafka when one partition key (a popular product ID, a big customer's account ID, a retry-storm source) gets a disproportionate share of messages — consumer lag climbs on that one partition while the rest of the consumer group sits idle. It shows up in consistent-hash caches when a handful of keys go viral. The mechanism is always the same: your scaling unit is the partition, but your load isn't partitioned evenly, so adding partitions increases total capacity without increasing the capacity available to the bottlenecked one.

What to do about it

First, instrument per-partition, not just aggregate. If you can't produce a chart of "CPU/QPS/lag by shard ID" today, you can't diagnose this — you'll just keep seeing "we have headroom" in the aggregate while one shard is on fire.

Second, decouple the partition key from the thing causing skew. A few concrete options, roughly in order of how much they cost to retrofit:

• Composite keys. Instead of hashing on tenant_id alone, hash on tenant_id + user_id or tenant_id + entity_id. This spreads a single hot tenant's writes across many shards instead of pinning them to one. You lose the ability to co-locate all of a tenant's data on one shard, which matters if you rely on that for joins or transactions — know the trade-off before you take it.
• Salting. Append a small random or round-robined suffix (tenant_id#0 through tenant_id#7) to fan a hot key out across N sub-buckets, then merge on read. Cheap to add, adds read-side complexity.
• Consistent hashing with enough virtual nodes. If your skew is closer to "some shards run hotter than others" rather than "one key dominates everything," increasing virtual nodes per physical shard (100+, not 10) smooths the distribution meaningfully without a key redesign.
• Explicit hot-key isolation. For the extreme outlier — the one customer, the one product — sometimes the honest fix is giving them a dedicated shard on purpose instead of pretending the hash function will treat them like everyone else.

Whatever you pick, load-test it by simulating your actual skew, not a uniform key distribution. A rebalancing scheme that looks great against synthetic evenly-distributed traffic can fail identically to the original design against your real traffic, because the thing you're fixing is a property of your data, not your algorithm.

Key takeaways

• Aggregate capacity metrics can look completely healthy while one partition is saturated — always graph per-shard, not just fleet-wide averages.
• Adding nodes only helps if the added capacity is reachable by the bottlenecked traffic; a naive partition key can make that structurally impossible.
• Real-world keys (tenant, customer, product) are rarely uniform in activity, even when they're uniform in count.
• Fixes (composite keys, salting, more virtual nodes, explicit isolation) trade off against co-location and read complexity — pick based on what you actually need from the key, not just what's easiest to bolt on.
• Test rebalancing against your real skew distribution, not synthetic uniform load, or you'll ship a fix that doesn't fix anything.