DEV Community

Cover image for Redis Cluster Won't Shard Your Hot Leaderboard
Trung Duong
Trung Duong

Posted on

Redis Cluster Won't Shard Your Hot Leaderboard

#redis #architecture #scalability #backend

"We use Redis Cluster" can mean two very different things:

  1. Our dataset is distributed across Redis nodes.
  2. Every individual data structure is distributed across Redis nodes.

The first can be true while the second is false.

That distinction matters for leaderboards.

In Podium, each leaderboard uses
several Redis keys and atomic Lua scripts. Redis Cluster helps us scale a large
fleet of independent leaderboards, but it cannot split one giant sorted set
across primaries.

We are sharing this architecture because "Redis Cluster scales horizontally"
is true only after you define what the system actually shards.

GitHub logo TeneficGames / podium

High-performance, Redis-backed leaderboards for games and competitive applications.

Podium

CI codecov

Podium high-performance distributed leaderboard

High-performance, Redis-backed leaderboards for games and competitive applications.

Podium provides ready-to-run HTTP and gRPC APIs for scores, ranks, seasons, and player-relative views. It is designed for backend teams operating large fleets of independent leaderboards without provisioning each leaderboard in advance.

  • Fair, deterministic ordering when scores are equal.
  • Single and bulk score updates, including multi-leaderboard fan-out.
  • Standalone Redis and real Redis Cluster integration coverage.
  • Deploy one multi-architecture OCI image with Docker, containerd, Kubernetes or another OCI-compatible runtime.

Quickstart · Performance · API · Documentation · Helm chart · Docker Hub · GHCR

Quickstart

Start Redis 8.2 and the latest stable Podium image:

docker network create podium
docker run --detach --name podium-redis --network podium redis:8.2-alpine

docker run --detach --rm --name podium \
  --network podium \
  --publish 8880:8880 \
  --publish 8881:8881 \
  --env PODIUM_REDIS_HOST=podium-redis \
  --env PODIUM_REDIS_PORT=6379 \
  trungdlp/podium:latest start
Enter fullscreen mode Exit fullscreen mode

Verify the service:

curl http://localhost:8880/healthcheck
Enter fullscreen mode Exit fullscreen mode 
WORKING

Submit two equal scores:

curl --request
…
Enter fullscreen mode Exit fullscreen mode
View on GitHub

Here is how the design works, why hash tags are necessary, and where the
scaling boundary really is.

One logical leaderboard, five physical keys

A minimal leaderboard can live in one sorted set:

weekly-global -> [(alice, 120), (bob, 100), ...]
Enter fullscreen mode Exit fullscreen mode

Podium guarantees that the first member to reach a tied score ranks higher. To
support that rule in both ascending and descending order, plus member
expiration, a logical leaderboard uses five keys:

scores       sorted set   descending ranking index
scores-asc   sorted set   ascending ranking index
members      hash         public ID -> encoded internal member
sequence     string       next tie-break sequence
ttl          sorted set   member expiration timestamps
Enter fullscreen mode Exit fullscreen mode

A write may update the score indexes, mapping, and sequence in one Lua script.
A delete may need to clean both indexes, the mapping, and TTL metadata.

On standalone Redis, those keys can have any names. On Redis Cluster, their
names determine whether the operation is even legal.

The cross-slot problem

Redis Cluster divides its keyspace into 16,384 hash slots. By default, each key
name is hashed independently.

Imagine these keys land on different primaries:

weekly-global:scores      -> node A
weekly-global:members     -> node B
weekly-global:sequence    -> node C
Enter fullscreen mode Exit fullscreen mode

A Lua script cannot atomically update arbitrary keys across those nodes. Redis
will reject the operation with a CROSSSLOT error.

The fix is a Redis Cluster hash tag. If a key contains {...}, only the text
inside the braces is used to choose the slot.

Podium first URL-safe-base64-encodes the leaderboard ID and then constructs keys
like these:

podium:{b<encoded-leaderboard-id>}:scores
podium:{b<encoded-leaderboard-id>}:scores-asc
podium:{b<encoded-leaderboard-id>}:members
podium:{b<encoded-leaderboard-id>}:sequence
podium:{b<encoded-leaderboard-id>}:ttl
Enter fullscreen mode Exit fullscreen mode

Every key contains the same hash tag, so every key for one leaderboard lands in
the same slot.

Encoding the external ID matters. A name containing braces or other surprising
characters cannot accidentally control the hash tag syntax. The key-generation
test specifically uses a brace-containing leaderboard name to guard that
boundary.

The implementation is small:
redis_keys.go.
Small key-building functions deserve serious tests because changing one can
move production data.

Co-location restores atomicity

Once all related keys share a slot, one Redis primary can execute the
leaderboard's Lua script atomically.

For a score update, that means:

allocate tie-break sequence
remove stale internal members
write descending index
write ascending index
update public-ID mapping
return final rank
Enter fullscreen mode Exit fullscreen mode

No other command can observe the operation halfway through.

Co-location also keeps whole-leaderboard expiration and deletion predictable.
All five keys can receive the same expiration or be removed together without a
cross-slot operation.

This is the useful Redis Cluster rule:

State that must change atomically must share a slot.

But co-location has a direct consequence.

One leaderboard still belongs to one primary

All five keys for weekly-global share a slot. One primary owns that slot.

Therefore:

  • every write to weekly-global reaches that primary;
  • every read of its sorted sets reaches that primary;
  • its memory lives on that primary;
  • its Lua scripts consume time on that primary.

Adding Redis Cluster nodes does not stripe that leaderboard across them.

What Redis Cluster does distribute is different leaderboard IDs:

weekly-global  -> slot 2,140  -> primary A
daily-eu       -> slot 9,811  -> primary B
clan-42        -> slot 14,002 -> primary C
Enter fullscreen mode Exit fullscreen mode

With enough independent boards and well-distributed names, the fleet spreads
across the cluster. This is a good match for games with regional, seasonal,
event, clan, or per-mode leaderboards.

It is not a solution for one board whose traffic or member count exceeds a
single Redis primary.

Partition exceptional boards at the application level

If one leaderboard is too hot or too large for one node, it needs an
application-level partitioning strategy.

Possible boundaries include:

  • region;
  • game mode;
  • season;
  • skill tier;
  • tournament bracket.

For example, replace one global board with:

season-12:region-na
season-12:region-eu
season-12:region-apac
Enter fullscreen mode Exit fullscreen mode

This distributes writes, but global reads now require a merge. The application
must fetch candidates from each partition and combine them with the same score
and tie-break semantics.

That is not a free optimization. Partition only when measurement shows that
the exceptional leaderboard needs it.

Multi-leaderboard writes cross a different boundary

Sometimes one player update must fan out to many independent boards:

global
region-eu
mode-solo
season-12
clan-42
Enter fullscreen mode Exit fullscreen mode

Those boards intentionally occupy different slots. They cannot be one atomic
Redis script.

Podium handles this at the API layer with an errgroup and caps concurrency at
32 workers per request:

group, groupCtx := errgroup.WithContext(ctx)
group.SetLimit(32)

for _, leaderboardID := range leaderboardIDs {
    leaderboardID := leaderboardID
    group.Go(func() error {
        return updateOneLeaderboard(groupCtx, leaderboardID)
    })
}

return group.Wait()
Enter fullscreen mode Exit fullscreen mode

The cap prevents one request containing hundreds of boards from creating
unbounded goroutines and flooding Redis.

It also exposes an important semantic difference:

  • A multi-member write to one leaderboard can be atomic inside one slot.
  • A one-member write to many leaderboards is concurrent, but not globally atomic across slots.

If your product requires all-or-nothing state across leaderboards, you need a
higher-level workflow with idempotency, durable intent, retries, and
compensation. Redis Cluster cannot manufacture a cross-shard transaction for
you.

Test the topology you claim to support

A mocked Redis client cannot prove that keys share slots. A standalone Redis
instance cannot produce CROSSSLOT.

Podium's CI runs integration tests against a real cluster with at least three
primaries and verifies coverage of all 16,384 slots. The same tie-break,
expiration, and deletion paths then run through the cluster client.

The topology check lives in
redis_cluster_test.go.

This catches a class of failure that unit tests naturally miss:

  • malformed cluster creation;
  • incomplete slot coverage;
  • accidental cross-slot key changes;
  • code paths that work only with the standalone client.

If Redis Cluster is in your production architecture diagram, a real cluster
belongs somewhere in your test matrix.

A more honest scaling checklist

Before calling a Redis-backed system "horizontally scalable," ask:

  1. What is the unit of sharding: tenant, leaderboard, member, or key?
  2. Which keys must be co-located for atomic operations?
  3. What is the hottest possible unit?
  4. Can that unit fit on one primary?
  5. Which workflows cross slots?
  6. Are those workflows atomic, eventually consistent, or compensating?
  7. Does CI exercise a real cluster topology?

For Podium, the unit is the leaderboard.

That choice preserves simple, atomic ranking operations and distributes fleets
of independent boards well. It also means one exceptional board has a
single-node ceiling.

Architecture gets easier to reason about when we state both halves.

If you are designing with Redis Cluster, what is your real unit of sharding?
The answer is often more useful than the number of nodes. Share your design or
questions in the comments.

Explore Podium on GitHub

Top comments (0)