Correctness Has a Price: We Benchmarked Fair Leaderboards

Engineering posts often end with:

The new design is correct, scalable, and fast.

Fast compared with what?

When we changed Podium so tied players
rank by arrival time instead of player ID, we added:

• a Lua script;
• a per-leaderboard sequence;
• a public-ID mapping;
• a second sorted set for ascending order.

That design is fairer. It is also impossible for it to be free.

So we built two benchmark layers: direct Redis strategy benchmarks to isolate
the data-model cost, and end-to-end HTTP benchmarks to show what users actually
experience.

We are publishing the results, including the regression, because performance
claims are useful only when readers can inspect the workload and reproduce the
measurement.

TeneficGames / podium

High-performance, Redis-backed leaderboards for games and competitive applications.

Podium

High-performance, Redis-backed leaderboards for games and competitive applications.

Podium provides ready-to-run HTTP and gRPC APIs for scores, ranks, seasons, and player-relative views. It is designed for backend teams operating large fleets of independent leaderboards without provisioning each leaderboard in advance.

• Fair, deterministic ordering when scores are equal.
• Single and bulk score updates, including multi-leaderboard fan-out.
• Standalone Redis and real Redis Cluster integration coverage.
• Deploy one multi-architecture OCI image with Docker, containerd, Kubernetes or another OCI-compatible runtime.

Quickstart · Performance · API · Documentation · Helm chart · Docker Hub · GHCR

Quickstart

Start Redis 8.2 and the latest stable Podium image:

docker network create podium
docker run --detach --name podium-redis --network podium redis:8.2-alpine

docker run --detach --rm --name podium \
  --network podium \
  --publish 8880:8880 \
  --publish 8881:8881 \
  --env PODIUM_REDIS_HOST=podium-redis \
  --env PODIUM_REDIS_PORT=6379 \
  trungdlp/podium:latest start

Verify the service:

curl http://localhost:8880/healthcheck

WORKING

Submit two equal scores:

curl --request

…

View on GitHub

The results were reassuring in some places and uncomfortable in one. Both were
useful.

Start with the correctness contract

The benchmark cannot tell us whether a faster implementation is acceptable.
Tests do that first.

Every candidate tie-break strategy had to satisfy the same behavior:

alice reaches 100
bob reaches 100
top => alice, bob

alice submits 100 again
top => alice, bob

alice changes to 90, then returns to 100
top => bob, alice

The production strategy additionally supports both ascending and descending
boards, atomic increments, TTL cleanup, and concurrent arrivals.

This ordering matters. A benchmark suite without a behavior gate can crown an
implementation that solves an easier problem.

Compare representations, not just commits

We tested four strategies:

1. Plain sorted-set baseline

ZADD scores <score> <public-id>

This is the smallest design. It also lets Redis break equal scores by member
name, so it is a performance baseline rather than a valid solution.

2. Production encoded-member design

The production model keeps:

descending score index
ascending score index
sequence counter
public ID -> internal token hash

An atomic script assigns a fixed-width sequence prefix whenever a member
reaches a new score.

3. Single-index encoded-member design

This alternative also uses a sequence-prefixed internal member and a lookup
hash, but keeps only one score index. It helps separate the general cost of
encoding members from the cost of supporting both sort directions.

4. Packed composite score

This version combines the score and tie-break sequence into the sorted-set
score:

composite = score * sequence_capacity + remaining_sequence

It uses less metadata, but the safe score range and sequence capacity must fit
inside Redis's exact integer range for double-precision scores. The benchmark
implementation rejects values outside those explicit limits.

An alternative is only useful if its constraints appear next to its speed.

The full harness is in
tiebreak_benchmark_test.go.

Benchmark operations separately

A single "leaderboard ops/sec" number hides too much.

The direct Redis suite measures five distinct workloads:

insert a member and return rank
change an existing score and return rank
submit an unchanged score and return rank
read one rank
read the top 50

This division exposed an important optimization: unchanged score submissions
preserve the existing tie-break token. They should not pay the full cost of
removing and recreating a member.

It also distinguishes write amplification from read overhead. A design may
make writes substantially heavier while leaving the dominant read paths nearly
unchanged.

Measure Redis memory, not only Go allocations

go test -benchmem reports allocations in the benchmark process. It does not
tell us how much memory Redis used for its sorted sets and hashes.

After the insert benchmark, the harness calls Redis MEMORY USAGE for every
key owned by the strategy, sums the result, and reports:

redis-B/member

In simplified Go:

func memoryPerMember(ctx context.Context, keys []string, members int) float64 {
    var total int64
    for _, key := range keys {
        total += redisMemoryUsage(ctx, key)
    }
    return float64(total) / float64(members)
}

This metric made the largest tradeoff impossible to miss. In our measured
setup:

plain baseline:       about 99 Redis bytes/member
production tie-break: about 287 Redis bytes/member

The exact number will change with member length, Redis version, allocator,
encoding thresholds, and dataset cardinality. The ratio still tells us that
fair ordering with two indexes has a meaningful capacity cost.

What the direct comparison showed

Against the plain sorted-set baseline, the production deterministic tie-break
added approximately:

Operation	Latency overhead
Insert and return rank	11%
Change score and return rank	17%
Submit an unchanged score	5%
Read one rank	1%
Read the top 50	3%

The write cost is unsurprising: the script reads metadata and may update two
sorted sets, a hash, and a sequence.

The small rank and top-50 overhead is more interesting. Encoding the internal
member does not fundamentally change sorted-set rank lookup, and top-page reads
only need to strip the fixed-width prefix before returning public IDs.

The memory result is the sharper warning. If your capacity model assumed one
sorted-set entry per member, this correctness feature changes it materially.

Then measure through the real API

Microbenchmarks answer:

What did this data model cost?

Users care about:

How long does the operation take through the service?

Podium's end-to-end suite sends sequential HTTP requests through the real API
to a local Redis instance. Each benchmark invocation uses an isolated
leaderboard ID and removes its data afterward. Setup and cleanup remain outside
the timed section.

Five-run medians recorded on July 30, 2026, with Go 1.26.5, Redis 8.2, and an
Apple M4 Pro included:

End-to-end HTTP operation	Median	Allocated bytes
Set one member score	305 µs	6.7 KB
Set 50 member scores	657 µs	36.1 KB
Get one member rank	282 µs	5.4 KB
Get a top-members page	441 µs	9.5 KB
Update one member across 100 boards	3.58 ms	78.3 KB
Get 501 members	3.17 ms	222.6 KB

These are local, sequential measurements, not throughput limits, production
SLAs, or Redis Cluster capacity results.

That sentence should accompany every benchmark table.

The uncomfortable result was the most valuable

The 501-member bulk lookup was approximately 78% slower than the
pre-tie-break implementation.

It would be easy to hide that result behind the faster common operations.
Instead, it is documented as an optimization target.

Why did this path stand out?

The deterministic model cannot query ranks using public IDs directly. It first
resolves each public ID to an internal token, then reads the appropriate score,
rank, and optional TTL. The production Lua script returns aligned triples for
every requested member:

score, rank, ttl
score, rank, ttl
...

That preserves missing-member positions and atomic read semantics, but a
501-member request makes the extra work visible.

This is exactly what benchmarks are for: finding the path where an acceptable
general tradeoff becomes a specific bottleneck.

Reproducibility is a feature

The repository exposes the benchmark workflow as Make targets:

make bench-redis
make bench-tiebreak
make bench-redis-kill

For the HTTP suite:

make bench-redis
make bench-podium-app
make bench-run
make bench-podium-app-kill
make bench-redis-kill

The run count is configurable, and the documentation recommends comparing
medians from multiple repetitions, running old and new builds back-to-back on
the same idle machine, and resetting Redis between implementations.

See the
benchmark guide
for the complete procedure.

What we learned

Our benchmarking checklist now looks like this:

1. Define the behavior every candidate must preserve.
2. Keep an intentionally simpler baseline.
3. Benchmark operations separately.
4. Measure memory in the server, not only allocations in the client.
5. Add an end-to-end layer through the real protocol.
6. Use isolated data and exclude setup and cleanup from timing.
7. Publish the machine, versions, repetition count, and limitations.
8. Keep regressions visible, even when the overall design is worth shipping.

The goal is not to prove that a design is "fast."

The goal is to know what you bought, what you paid, and which bill needs
attention next.

What correctness feature in your system has the most surprising performance
cost? Share the result or your benchmarking questions in the comments.

Explore Podium on GitHub