DEV Community: Trung Duong

Correctness Has a Price: We Benchmarked Fair Leaderboards

Trung Duong — Fri, 31 Jul 2026 06:44:23 +0000

Engineering posts often end with:

The new design is correct, scalable, and fast.

Fast compared with what?

When we changed Podium so tied players
rank by arrival time instead of player ID, we added:

a Lua script;
a per-leaderboard sequence;
a public-ID mapping;
a second sorted set for ascending order.

That design is fairer. It is also impossible for it to be free.

So we built two benchmark layers: direct Redis strategy benchmarks to isolate
the data-model cost, and end-to-end HTTP benchmarks to show what users actually
experience.

We are publishing the results, including the regression, because performance
claims are useful only when readers can inspect the workload and reproduce the
measurement.

TeneficGames / podium

High-performance, Redis-backed leaderboards for games and competitive applications.

Podium

High-performance, Redis-backed leaderboards for games and competitive applications.

Podium provides ready-to-run HTTP and gRPC APIs for scores, ranks, seasons, and player-relative views. It is designed for backend teams operating large fleets of independent leaderboards without provisioning each leaderboard in advance.

Fair, deterministic ordering when scores are equal.
Single and bulk score updates, including multi-leaderboard fan-out.
Standalone Redis and real Redis Cluster integration coverage.
Deploy one multi-architecture OCI image with Docker, containerd, Kubernetes or another OCI-compatible runtime.

Quickstart · Performance · API · Documentation · Helm chart · Docker Hub · GHCR

Quickstart

Start Redis 8.2 and the latest stable Podium image:

docker network create podium
docker run --detach --name podium-redis --network podium redis:8.2-alpine

docker run --detach --rm --name podium \
  --network podium \
  --publish 8880:8880 \
  --publish 8881:8881 \
  --env PODIUM_REDIS_HOST=podium-redis \
  --env PODIUM_REDIS_PORT=6379 \
  trungdlp/podium:latest start

Verify the service:

curl http://localhost:8880/healthcheck

WORKING

Submit two equal scores:

curl --request

…

View on GitHub

The results were reassuring in some places and uncomfortable in one. Both were
useful.

Start with the correctness contract

The benchmark cannot tell us whether a faster implementation is acceptable.
Tests do that first.

Every candidate tie-break strategy had to satisfy the same behavior:

alice reaches 100
bob reaches 100
top => alice, bob

alice submits 100 again
top => alice, bob

alice changes to 90, then returns to 100
top => bob, alice

The production strategy additionally supports both ascending and descending
boards, atomic increments, TTL cleanup, and concurrent arrivals.

This ordering matters. A benchmark suite without a behavior gate can crown an
implementation that solves an easier problem.

Compare representations, not just commits

We tested four strategies:

1. Plain sorted-set baseline

ZADD scores <score> <public-id>

This is the smallest design. It also lets Redis break equal scores by member
name, so it is a performance baseline rather than a valid solution.

2. Production encoded-member design

The production model keeps:

descending score index
ascending score index
sequence counter
public ID -> internal token hash

An atomic script assigns a fixed-width sequence prefix whenever a member
reaches a new score.

3. Single-index encoded-member design

This alternative also uses a sequence-prefixed internal member and a lookup
hash, but keeps only one score index. It helps separate the general cost of
encoding members from the cost of supporting both sort directions.

4. Packed composite score

This version combines the score and tie-break sequence into the sorted-set
score:

composite = score * sequence_capacity + remaining_sequence

It uses less metadata, but the safe score range and sequence capacity must fit
inside Redis's exact integer range for double-precision scores. The benchmark
implementation rejects values outside those explicit limits.

An alternative is only useful if its constraints appear next to its speed.

The full harness is in
tiebreak_benchmark_test.go.

Benchmark operations separately

A single "leaderboard ops/sec" number hides too much.

The direct Redis suite measures five distinct workloads:

insert a member and return rank
change an existing score and return rank
submit an unchanged score and return rank
read one rank
read the top 50

This division exposed an important optimization: unchanged score submissions
preserve the existing tie-break token. They should not pay the full cost of
removing and recreating a member.

It also distinguishes write amplification from read overhead. A design may
make writes substantially heavier while leaving the dominant read paths nearly
unchanged.

Measure Redis memory, not only Go allocations

go test -benchmem reports allocations in the benchmark process. It does not
tell us how much memory Redis used for its sorted sets and hashes.

After the insert benchmark, the harness calls Redis MEMORY USAGE for every
key owned by the strategy, sums the result, and reports:

redis-B/member

In simplified Go:

func memoryPerMember(ctx context.Context, keys []string, members int) float64 {
    var total int64
    for _, key := range keys {
        total += redisMemoryUsage(ctx, key)
    }
    return float64(total) / float64(members)
}

This metric made the largest tradeoff impossible to miss. In our measured
setup:

plain baseline:       about 99 Redis bytes/member
production tie-break: about 287 Redis bytes/member

The exact number will change with member length, Redis version, allocator,
encoding thresholds, and dataset cardinality. The ratio still tells us that
fair ordering with two indexes has a meaningful capacity cost.

What the direct comparison showed

Against the plain sorted-set baseline, the production deterministic tie-break
added approximately:

Operation	Latency overhead
Insert and return rank	11%
Change score and return rank	17%
Submit an unchanged score	5%
Read one rank	1%
Read the top 50	3%

The write cost is unsurprising: the script reads metadata and may update two
sorted sets, a hash, and a sequence.

The small rank and top-50 overhead is more interesting. Encoding the internal
member does not fundamentally change sorted-set rank lookup, and top-page reads
only need to strip the fixed-width prefix before returning public IDs.

The memory result is the sharper warning. If your capacity model assumed one
sorted-set entry per member, this correctness feature changes it materially.

Then measure through the real API

Microbenchmarks answer:

What did this data model cost?

Users care about:

How long does the operation take through the service?

Podium's end-to-end suite sends sequential HTTP requests through the real API
to a local Redis instance. Each benchmark invocation uses an isolated
leaderboard ID and removes its data afterward. Setup and cleanup remain outside
the timed section.

Five-run medians recorded on July 30, 2026, with Go 1.26.5, Redis 8.2, and an
Apple M4 Pro included:

End-to-end HTTP operation	Median	Allocated bytes
Set one member score	305 µs	6.7 KB
Set 50 member scores	657 µs	36.1 KB
Get one member rank	282 µs	5.4 KB
Get a top-members page	441 µs	9.5 KB
Update one member across 100 boards	3.58 ms	78.3 KB
Get 501 members	3.17 ms	222.6 KB

These are local, sequential measurements, not throughput limits, production
SLAs, or Redis Cluster capacity results.

That sentence should accompany every benchmark table.

The uncomfortable result was the most valuable

The 501-member bulk lookup was approximately 78% slower than the
pre-tie-break implementation.

It would be easy to hide that result behind the faster common operations.
Instead, it is documented as an optimization target.

Why did this path stand out?

The deterministic model cannot query ranks using public IDs directly. It first
resolves each public ID to an internal token, then reads the appropriate score,
rank, and optional TTL. The production Lua script returns aligned triples for
every requested member:

score, rank, ttl
score, rank, ttl
...

That preserves missing-member positions and atomic read semantics, but a
501-member request makes the extra work visible.

This is exactly what benchmarks are for: finding the path where an acceptable
general tradeoff becomes a specific bottleneck.

Reproducibility is a feature

The repository exposes the benchmark workflow as Make targets:

make bench-redis
make bench-tiebreak
make bench-redis-kill

For the HTTP suite:

make bench-redis
make bench-podium-app
make bench-run
make bench-podium-app-kill
make bench-redis-kill

The run count is configurable, and the documentation recommends comparing
medians from multiple repetitions, running old and new builds back-to-back on
the same idle machine, and resetting Redis between implementations.

See the
benchmark guide
for the complete procedure.

What we learned

Our benchmarking checklist now looks like this:

Define the behavior every candidate must preserve.
Keep an intentionally simpler baseline.
Benchmark operations separately.
Measure memory in the server, not only allocations in the client.
Add an end-to-end layer through the real protocol.
Use isolated data and exclude setup and cleanup from timing.
Publish the machine, versions, repetition count, and limitations.
Keep regressions visible, even when the overall design is worth shipping.

The goal is not to prove that a design is "fast."

The goal is to know what you bought, what you paid, and which bill needs
attention next.

What correctness feature in your system has the most surprising performance
cost? Share the result or your benchmarking questions in the comments.

Explore Podium on GitHub

Redis Cluster Won't Shard Your Hot Leaderboard

Trung Duong — Fri, 31 Jul 2026 06:43:59 +0000

"We use Redis Cluster" can mean two very different things:

Our dataset is distributed across Redis nodes.
Every individual data structure is distributed across Redis nodes.

The first can be true while the second is false.

That distinction matters for leaderboards.

In Podium, each leaderboard uses
several Redis keys and atomic Lua scripts. Redis Cluster helps us scale a large
fleet of independent leaderboards, but it cannot split one giant sorted set
across primaries.

We are sharing this architecture because "Redis Cluster scales horizontally"
is true only after you define what the system actually shards.

TeneficGames / podium

High-performance, Redis-backed leaderboards for games and competitive applications.

Podium

High-performance, Redis-backed leaderboards for games and competitive applications.

Fair, deterministic ordering when scores are equal.
Single and bulk score updates, including multi-leaderboard fan-out.
Standalone Redis and real Redis Cluster integration coverage.
Deploy one multi-architecture OCI image with Docker, containerd, Kubernetes or another OCI-compatible runtime.

Quickstart · Performance · API · Documentation · Helm chart · Docker Hub · GHCR

Quickstart

Start Redis 8.2 and the latest stable Podium image:

docker network create podium
docker run --detach --name podium-redis --network podium redis:8.2-alpine

docker run --detach --rm --name podium \
  --network podium \
  --publish 8880:8880 \
  --publish 8881:8881 \
  --env PODIUM_REDIS_HOST=podium-redis \
  --env PODIUM_REDIS_PORT=6379 \
  trungdlp/podium:latest start

Verify the service:

curl http://localhost:8880/healthcheck

WORKING

Submit two equal scores:

curl --request

…

View on GitHub

Here is how the design works, why hash tags are necessary, and where the
scaling boundary really is.

One logical leaderboard, five physical keys

A minimal leaderboard can live in one sorted set:

weekly-global -> [(alice, 120), (bob, 100), ...]

Podium guarantees that the first member to reach a tied score ranks higher. To
support that rule in both ascending and descending order, plus member
expiration, a logical leaderboard uses five keys:

scores       sorted set   descending ranking index
scores-asc   sorted set   ascending ranking index
members      hash         public ID -> encoded internal member
sequence     string       next tie-break sequence
ttl          sorted set   member expiration timestamps

A write may update the score indexes, mapping, and sequence in one Lua script.
A delete may need to clean both indexes, the mapping, and TTL metadata.

On standalone Redis, those keys can have any names. On Redis Cluster, their
names determine whether the operation is even legal.

The cross-slot problem

Redis Cluster divides its keyspace into 16,384 hash slots. By default, each key
name is hashed independently.

Imagine these keys land on different primaries:

weekly-global:scores      -> node A
weekly-global:members     -> node B
weekly-global:sequence    -> node C

A Lua script cannot atomically update arbitrary keys across those nodes. Redis
will reject the operation with a CROSSSLOT error.

The fix is a Redis Cluster hash tag. If a key contains {...}, only the text
inside the braces is used to choose the slot.

Podium first URL-safe-base64-encodes the leaderboard ID and then constructs keys
like these:

podium:{b<encoded-leaderboard-id>}:scores
podium:{b<encoded-leaderboard-id>}:scores-asc
podium:{b<encoded-leaderboard-id>}:members
podium:{b<encoded-leaderboard-id>}:sequence
podium:{b<encoded-leaderboard-id>}:ttl

Every key contains the same hash tag, so every key for one leaderboard lands in
the same slot.

Encoding the external ID matters. A name containing braces or other surprising
characters cannot accidentally control the hash tag syntax. The key-generation
test specifically uses a brace-containing leaderboard name to guard that
boundary.

The implementation is small:
redis_keys.go.
Small key-building functions deserve serious tests because changing one can
move production data.

Co-location restores atomicity

Once all related keys share a slot, one Redis primary can execute the
leaderboard's Lua script atomically.

For a score update, that means:

allocate tie-break sequence
remove stale internal members
write descending index
write ascending index
update public-ID mapping
return final rank

No other command can observe the operation halfway through.

Co-location also keeps whole-leaderboard expiration and deletion predictable.
All five keys can receive the same expiration or be removed together without a
cross-slot operation.

This is the useful Redis Cluster rule:

State that must change atomically must share a slot.

But co-location has a direct consequence.

One leaderboard still belongs to one primary

All five keys for weekly-global share a slot. One primary owns that slot.

Therefore:

every write to weekly-global reaches that primary;
every read of its sorted sets reaches that primary;
its memory lives on that primary;
its Lua scripts consume time on that primary.

Adding Redis Cluster nodes does not stripe that leaderboard across them.

What Redis Cluster does distribute is different leaderboard IDs:

weekly-global  -> slot 2,140  -> primary A
daily-eu       -> slot 9,811  -> primary B
clan-42        -> slot 14,002 -> primary C

With enough independent boards and well-distributed names, the fleet spreads
across the cluster. This is a good match for games with regional, seasonal,
event, clan, or per-mode leaderboards.

It is not a solution for one board whose traffic or member count exceeds a
single Redis primary.

Partition exceptional boards at the application level

If one leaderboard is too hot or too large for one node, it needs an
application-level partitioning strategy.

Possible boundaries include:

region;
game mode;
season;
skill tier;
tournament bracket.

For example, replace one global board with:

season-12:region-na
season-12:region-eu
season-12:region-apac

This distributes writes, but global reads now require a merge. The application
must fetch candidates from each partition and combine them with the same score
and tie-break semantics.

That is not a free optimization. Partition only when measurement shows that
the exceptional leaderboard needs it.

Multi-leaderboard writes cross a different boundary

Sometimes one player update must fan out to many independent boards:

global
region-eu
mode-solo
season-12
clan-42

Those boards intentionally occupy different slots. They cannot be one atomic
Redis script.

Podium handles this at the API layer with an errgroup and caps concurrency at
32 workers per request:

group, groupCtx := errgroup.WithContext(ctx)
group.SetLimit(32)

for _, leaderboardID := range leaderboardIDs {
    leaderboardID := leaderboardID
    group.Go(func() error {
        return updateOneLeaderboard(groupCtx, leaderboardID)
    })
}

return group.Wait()

The cap prevents one request containing hundreds of boards from creating
unbounded goroutines and flooding Redis.

It also exposes an important semantic difference:

A multi-member write to one leaderboard can be atomic inside one slot.
A one-member write to many leaderboards is concurrent, but not globally atomic across slots.

If your product requires all-or-nothing state across leaderboards, you need a
higher-level workflow with idempotency, durable intent, retries, and
compensation. Redis Cluster cannot manufacture a cross-shard transaction for
you.

Test the topology you claim to support

A mocked Redis client cannot prove that keys share slots. A standalone Redis
instance cannot produce CROSSSLOT.

Podium's CI runs integration tests against a real cluster with at least three
primaries and verifies coverage of all 16,384 slots. The same tie-break,
expiration, and deletion paths then run through the cluster client.

The topology check lives in
redis_cluster_test.go.

This catches a class of failure that unit tests naturally miss:

malformed cluster creation;
incomplete slot coverage;
accidental cross-slot key changes;
code paths that work only with the standalone client.

If Redis Cluster is in your production architecture diagram, a real cluster
belongs somewhere in your test matrix.

A more honest scaling checklist

Before calling a Redis-backed system "horizontally scalable," ask:

What is the unit of sharding: tenant, leaderboard, member, or key?
Which keys must be co-located for atomic operations?
What is the hottest possible unit?
Can that unit fit on one primary?
Which workflows cross slots?
Are those workflows atomic, eventually consistent, or compensating?
Does CI exercise a real cluster topology?

For Podium, the unit is the leaderboard.

That choice preserves simple, atomic ranking operations and distributes fleets
of independent boards well. It also means one exceptional board has a
single-node ceiling.

Architecture gets easier to reason about when we state both halves.

If you are designing with Redis Cluster, what is your real unit of sharding?
The answer is often more useful than the number of nodes. Share your design or
questions in the comments.

Explore Podium on GitHub

Your Redis Leaderboard Is Probably Breaking Ties Wrong

Trung Duong — Fri, 31 Jul 2026 06:41:29 +0000

A leaderboard looks like a one-command problem:

ZADD weekly 100 alice
ZADD weekly 100 bob
ZREVRANGE weekly 0 -1 WITHSCORES

While building
Podium, an open-source Redis-backed
leaderboard service, we discovered that the difficult part begins when two
players have the same score. We are sharing the design because this edge case
can silently turn player IDs into ranking rules.

TeneficGames / podium

High-performance, Redis-backed leaderboards for games and competitive applications.

Podium

High-performance, Redis-backed leaderboards for games and competitive applications.

Fair, deterministic ordering when scores are equal.
Single and bulk score updates, including multi-leaderboard fan-out.
Standalone Redis and real Redis Cluster integration coverage.
Deploy one multi-architecture OCI image with Docker, containerd, Kubernetes or another OCI-compatible runtime.

Quickstart · Performance · API · Documentation · Helm chart · Docker Hub · GHCR

Quickstart

Start Redis 8.2 and the latest stable Podium image:

docker network create podium
docker run --detach --name podium-redis --network podium redis:8.2-alpine

docker run --detach --rm --name podium \
  --network podium \
  --publish 8880:8880 \
  --publish 8881:8881 \
  --env PODIUM_REDIS_HOST=podium-redis \
  --env PODIUM_REDIS_PORT=6379 \
  trungdlp/podium:latest start

Verify the service:

curl http://localhost:8880/healthcheck

WORKING

Submit two equal scores:

curl --request

…

View on GitHub

Both players have 100 points. Alice arrived first, so most game designers would
expect:

1. alice: 100
2. bob:   100

But that is not what the data model says.

Redis sorted sets order members with equal scores lexicographically. With a
reverse range, that secondary ordering is reversed too. Your "fair" tie may
therefore be decided by a player ID.

That is deterministic, but it is not meaningful.

We wanted a rule players could understand:

When scores are equal, the player who reached the current score first ranks
higher.

It sounds simple. Implementing it correctly under concurrent writes was not.

Why the obvious fixes fail

The first idea is usually a timestamp:

tie_break = current_time_in_milliseconds

This creates three problems.

First, two requests can land in the same millisecond. More precision makes that
less likely, but never makes it impossible.

Second, application clocks are not a reliable global ordering mechanism. If
multiple API replicas write to the same Redis instance, clock skew can reorder
arrivals.

Third, the score update and timestamp allocation are separate operations unless
you add a transaction or script. A process can fail between them and leave
partial state.

Another tempting approach is to pack the score and timestamp into one floating
point number:

composite = score * scale + tie_break

Redis sorted-set scores are IEEE 754 doubles. Integers are represented exactly
only up to 2^53 - 1. Packing two independently growing values into that space
quietly reduces the safe range of both. It can work when your limits are small
and explicit, but it is a dangerous default for a general leaderboard service.

We needed an ordering token that was:

unique under concurrency;
independent of wall clocks;
assigned only when a player reaches a new score;
committed atomically with every affected index.

Redis itself was the right place to allocate it.

Use a Redis sequence, not time

Podium stores a per-leaderboard sequence initialized to the largest signed
64-bit integer:

9223372036854775807

When a member reaches a new score, an atomic Lua script decrements that value
and formats it as a fixed-width 19-digit prefix.

Conceptually:

redis.call("DECR", sequence_key)
local sequence = redis.call("GET", sequence_key)
local token = left_pad_to_19_digits(sequence) .. public_member_id

The first arrival receives a larger token than the second:

alice -> 9223372036854775806alice
bob   -> 9223372036854775805bob

For a descending leaderboard, Redis's reverse lexicographic ordering now does
exactly what we need. At equal scores, Alice's larger internal token comes
first.

The public ID has not changed. The encoded value is an internal member name,
and a Redis hash maps each public ID to its current token:

public member ID -> current internal token

This mapping is essential when a score changes. We remove the old internal
member, allocate a new sequence, add the replacement, and update the mapping in
one script.

You can see the production implementation in
tiebreak.go.

Idempotency is part of fairness

Suppose Alice reaches 100 first and Bob reaches 100 second:

1. alice: 100
2. bob:   100

Alice's client retries the same request because the network response was lost.
Should she get a new arrival token?

No. A retry must not change rank.

Before allocating a sequence, the Lua script looks up Alice's current internal
token and reads its score. If the submitted score is unchanged, it preserves
the token.

That gives the rule precise semantics:

Submitting the same score again keeps the current position.
Leaving a score and returning later creates a new arrival.
Removing and re-adding a member creates a new arrival.
Incrementing by zero internally preserves the arrival token.

These are not edge cases to decide after launch. They are part of the product
contract.

Ascending leaderboards contain a trap

Descending leaderboards are common: more points is better.

But some competitions are ascending: less time, fewer moves, or fewer strokes
is better.

Redis uses normal lexicographic order for ZRANK and reverse lexicographic
order for ZREVRANK. One encoded member cannot put the earlier arrival first in
both directions.

Podium therefore maintains two score indexes:

:scores      -> descending tie order
:scores-asc  -> ascending tie order

The descending token uses the decreasing sequence directly. The ascending
token inverts each sequence digit:

9 - digit

An earlier, larger sequence becomes a lexicographically smaller prefix in the
ascending index. The same player-facing rule now holds in both directions.

This second index costs memory, but it avoids conditional definitions of
"first" that change with leaderboard direction.

Atomicity is the real feature

One score update can touch four pieces of state:

descending sorted set
ascending sorted set
public-ID-to-token hash
sequence counter

Doing that with four client round trips creates four places for races and
partial failure.

Podium runs the operation as one Redis Lua script. Redis executes scripts
atomically, so concurrent arrivals receive distinct sequence values and readers
never observe a half-updated member.

The script also returns the resulting rank. The caller does not need a separate
"write, then read rank" round trip that another request could interleave with.

The concurrency tests launch 128 simultaneous arrivals at the same score and
verify that every member receives a unique rank. Separate tests cover retries,
score changes, removal, re-entry, ascending order, and sequence exhaustion.

The tests are worth reading alongside the implementation:
tiebreak_test.go.

Correctness is not free

The plain sorted-set design is smaller and faster.

In our direct Redis benchmarks, deterministic ordering added approximately:

Operation	Latency overhead
Insert and return rank	11%
Change score and return rank	17%
Submit the same score	5%
Read one rank	1%
Read the top 50	3%

Supporting both directions used about 287 Redis bytes per member, compared with
99 bytes for the plain baseline in the measured setup.

Those numbers are not universal capacity promises. They came from local
benchmarks with a specific Redis version, machine, and dataset. They do make
the tradeoff visible: we spend memory and a modest amount of latency to replace
an arbitrary player-ID tie-break with a stable product rule.

For a game where tied players care who arrived first, that is an easy trade to
explain.

The deeper lesson

Redis gives you a total order. That does not mean it gives you the right
order.

Before shipping a leaderboard, write down the answers to these questions:

What happens when scores tie?
Does retrying the same write change position?
What happens after a player leaves and returns to a score?
Can concurrent writes receive the same tie-break value?
Does the rule behave identically for ascending and descending boards?
Are the score and tie-break committed atomically?

If the answer to the first question is "Redis handles it," your member IDs may
already be deciding matches.

Podium exposes this behavior through HTTP and gRPC. I would love to hear how
your system resolves equal scores and which tie rule your players consider
fair. Share your approach or questions in the comments.

Explore Podium on GitHub

😴 Go Code You Can Trust: Sleep Well After You Commit

Trung Duong — Thu, 13 Mar 2025 03:11:52 +0000

It's Friday afternoon, 4:45 PM. My teammates are already discussing weekend plans in Slack. I'm finishing up a critical piece of our payment processing service. The code is done, tests pass, and I'm about to commit it before heading into the weekend. My finger hovers over the enter key for a moment...

But there's no hesitation, no worry. I commit the code, close my laptop, and join the weekend conversation without a second thought about my code failing in production.

This wasn't always the case. Ten years ago, I'd be checking my phone all weekend, worried about 3 AM calls from our on-call engineer.

What changed? Let me tell you my story of how I learned to write Go code I could trust completely.

Image source: Medium

The Cost of Uncertain Code

My journey with Go started in 2021 when I was working at a rapidly growing startup. We were moving from a monolithic Java application to microservices, and Go seemed like the perfect fit: fast, simple, and with built-in concurrency.

But my early Go code looked like this:

func ProcessPayment(paymentID string) error {
    // Get payment details
    payment, err := db.GetPayment(paymentID)
    if err != nil {
        return err // Which error? What happened?
    }

    // Process the payment
    err = paymentGateway.Process(payment)
    if err != nil {
        return err // Again, what went wrong?
    }

    // Update payment status
    err = db.UpdateStatus(paymentID, "processed")
    if err != nil {
        return err // Did the payment go through? Is it in a bad state?
    }

    return nil
}

Functionally, this code worked. But it haunted me at night because:

Error handling was minimal and provided no context
There was no logging for debugging
No consideration for partial failures
No validation of inputs
No testing for edge cases

The result? Support tickets on Saturday mornings. Mysterious payment failures. Hours spent on debugging sessions. And the constant anxiety that something might be failing silently.

The Path to Trustworthy Code

Everything changed when I joined a team at WF. My mentor there had a simple philosophy: "Write every line of code as if you'll be on vacation when it runs in production."

This mindset shift transformed how I approached Go development. Here's what I learned, and what now lets me sleep well after committing code:

1. Errors Are Your Friends, Not Exceptions

Go's error handling is verbose but powerful when used correctly. The secret is to treat errors as valuable information carriers, not just failure signals:

func ProcessPayment(paymentID string) error {
    // Get payment details
    payment, err := db.GetPayment(paymentID)
    if err != nil {
        return fmt.Errorf("failed to retrieve payment %s: %w", paymentID, err)
    }

    // Validate before proceeding
    if err := validatePayment(payment); err != nil {
        return fmt.Errorf("payment validation failed for %s: %w", paymentID, err)
    }

    // Process the payment with context
    err = paymentGateway.Process(payment)
    if err != nil {
        // Log additional details for debugging
        log.WithFields(log.Fields{
            "payment_id": paymentID,
            "amount":     payment.Amount,
            "currency":   payment.Currency,
        }).Error("Payment processing failed")

        return fmt.Errorf("gateway failed to process payment %s: %w", paymentID, err)
    }

    // Success path is clearly logged too
    log.WithField("payment_id", paymentID).Info("Payment processed successfully")
    return nil
}

The difference? When something goes wrong, I have rich context. The error messages tell a story, making debugging faster and easier.

2. Tests That Give You Confidence

My earlier self wrote tests that proved the code worked in the happy path. My current self writes tests that prove the code won't break in production:

func TestProcessPayment(t *testing.T) {
    // Happy path test
    t.Run("successful payment processing", func(t *testing.T) {
        // Setup and assertions
    })

    // What happens when things go wrong
    t.Run("database unavailable", func(t *testing.T) {
        // Simulate DB failure
    })

    t.Run("payment gateway timeout", func(t *testing.T) {
        // Simulate slow payment gateway
    })

    t.Run("invalid payment data", func(t *testing.T) {
        // Test validation logic
    })

    t.Run("partial failure - payment processed but status update fails", func(t *testing.T) {
        // Test recovery mechanisms
    })
}

A comprehensive test suite is like insurance. It doesn't prevent all problems, but it significantly reduces the likelihood of common failures reaching production.

3. Graceful Degradation

One key insight: not all failures are equal. The best Go code doesn't just handle errors; it gracefully degrades functionality:

func GetUserRecommendations(userID string) ([]Recommendation, error) {
    // Try to get personalized recommendations
    recommendations, err := recommendationService.GetPersonalized(userID)
    if err != nil {
        // Log the error
        log.WithError(err).Warn("Failed to get personalized recommendations, falling back to popular items")

        // Fall back to popular recommendations
        return recommendationService.GetPopular()
    }

    return recommendations, nil
}

This pattern means your code tries its best to provide value, even when some components fail.

4. Monitoring and Observability Built-In

Code I can trust doesn't just work well; it tells me how it's working:

func ProcessOrder(ctx context.Context, order Order) error {
    // Start timing the operation
    start := time.Now()

    // Use defer to ensure metrics are always recorded
    defer func() {
        metrics.ObserveOrderProcessingTime(time.Since(start))
        metrics.IncrementOrdersProcessed()
    }()

    // Trace this operation for distributed tracing
    span, ctx := tracer.StartSpanFromContext(ctx, "process_order")
    defer span.Finish()

    // Add helpful information to the trace
    span.SetTag("order_id", order.ID)
    span.SetTag("customer_id", order.CustomerID)

    // Process the order with the traced context
    if err := orderProcessor.Process(ctx, order); err != nil {
        // Record error in metrics and trace
        metrics.IncrementOrderErrors()
        span.SetTag("error", true)
        span.LogKV("error.message", err.Error())

        return fmt.Errorf("processing order %s failed: %w", order.ID, err)
    }

    return nil
}

When your code has built-in monitoring, you don't need to wonder if it's working correctly in production. You know.

Real-Life Example: The 3 AM Bug That Never Called

Last year, we deployed a new feature to our payment system right before a long holiday weekend. The old me would have been anxious the entire time, but I wasn't worried at all.

Sure enough, something unexpected happened: a third-party API we depended on changed their response format. But instead of a production outage, here's what happened:

Our validation caught the changed format and returned a clear error
The circuit breaker we implemented prevented cascading failures
The system fell back to a secondary processing method
Our monitoring alerted the on-call engineer with the exact issue
Detailed logs showed exactly where and how the format had changed

The on-call engineer fixed it with a simple config change—no emergency, no all-hands debugging session.

The best part? I only found out about this when I read the incident report on Tuesday morning.

How to Write Go Code You Can Trust

After a decade of writing Go, here's my checklist for code I can trust enough to disconnect completely from work:

Handle errors with context: Wrap errors, add information, make debugging easy
Test failure modes: Don't just test success cases; test what happens when things fail
Build in graceful degradation: Design systems that bend rather than break
Make it observable: Logging, metrics, and tracing are not afterthoughts
Validate early and thoroughly: Catch bad inputs before they cause damage
Document assumptions: Clear documentation helps future you and your teammates

Conclusion

The difference between Go code that keeps you up at night and Go code you can trust isn't about clever algorithms or advanced features. It's about care, attention to detail, and a mindset that prepares for the unexpected.

When I interview Go developers now, I don't just look at how well they can write code that works. I look at how they handle the edge cases, how they think about failures, and whether their code would let them enjoy their weekends without worry.

Because in the end, the best code isn't just functionally correct—it's trustworthy enough that you can commit it on Friday afternoon and genuinely disconnect until Monday morning.

And for me, that peace of mind is worth every extra line of error handling, every additional test case, and every minute spent making my code more robust.

So next time you're writing Go code, ask yourself: "Would I sleep well tonight if this ran in production after I left?" If the answer isn't a confident "yes," you have more work to do.

Your future self—possibly on a beach somewhere without laptop access—will thank you.

ByteDance/Sonic: A Lightning-Fast JSON Library for Go

Trung Duong — Tue, 04 Feb 2025 04:20:16 +0000

"In the world of microservices, every millisecond counts. See how TikTok's engineering team revolutionized JSON processing in Go."

Have you ever deployed a Go service and watched your CPU spike when processing thousands of JSON requests? You're not alone. TikTok's engineers faced this exact challenge at a massive scale - billions of requests per day. Their solution? They created Sonic, a JSON library that's changing the game for Go developers.

The JSON Problem Every Go Developer Knows

If you're a Go developer, you've probably used the standard encoding/json package. It works, but let's be real - it's not the fastest kid on the block. If you're building modern web services or APIs, you're working with JSON every day - from REST APIs to configuration files. When TikTok's engineers found their services processing millions of JSON requests per second, even small performance improvements in JSON handling could make a huge difference in server costs and user experience.

Let's look at a common scenario:

// Your typical JSON processing with standard library
import "encoding/json"

type User struct {
    Name  string `json:"name"`
    Posts []Post `json:"posts"`
}

// Processing this for millions of requests...
json.Unmarshal(data, &user)

At small scale, this works fine. But when you're handling TikTok-scale traffic, those milliseconds add up to significant server costs and latency. That's exactly why ByteDance's team decided to tackle this challenge head-on.

The Speed Champion: Meet Sonic

Let's look at some real numbers first. When working with a medium-sized JSON file (about 13KB):

// Processing user profile with posts and metadata
Standard JSON: 106,322 ns/op (nanoseconds per operation)
Sonic:         32,393 ns/op
Memory Usage:
- Standard:    49,136 bytes with 789 allocations
- Sonic:       11,965 bytes with just 4 allocations

Small (400B, 11 keys, 3 layers)
Large (635KB, 10000+ key, 6 layers)

See bench.sh for benchmark codes.

The Secret Sauce: Four Simple Tricks

1. Just-In-Time Compilation (JIT)

Imagine you're a chef. Instead of following a generic recipe every time, you create a special, optimized recipe for each specific dish you make frequently. That's what JIT does!

Regular Go JSON library: Uses the same generic code for all JSON
Sonic: Creates specialized code paths for your specific JSON structures

2. SIMD: Doing More Work at Once

Think of SIMD like having multiple hands to do a task:

Regular way: Sort cards one by one
SIMD way: Sort multiple cards at once

Sonic uses these "multiple hands" (SIMD instructions) to process JSON data in parallel, making everything faster.

3. Smart Memory Usage

Here's a clever trick Sonic uses: When it finds a string in your JSON that doesn't have any special characters, it doesn't make a copy. Instead, it just points to the original string. It's like giving directions instead of drawing a new map!

// Example JSON
{"name": "John", "city": "New York"}

// Standard library: Makes copies of "John" and "New York"
// Sonic: Just references these strings if they're simple

4. Optional Features = Better Speed

Sonic makes some smart choices about what features to make optional:

Doesn't sort map keys by default (saves 10% processing time)
Doesn't escape HTML by default (saves 15% processing time)

You can still turn these features on if you need them, but by making them optional, Sonic stays fast for most common uses.

Design

The design is easy to implement:

Aiming at the function-call overhead cost by the codec dynamic-assembly, JIT tech is used to assemble opcodes (asm) corresponding to the schema at runtime, which is finally cached into the off-heap memory in the form of Golang functions.
For practical scenarios where big data and small data coexist, we use pre-conditional judgment (string size, floating precision, etc.) to combine SIMD with scalar instructions to achieve the best adaptation.
As for insufficiency in compiling optimization of go language, we decided to use C/Clang to write and compile core computational functions, and developed a set of asm2asm tools to translate the fully optimized x86 assembly into plan9 and finally load it into Golang runtime.
Giving the big speed gap between parsing and skipping, the lazy-load mechanism is certainly used in our AST parser, but in a more adaptive and efficient way to reduce the overhead of multiple-key queries.

In detail, we conducted some further optimization:

Since the native-asm functions cannot be inlined in Golang, we found that its cost even exceeded the improvement brought by the optimization of the C compiler. So we reimplemented a set of lightweight function-calls in JIT:
- Global-function-table + static offset for calling instruction
- Pass parameters using registers
Sync.Map was used to cache the codecs at first, but for our quasi-static (read far more than write), fewer elements (usually no more than a few dozen) scenarios, its performance is not optimal, so we reimplement a high-performance and concurrent-safe cache with open-addressing-hash + RCU tech.

From Sonic Design

Real-World Benefits

Let's talk numbers that matter in the real world:

Memory Usage (for medium JSON):
- Standard Go: Uses about 49KB of memory with 789 allocations
- Sonic: Uses only about 12KB with just 4 allocations
- That's huge when you're processing millions of requests!
Practical Impact:
- Faster API responses
- Lower server costs
- Better user experience
- More requests handled per server

Should You Switch to Sonic?

Sonic might be great for you if:

You process lots of JSON data
Performance is important for your application
You're working on AMD64 or ARM64 processors

But remember:

It requires Go version 1.17 or higher
It works on Linux, MacOS, and Windows
Some features (like HTML escaping) need to be explicitly enabled if you need them

Quick Start Example

Here's how simple it is to use Sonic:

import "github.com/bytedance/sonic"

// Encoding
data := map[string]string{"hello": "world"}
bytes, err := sonic.Marshal(data)

// Decoding
var result map[string]string
err = sonic.Unmarshal(bytes, &result)

Wrapping Up

Sonic shows us that even with something as common as JSON processing, there's still room for impressive improvements. By using modern CPU features (SIMD), smart compilation (JIT), and thoughtful design choices, it achieves remarkable performance gains over the standard library.

Remember: In software development, it's not just about making things work - sometimes, making them work faster can open up new possibilities for what your applications can achieve!

DEV Community: Trung Duong

Correctness Has a Price: We Benchmarked Fair Leaderboards

TeneficGames / podium

High-performance, Redis-backed leaderboards for games and competitive applications.

Podium

Quickstart

Start with the correctness contract

Compare representations, not just commits

1. Plain sorted-set baseline

2. Production encoded-member design

3. Single-index encoded-member design

4. Packed composite score

Benchmark operations separately

Measure Redis memory, not only Go allocations

What the direct comparison showed

Then measure through the real API

The uncomfortable result was the most valuable

Reproducibility is a feature

What we learned

Redis Cluster Won't Shard Your Hot Leaderboard

TeneficGames / podium

High-performance, Redis-backed leaderboards for games and competitive applications.

Podium

Quickstart

One logical leaderboard, five physical keys

The cross-slot problem

Co-location restores atomicity

One leaderboard still belongs to one primary

Partition exceptional boards at the application level

Multi-leaderboard writes cross a different boundary

Test the topology you claim to support

A more honest scaling checklist

Your Redis Leaderboard Is Probably Breaking Ties Wrong

TeneficGames / podium

High-performance, Redis-backed leaderboards for games and competitive applications.

Podium

Quickstart

Why the obvious fixes fail

Use a Redis sequence, not time

Idempotency is part of fairness

Ascending leaderboards contain a trap

Atomicity is the real feature

Correctness is not free

The deeper lesson

😴 Go Code You Can Trust: Sleep Well After You Commit

The Cost of Uncertain Code

The Path to Trustworthy Code

1. Errors Are Your Friends, Not Exceptions

2. Tests That Give You Confidence

3. Graceful Degradation

4. Monitoring and Observability Built-In

Real-Life Example: The 3 AM Bug That Never Called

How to Write Go Code You Can Trust

Conclusion

ByteDance/Sonic: A Lightning-Fast JSON Library for Go

The JSON Problem Every Go Developer Knows

The Speed Champion: Meet Sonic

The Secret Sauce: Four Simple Tricks

1. Just-In-Time Compilation (JIT)

2. SIMD: Doing More Work at Once

3. Smart Memory Usage

4. Optional Features = Better Speed

Design

Real-World Benefits

Should You Switch to Sonic?

Quick Start Example

Wrapping Up

Further Reading