DEV Community: Sam Gozman

Temporal workflows in Golang: the 3 things that bite in production

Sam Gozman — Fri, 03 Jul 2026 13:37:29 +0000

I like Temporal. It's the rare orchestration tool that delivers on durable execution, and I've leaned on it for real work. But after running it in production for a while, I've noticed that the bugs that hurt are almost never the ones the Go compiler catches. They slip past go build, past go vet, past code review, and then they pick the worst possible moment to show up.

There are three classes of these that I keep running into, roughly in order of how often they ruin someone's afternoon:

Workers panicking on bad activity calls, because the SDK erases your types.
Workflows breaking replay determinism.
Event history bloating until a worker runs out of memory.

Two of the three now have tooling. One is pure architecture and no linter will save you from it. I'll go through all three: what causes it, why it's invisible until runtime, and what actually fixes it. Part of the first section is about a linter I wrote, but this post is really about the failure modes, not the tool.

1. The compiler can't see your activity calls.

Here's the signature behind most of the trouble:

func ExecuteActivity(ctx Context, activity interface{}, args ...interface{}) Future

Everything after ctx is untyped. The SDK has no idea what activity is or what arguments it expects, so it can't warn you when you get it wrong. Take a normal activity and a workflow that calls it with one argument missing:

func (a *Activities) Greet(ctx context.Context, name string) (string, error) {
    return "Hello, " + name, nil
}

func GreetWorkflow(ctx workflow.Context) (string, error) {
    ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
        StartToCloseTimeout: time.Minute,
    })

    var greeting string
    // Greet wants (name string). We pass nothing.
    err := workflow.ExecuteActivity(ctx, a.Greet).Get(ctx, &greeting)
    return greeting, err
}

This compiles. It looks fine in review. Then it runs, and the worker panics because Greet expected one argument and got zero.

Why a runtime panic is worse than it sounds.

When workflow code panics, what happens next depends on the worker's WorkflowPanicPolicy. The default is BlockWorkflow, and it does not fail the workflow. It puts the workflow task into a retry loop and leaves it there, on the assumption that you'll notice, fix the bug, and redeploy. The other option, FailWorkflow, fails the execution immediately, which is handy in development and dangerous in production, because one bad deploy can fail every open workflow at once.

So with the safe default, a missing argument doesn't fail one run and move on. It wedges every execution that reaches that line, and they all sit there retrying until you ship a fix. The workflow isn't dead, it's stuck, which is somehow more annoying.

This is a static fact about your code. There's no reason to learn about it from a stuck worker at 2am.

A linter for the type safety the SDK throws away.

This is the part I got tired of, so I wrote temporalcheck-lint, a golangci-lint module plugin that recovers the checks interface{} erases. The core analyzer is execargs. It resolves the real signature of the function behind ExecuteActivity, ExecuteChildWorkflow and ExecuteLocalActivity, then checks your call against it:

workflow.go:14  ExecuteActivity: activity "Greet" expects 1 argument, got 0 (arity)

Argument count is checked by default because it's never a false positive. The type-level checks are stricter and opt-in: strict-types flags a wrong argument type, strict-pointers flags T vs *T mismatches the DataConverter hides, strict-struct-shape flags the wrong struct, and strict-tests extends the same arity checking to your OnActivity / OnWorkflow mock matchers so the tests can't drift from the real signatures.

There's one thing to fix first. execargs can only check a target it can resolve, and a target named by its registered string is opaque:

// stringtarget flags this. The string can't be checked against a signature.
workflow.ExecuteActivity(ctx, "Greet", name)
// Use a function reference instead, which execargs can resolve.
workflow.ExecuteActivity(ctx, a.Greet, name)

So enable stringtarget, replace string targets with function references, and the rest of the plugin can suddenly see far more. The remaining analyzers cover the quieter mistakes:

optionsdiscard flags a WithActivityOptions / WithChildOptions call whose returned context you throw away, so the options never apply.
optionscontext flags using a context built by the wrong helper, like configuring child-workflow options and then running an activity with that context.
activitytimeout flags an ActivityOptions literal with no StartToCloseTimeout or ScheduleToCloseTimeout, which Temporal rejects at runtime.
futureget flags a Future.Get whose returned error you discard, silently swallowing an activity or child-workflow failure.
continueasnew flags a NewContinueAsNewError result that's dropped instead of returned, so the workflow quietly ends instead of continuing.
lossynumber flags any / map[string]any / []any parameters, where the JSON converter decodes every number as float64 and an int64 past 2^53 loses its low bits.
nonserializable flags chan and func parameters the DataConverter can't serialize at all.
workeroptions flags a worker.Options literal that sets a MaxConcurrentWorkflowTask* field to 1, which panics the worker on start.

Installing it is the standard golangci-lint module-plugin dance: a .custom-gcl.yml listing the plugin, golangci-lint custom to build a custom binary, then enable temporalcheck in .golangci.yml. The README has the full config with every strict toggle. The same binary plugs into GoLand and CI like the stock one.

That covers the type-safety class. It does nothing for the next one.

2. Replay determinism.

Temporal workflows have to be deterministic, and the reason is replay. The engine doesn't snapshot your workflow's memory. It records an event history and, whenever a worker needs to resume a workflow, it re-runs your workflow code from the top and feeds it that history to rebuild state. If your code does something that produces a different result the second time through, the replay diverges from the recorded history and you get a non-determinism error.

The usual culprits are the things that change between runs: time.Now, time.Sleep, math/rand, crypto/rand, reading the os standard streams. Plus a few Go constructs that are non-deterministic by nature: starting a goroutine, sending or receiving on a channel, ranging over a channel, and ranging over a map. The map one catches people, because Go randomizes map iteration order on purpose, so any logic that depends on that order replays differently.

workflowcheck for determinism across the call graph.

Temporal ships a static analyzer for exactly this, workflowcheck. It analyzes every function registered with RegisterWorkflow and reports non-deterministic code, and it follows the call graph. If a workflow calls Y which calls Z and Z ranges over a map, it reports the whole chain.

go install go.temporal.io/sdk/contrib/tools/workflowcheck@v0.5.0
workflowcheck ./...

The output is hierarchical, so you see why each call is a problem:

/path/to/workflow.go:12:2: MyWorkflow is non-deterministic, reason: calls non-deterministic function time.Now
  time.Now is non-deterministic, reason: declared non-deterministic
/path/to/workflow.go:12:2: MyWorkflow is non-deterministic, reason: calls non-deterministic function fmt.Printf
  fmt.Printf is non-deterministic, reason: accesses non-deterministic var os.Stdout

It works through go vet too, which is the setup I'd actually recommend you to use, since go vet output is cached and most editors surface it:

go vet -vettool $(which workflowcheck) ./...

False positives can be silenced inline with a //workflowcheck:ignore comment, or whitelisted globally in a config file when a flagged function is safe on the path you use.

There's one gap, and it's stated at the top of its own docs: workflowcheck does not catch global variable mutation. Reliably telling a deterministic global read apart from a non-deterministic mutation is hard in the general case, so it doesn't try. That's the one spot where my linter fills in. temporalcheck's workflowstate analyzer flags mutating a package-level variable from workflow code, and workflowlogger flags non-replay-aware logging (a plain log.Println re-emits on every replay, so one event shows up many times). The tradeoff is that those two only inspect the workflow function body and its closures, not helper calls. So the two tools cover opposite gaps: workflowcheck goes deep across the call graph on the determinism it knows about, and temporalcheck covers global mutation and replay logging that it skips.

Both of those classes are at least detectable. The third one isn't, and it's the one that actually took down a worker for me.

3. History size, and the OOM nobody warns you about.

This is the failure mode with no linter. It comes from the same replay mechanism that makes determinism matter, viewed from the other side.

When a worker needs to resume a workflow that isn't in its cache, because it was evicted, or the worker crashed and restarted, or it landed on a fresh worker, it downloads the entire event history and replays it from the beginning to rebuild state. That history lives in the worker's memory while it replays. The bigger the history, the more memory each replay holds, and the deserialized object graph in memory is larger than the bytes on the wire. Run a lot of workflows per worker, evict and reload a few of the fat ones at once, and the pod hits its memory limit and gets OOM-killed.

Temporal puts hard caps on history size precisely to stop this from getting unbounded. The numbers are worth knowing:

A single payload (each workflow and activity argument and return value) warns at 256KB and is capped at 2MB.
A single gRPC message, and a single event history transaction, are capped at 4MB. You can blow past this by scheduling many activities in one workflow task even when each individual payload is under 2MB.
The whole event history warns at 10MB (or 10,240 events) and is hard-terminated at 50MB (or 51,200 events). On Temporal Cloud these are non-configurable.

So the practical ceiling is 50MB of history, and in my experience the worker starts feeling it well before the server terminates the workflow. That lines up with the documented reason the cap exists: loading large histories into worker memory on replay is exactly what these limits are there to prevent.

What actually bloats the history.

Two things, and they compound.

The first is big payloads. Every activity argument and return value gets recorded in the event history. If an activity returns 500KB, you can run fewer than 100 of them before the history alone crosses 50MB, before counting any other events. Signal and Update inputs count too. The data you pass around is the data Temporal durably stores, forever, in history.

The second is too many events. Every activity, timer, signal, and child workflow adds events, each carrying its payload. A workflow that does a lot of small steps gets there a different way than one that does a few huge ones, but both walk toward the same 50MB wall.

I've written before about hitting serialization size limits with Protobuf on Kafka topics, and the instinct is the same here: the moment you're routinely passing multi-megabyte blobs through a system that records everything, you're going to hit a wall. Temporal just draws the wall at 50MB and enforces it.

The fix: keep big data out of history.

The pattern is to stop passing the data itself and start passing a reference to it. Store the blob somewhere else, pass a key through the workflow, and let the activity that needs the data fetch it by key. This is sometimes called the claim-check pattern.

Where you store it depends on the size and lifetime:

For large or durable blobs, put them in object storage like S3 and pass the URL or key. The 50MB-per-history problem becomes a tiny string per event.
For smaller, hotter, short-lived data, Redis is a better fit, with a TTL so it cleans itself up. If you want it compact in there, the msgp plus zstd approach I used for shrinking Redis cache drops structured payloads by around 90%.

// Instead of passing a 5MB report through the workflow...
func ProcessWorkflow(ctx workflow.Context, report Report) error { /* ... */ }

// ...store it once and pass the key.
func ProcessWorkflow(ctx workflow.Context, reportKey string) error {
    // activities fetch the blob by key from S3/Redis and return only small results
}

The activity does the upload at the edge and returns a key, the next activity downloads by key, and the workflow only ever sees identifiers. History stays small, replay gets cheap again, and the worker stops falling over. As a lighter measure, you can also compress payloads with a custom Data Converter before they ever reach history, which buys you headroom without restructuring anything, though it doesn't change the fundamental shape of the problem.

Where ContinueAsNew fits, and where it doesn't.

For workflows that bloat by event count rather than payload size, the long-running ones that loop forever, Temporal's answer is ContinueAsNew. It completes the current run and atomically starts a fresh one with the same Workflow ID, a new Run ID, and an empty history. You can decide when to call it with info.GetContinueAsNewSuggested() instead of hardcoding a threshold.

It works, but I want to be honest about the cost. ContinueAsNew is itself an action that consumes server and worker resources, and on Temporal Cloud it contributes to your overall usage and bill. The plumbing also doesn't scale gracefully: once you commit to it, every new timer, activity, or child workflow you add has to be threaded through the same continue-as-new bookkeeping, and that bookkeeping is some of the most intrusive code you'll write in a workflow. So I reach for it when a workflow really needs to run forever, not as a way to paper over giant payloads. For giant payloads, the claim-check pattern is the real fix. ContinueAsNew resets the event count, but if a single payload is the problem, a fresh history just fills up again.

Catching it at review time with an AI reviewer.

The catch is that none of this is something a linter can check, because whether you'll blow the 50MB cap depends on data volume the compiler never sees. A workflow that fans out one activity per user account is perfectly fine for a user with three accounts and a problem for a user with three thousand. The code looks identical either way.

What's worked for me is pushing this onto the AI code reviewer. I write the hard limits straight into the guardrails file the reviewer reads, the CLAUDE.md or AGENTS.md or whatever your tool picks up: single payload caps at 2MB, a transaction at 4MB, history warns at 10MB and dies at 50MB or 51,200 events. Then, and this is the part that actually does the work, I document the approximate data shapes inside the project itself. Things like "a user has on the order of N accounts of this type and status," or "reading accounts across all users lands somewhere in this range." Rough numbers, not exact, but enough to reason with.

With both the limits and the rough cardinalities written down, the reviewer can do the arithmetic on a pull request. It multiplies the documented per-record size by the expected count and tells you a given workflow is going to schedule tens of thousands of activities in one run, or that an activity return is going to push history past the warn threshold once real data hits it. Modern models are good at this kind of back-of-the-envelope estimate, better than I expected, and it has caught a few workflows for us before they shipped. It's not a guarantee. It is the difference between finding the problem as a review comment and finding it as a 2am OOM, which is the whole game.

Prefer the deterministic tool.

I want to be careful not to oversell that last trick, because it cuts against something I believe more strongly. A deterministic check beats an AI judgment every time you can have one. A linter gives the same answer on the same code today, next week, and in CI on a Friday night. An AI reviewer is a probabilistic helper that has good days and bad days, and the bad day is the day the risky workflow ships anyway. So the AI estimate is a stopgap for the one case that resists static analysis, the data-volume problem, and not a license to stop building tools for everything else.

If anything, the better use of AI here is the opposite of catching bugs at review time. It's building the thing that catches them deterministically. temporalcheck-lint exists in part because I could lean on AI to move through the analyzer scaffolding, the AST matching, and the test cases far faster than I would have alone. That's the leverage worth chasing: point the model at shaping a tool that then runs the same way forever, instead of asking it to re-derive the same judgment on every pull request. The moment you catch yourself piling rule after rule into an AGENTS.md and hoping the reviewer holds them all in mind, that's usually the signal the rule wants to be a linter instead.

Putting it together.

Three failure modes, and they fail in three different ways. The worker panics on a bad activity call. The replay diverges on non-deterministic code. The history bloats until a worker runs out of memory. None of them are exotic. They're the ordinary mistakes that happened to be invisible until production.

Two of them you can catch in CI today. Run temporalcheck-lint for the type-safety and panic-prevention class, and Temporal's workflowcheck for determinism across the call graph. They cover opposite gaps, so use both. The third one is a habit rather than a tool: treat event history as expensive, keep big data out of it by reference, reach for ContinueAsNew deliberately rather than reflexively, and write the limits and your rough data sizes into the file your AI reviewer reads so it can do the math you can't see at compile time.

The thing all three share is that the compiler is happy and the tests pass right up until the moment they don't. Temporal gives you durable execution, but durability means it remembers every argument you ever passed and replays every line you ever wrote. It pays to assume it will.

See you in the next post 👾

Sources:

temporalcheck-lint: https://github.com/samgozman/temporalcheck-lint
Temporal workflowcheck: https://pkg.go.dev/go.temporal.io/sdk/contrib/tools/workflowcheck
Workflow Execution limits: https://docs.temporal.io/workflow-execution/limits
Self-hosted Temporal Service defaults (payload, gRPC, history limits): https://docs.temporal.io/self-hosted-guide/defaults
Troubleshoot payload and gRPC message size limit errors: https://docs.temporal.io/troubleshooting/blob-size-limit-error
Managing very long-running Workflows with Temporal: https://temporal.io/blog/very-long-running-workflows
WorkflowPanicPolicy (worker package): https://pkg.go.dev/go.temporal.io/sdk/worker#WorkflowPanicPolicy

Context engineering for Claude Code: improving token efficiency

Sam Gozman — Fri, 19 Jun 2026 07:52:54 +0000

Token bills are the new cloud bills. If you run an agentic coding tool all day, you've probably watched the number climb and wondered where it all went. A lot of it goes to noise: the agent runs go test, dumps four thousand tokens of passing-test noise into the context, re-reads the same 600-line file for the third time, then pastes another wall of git status output. None of that is the work. It's the exhaust.

Most people reach for shorter prompts first. Wrong lever. The tokens aren't in your wording, they're in the exhaust, which makes token efficiency a context-engineering problem more than a writing one.

I've been running two tools to cut that exhaust for a while now: rtk and context-mode. And you know how I love small changes with low effort that give huge improvements in optimization. Same idea as compressing a Redis cache down by ~90% with zstd, except here the bytes you're saving are tokens. You install a thing, you adjust a setting or two, and the bill drops. This is one of those.

A note on scope. Everything here is about the CLI version of Claude Code. All of these tools work with Gemini CLI and Codex too, but Claude Code is where I tested, so that's what I'll talk about.

It's not just the bill.

The obvious reason to trim tokens is cost. The less obvious reason is quality.

Long, junk-filled context doesn't just sit there harmlessly until you hit the window limit. It actively drags the model down. Accuracy slides as the input grows, well before the window is full, and models are especially bad at using information buried in the middle of a long context. You feel it in practice: an agent that's read fifty files and three log dumps gets vague in a way it wasn't twenty minutes earlier.

So trimming the exhaust does two things at once. It saves money, and it keeps the model sharper for longer. That's a rare combination, and it's why I bother.

rtk: compress command output before it hits the model.

rtk is a CLI proxy. It sits between the agent's shell commands and the model, filters out the noise, and hands back a compact version. It's a single Rust binary, so installation is boring in the good way:

brew install rtk
rtk init -g       # installs the hook and an RTK.md, for Claude Code

Then restart Claude Code. That init step is the "adjust some Claude settings" part. It drops a PreToolUse hook that quietly rewrites your agent's Bash commands. When Claude runs git status, the hook turns it into rtk git status before it executes, and the agent gets the trimmed output without knowing anything changed.

Under the hood it leans on four ideas: strip the boilerplate, group similar items together, truncate the parts that don't carry signal, and collapse repeated lines into counts. The effect on something chatty like a push is dramatic:

# git push, raw
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
... about a dozen more lines ...

# rtk git push
ok main

The agent didn't need any of the enumeration chatter. It needed to know the push worked and which branch it landed on.

In my own usage, rtk cut well over 80% off the tokens spent on CLI command output. I haven't hit a problem caused by it so far, but that number comes with an asterisk, and the asterisk is the whole next section.

One thing to know about scope.

The hook only fires on Bash tool calls. Claude Code's built-in Read, Grep, and Glob tools don't pass through it, so they aren't compressed automatically. If you want rtk's trimming there too, use shell commands (cat, rg, find) or call rtk read and rtk grep directly. Worth knowing so you're not confused when a big file read shows up at full size.

Where I hold rtk back.

There's a good study floating around where someone benchmarked these optimizers properly, and the conclusion stuck with me: they can be silently dangerous. The tool decides what counts as "junk," and one bad cut makes your agent quietly dumber without you ever seeing why. Nothing errors out. The answers just get a little worse, and you have no idea the filter ate the reason.

That risk isn't the same for every command, and this is where you get to be smart about it.

For something like git status, git log, or a generic log dump, the noise-to-signal ratio is terrible and the cost of a mistake is low. If the filter trims a git log a bit too aggressively, the agent asks again or moves on. Fine. So I let rtk handle those freely.

Test runners and linters feel like the opposite, and this is where it gets interesting. When a test fails, the exact assertion and the line number are the whole point. When a linter flags a rule on a specific file, that detail is what the agent is supposed to act on. Those are exactly the things an aggressive filter is tempted to drop, because to a compressor they read like repetitive structured noise. So the obvious move is to exclude tests outright. I didn't, and I'm glad I didn't.

The thing that saves you is the pass/fail asymmetry. A passing test run is pure ceremony, hundreds of lines of "ok" the model learns nothing from. A failing run is all signal. So instead of excluding tests, I lean on rtk's tee: passing runs collapse to a one-liner, and the moment something fails, rtk writes the full unfiltered output to disk so the failure detail survives intact and the agent reads it without re-running anything.

You'll find the config file location and its full structure in the rtk docs. The relevant snippet for the tee setting looks like this:

[tee]
enabled = true
mode = "failures"   # keep the full output on disk whenever something breaks

[hooks]
# the escape hatch: only list a command here if you catch rtk
# dropping something you actually needed
exclude_commands = []

That tee line is the real guard, not the exclude list. It's what lets me compress the noisiest commands I run without losing the one output that ever matters. If I catch rtk eating a detail I needed, the command goes in exclude_commands, but so far I haven't had to put anything there.

context-mode: keep raw tool output out of context entirely.

rtk handles shell output. context-mode goes after the other big offender: MCP tools.

Every MCP call tends to dump its raw payload straight into the conversation. A browser snapshot, twenty fetched issues, a fat access log. It piles up fast and you lose a chunk of your window to data you looked at once. context-mode is an MCP server that keeps that raw data out of the context window. It runs the work in a sandbox, stores the request and response bits in a local SQLite database, and surfaces only the slice that's actually relevant.

The mental model it pushes is "think in code." Instead of reading fifty files into context to count something, the agent writes a small script that does the counting and prints just the answer. One script replaces ten tool calls, and only the result lands in the chat.

It also does session continuity. As you work, it records file edits, git operations, tasks, and errors into SQLite. When the conversation compacts, it can rebuild that working state so the model picks up where you left off instead of asking you what you were doing.

Install on Claude Code is through the plugin marketplace:

/plugin marketplace add mksglu/context-mode
/plugin install context-mode@context-mode

Restart, then run /context-mode:ctx-doctor to confirm everything wired up.

Learn how to turn it off before you need to.

Here's the catch, and it's the reason I tell people to try it rather than blindly trust it. Because the raw MCP input and output never land in the chat, debugging an MCP tool gets painful. If your agent is calling some tool wrong, you can't just scroll up and read the exchange, because the exchange isn't there. It got sandboxed.

So learn how to disable context-mode before you actually need to. When an MCP integration is misbehaving, turn it off, debug the tool with the raw payloads visible, then turn it back on. If you wait until you're mid-incident to figure out the off switch, you'll be annoyed.

One more grain of salt: the project reports reductions up to 98%. Treat numbers like that as directional. They're self-reported and they swing hard depending on your workload. Measure your own.

Do you even need this?

Worth asking honestly. The harnesses are catching up on their own. Claude Code already compacts context automatically and clears stale tool results, so some of what these tools do may get absorbed into the platform over time.

But "premature optimization" is my usual warning, and the reason this clears that bar is the effort side, not the savings side. You're not rewriting anything or trading away readability. You install a binary, flip a couple of settings, and the work continues exactly as before, just cheaper and a bit clearer. When the cost of trying something is this low and the savings show up on day one, I'll take it.

Be careful what you cache.

Prompt caching deserves its own warning here, because it's the one optimization that can quietly work against you. The pitch sounds like free money: mark a stable chunk of the prompt, pay a small premium to write it once, then read it back at a tenth of the price on every later turn. The catch is what it nudges you to do. The bigger and more frozen your context, the more you save. And the bigger and more frozen your context, the faster it rots.

That's the trap. One day the agent starts missing things it caught an hour ago, and you decide the model got dumber. It didn't. You buried the signal under a pile of cheap, cached tokens and stopped noticing.

It cuts both ways, too. Too little context and the model guesses, because it doesn't have what it needs. Too much and it starts seeing patterns that aren't there, dragging stale facts from twenty turns ago into the answer and hallucinating off its own noise. No flag fixes that. The whole game is keeping it balanced, and caching pushes you toward "more" precisely because more is cheaper. Cheap is not the same as good.

So my honest take: don't hand-roll prompt caching unless you own the inference stack. Claude already caches the stable prefix for you, Codex does its own caching on the OpenAI side, and the rest are heading the same way. Unless you're running your own swarm of local models where you control the entire request structure, you're reinventing something the platform already does for free, and you inherit the rot along with it.

The caching worth your time is the safe kind: caching what your tools and MCP servers produce, not the conversation itself. That's exactly what rtk and context-mode do. They keep the volatile junk out of the live window instead of freezing a stale prefix in place. Compressing a git log or sandboxing a fat MCP payload can't poison the model's reasoning the way a bloated, lovingly cached context can. Cache the exhaust, not the conversation.

Numbers from my setup.

These are my own numbers, not the projects' headline figures. Two tools, two very different ways of counting, and I trust one of them more than the other.

rtk is the one I trust. Across roughly three thousand intercepted commands it saved about 83.5% of the tokens that command output would otherwise have spent. The surprising part was where the savings came from. Sort the breakdown by impact and the entire top of the list is one command: go test.

That's the command I was nervous about earlier, so here's the honest reconciliation. Running the suite is the noisiest thing I do, and on a passing run the output is pure ceremony the model gains nothing from, which is exactly why it compresses to almost nothing. The savings come entirely from the passes. The failures, which carry all the signal, get written out in full by tee. The command I was most tempted to exclude turned into my single biggest win, precisely because compressing it throws away only noise.

context-mode is the one to read with a grain of salt, because its own accounting is clearly off. Its lifetime totals are badly incomplete, undercounting my real usage by a wide margin. So I ignore the aggregate numbers it reports and look only at single sessions, and even then I trust what I see in my editor over what the dashboard claims.

The per-session results swing hard, and that swing is the real story. On a long Swift and iOS session, the kind where context had grown past 280k, context-mode reported keeping around 96% of the raw tool output out of the window, the difference between staying inside a 256k model and spilling over. On two smaller sessions, a Go code review and building a Jira reporting skill, it kept 0% out. Everything fit, so there was nothing to offload.

That's not the tool failing on the small ones. It's telling the truth: the benefit scales with how much raw tool output a session produces, and most of mine don't produce much. The exception is loud and consistent. In my experience, anything Xcode-related drags in far more context and tokens than my Go work or my Jira and Notion MCP calls, by a wide margin. So the iOS session is exactly where the offloading earned its keep, and the quieter sessions are where it sat idle and did no harm.

The number I actually believe isn't on any dashboard. It's that I almost never see context truncation anymore, and tasks that used to need a 1M-token window now sit comfortably inside a 256k model without it losing the plot. Most of that came from getting the MCP tool overhead out of the live context. That's the win, whatever the counter says.

The cannon: headroom.

There's a bigger weapon on the shelf, and it's worth knowing about even if you don't pick it up. headroom takes the opposite bet from the two tools above. Instead of one job done well, it tries to be the whole compression layer: it sits between your agent and the provider and compresses everything the model reads, tool output, logs, RAG chunks, files, and the conversation history itself, before any of it lands in the window.

The detail that surprised me is that it doesn't compete with rtk. It bundles it. headroom ships the rtk binary for shell-output rewriting and compresses everything downstream of that. So it's a superset, not an alternative. You run it as a library, as a proxy, or with a single headroom wrap claude, and it slots in front of the model.

A few things set it apart from the point tools. Its compression is reversible: it keeps the originals locally and lets the model pull them back on demand, which is a real safety net that rtk's failure-only fallback doesn't match. It aligns prefixes so the provider's own cache keeps hitting instead of getting busted every turn. And it carries a shared memory across Claude, Codex, and Gemini, so several agents can draw on the same context.

That last part is the tell for who it's actually for. This isn't a tool for one developer on one provider. It earns its keep when you're running multiple agents or multiple providers and want a single shared cache and memory underneath them, or when you own the inference stack and can justify the weight. And it's weight: a Python install, an ML compression model, a runtime to serve it, and an optional memory stack with real databases behind it. That's a platform, not a quick install. The aggressive, model-based compression is also the most likely to quietly drop something important, though the reversibility takes the edge off that risk.

Wrap up.

Context engineering for agentic coding comes down to a simple idea: stop paying to shuttle exhaust into the model. rtk trims the shell output, context-mode keeps raw MCP payloads out of the window, and between them you claw back both money and a sharper model.

Just be deliberate about what you let the filter throw away. Point the compression at the noisy, low-stakes stuff, keep your test and lint output honest, and know where the off switch is before you need it. Measure on your own workload, because the answer might surprise you.

As for the cannon, I'm leaving it on the shelf for now. On a single provider, rtk and context-mode already clear a decent chunk of the waste on their own, and that's where my attention goes: quick wins, low effort, nothing I have to babysit. I'll reach for headroom when the shape of the problem changes. Once I've squeezed local models far enough that building my own little swarm at home makes sense, or once I'm running enough providers that a shared cache and a shared context become the actual bottleneck, the weight will be worth it. Not today.

See you on the next one 👾

Sources:

rtk: https://github.com/rtk-ai/rtk
rtk docs: https://www.rtk-ai.app/docs/getting-started/configuration/#config-file-location
context-mode: https://github.com/mksglu/context-mode
headroom: https://github.com/chopratejas/headroom
Token optimizer study: https://reddit.com/r/ClaudeCode/comments/1spiy8t/token_optimizers_for_ai_coding_agents_are/

Shrinking Redis cache with msgp and zstd in Golang

Sam Gozman — Fri, 03 Apr 2026 15:04:53 +0000

If you are storing structured data in Redis using encoding/json, you might be surprised how much memory you are wasting. JSON is readable, easy to debug, and universally supported. It's also bloated. Field names repeat on every single record, numbers get stored as strings, and boolean values take 4-5 bytes instead of 1.

In my previous article High-performance Golang struct optimizations: Paddings and Alignments, I showed how reordering struct fields can save 25% of RAM. This time, we are going after the other side of the equation: how the data is serialized before it hits Redis.

I will compare four approaches: plain JSON, MessagePack via tinylib/msgp, JSON compressed with zstd via klauspost/compress, and msgp + zstd combined. We care mostly about stored size here. CPU cost matters too, but for cache serialization, the bottleneck is almost always memory. The results weren't what I expected.

The struct.

I will reuse the struct from my previous article, with msg tags added for msgp code generation:

//go:generate go tool msgp

type NestedLayout struct {
    ID    int64 `msg:"id"`
    Phone int64 `msg:"phone"`
    Age   int32 `msg:"age"`
}

type Layout struct {
    BalanceInCents int64        `msg:"balance_in_cents"`
    IdempotencyKey int64        `msg:"idempotency_key"`
    Key            float64      `msg:"key"`
    User           NestedLayout `msg:"user"`
    AreaID         int32        `msg:"area_id"`
    CreatedAt      int32        `msg:"created_at"`
    UpdatedAt      int32        `msg:"updated_at"`
    ID             uint32       `msg:"id"`
    Status         uint16       `msg:"status"`
    IsActive       bool         `msg:"is_active"`
    IsSpecial      bool         `msg:"is_special"`
    IsMigrated     bool         `msg:"is_migrated"`
    TenantID       int8         `msg:"tenant_id"`
}

// Layouts is a named slice type.
// msgp cannot generate methods on anonymous slices like []*Layout,
// so we need this named type for codegen to work.
type Layouts []*Layout

One thing to note here: msgp requires a named slice type to generate MarshalMsg and UnmarshalMsg for collections. You can't just pass []*Layout to msgp's codegen. The named type Layouts solves this and lets you marshal the whole array in one call.

After adding these tags, install msgp as a tool dependency and run code generation:

go get -tool github.com/tinylib/msgp@latest
go generate ./...

This produces *_gen.go files with MarshalMsg, UnmarshalMsg, and Msgsize() methods for both types. No reflection, no runtime overhead for field lookup.

Why msgp and not protobuf or other codecs.

I know what you are thinking. Why not protobuf? I wrote about pitfalls of using Protobuf for Kafka before, and some of those concerns apply to caching too: schema management, versioning, and the requirement to maintain .proto files separately from your Go structs. For a cache layer, I want something that works directly with existing Go types.

msgp generates code from Go struct tags. No separate schema files, no extra compilation step beyond go generate. The generated code is fast because it produces direct binary encoding with no reflection, similar to how protobuf works at runtime but without the schema overhead.

Other options like gob are Go-specific and not particularly compact. encoding/binary needs manual marshaling. msgp sits right where I want it: generates fast code from existing structs, no schema files required.

The encoder and decoder.

The zstd encoder and decoder are safe for concurrent use but expensive to create, so initialize them once at the package level:

import "github.com/klauspost/compress/zstd"

var (
    zstdEncoder, _ = zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedFastest))
    zstdDecoder, _ = zstd.NewReader(nil)
)

I use zstd.SpeedFastest here because we are optimizing for cache throughput. The compression ratio difference between fastest and default level is small for structured data like this, but the CPU savings are noticeable at high request rates.

Now here are the four serialization functions we will benchmark:

// 1. Plain JSON
func encodeJSON(data Layouts) ([]byte, error) {
    return json.Marshal(data)
}

// 2. msgp only
func encodeMsgp(data *Layouts) ([]byte, error) {
    return data.MarshalMsg(nil)
}

// 3. JSON + zstd
func encodeJSONZstd(data Layouts) ([]byte, error) {
    jsonBytes, err := json.Marshal(data)
    if err != nil {
        return nil, err
    }
    return zstdEncoder.EncodeAll(jsonBytes, nil), nil
}

// 4. msgp + zstd
func encodeMsgpZstd(data *Layouts) ([]byte, error) {
    msgpBytes, err := data.MarshalMsg(nil)
    if err != nil {
        return nil, err
    }
    return zstdEncoder.EncodeAll(msgpBytes, nil), nil
}

And decoding:

func decodeJSON(b []byte) (Layouts, error) {
    var result Layouts
    return result, json.Unmarshal(b, &result)
}

func decodeMsgp(b []byte) (Layouts, error) {
    var result Layouts
    _, err := result.UnmarshalMsg(b)
    return result, err
}

func decodeJSONZstd(b []byte) (Layouts, error) {
    decompressed, err := zstdDecoder.DecodeAll(b, nil)
    if err != nil {
        return nil, err
    }
    var result Layouts
    return result, json.Unmarshal(decompressed, &result)
}

func decodeMsgpZstd(b []byte) (Layouts, error) {
    decompressed, err := zstdDecoder.DecodeAll(b, nil)
    if err != nil {
        return nil, err
    }
    var result Layouts
    _, err = result.UnmarshalMsg(decompressed)
    return result, err
}

Benchmarks.

Let's create a test dataset of 1000 records with realistic data and measure the output size and encoding speed for each approach:

func generateTestData(n int) Layouts {
    data := make(Layouts, n)
    for i := range n {
        data[i] = &Layout{
            BalanceInCents: int64(i*100 + 42),
            IdempotencyKey: int64(1000000 + i),
            Key:            float64(i) * 1.337,
            User: NestedLayout{
                ID:    int64(i + 1),
                Phone: 15551234567,
                Age:   25 + int32(i%40),
            },
            AreaID:     int32(i % 50),
            CreatedAt:  1700000000 + int32(i),
            UpdatedAt:  1700000000 + int32(i) + 3600,
            ID:         uint32(i + 1),
            Status:     uint16(i % 5),
            IsActive:   i%2 == 0,
            IsSpecial:  i%7 == 0,
            IsMigrated: i%3 == 0,
            TenantID:   int8(i % 10),
        }
    }
    return data
}

func BenchmarkEncode(b *testing.B) {
    data := generateTestData(1000)

    b.Run("JSON", func(b *testing.B) {
        for b.Loop() {
            _, _ = encodeJSON(data)
        }
    })

    b.Run("Msgp", func(b *testing.B) {
        for b.Loop() {
            _, _ = encodeMsgp(&data)
        }
    })

    b.Run("JSON+Zstd", func(b *testing.B) {
        for b.Loop() {
            _, _ = encodeJSONZstd(data)
        }
    })

    b.Run("Msgp+Zstd", func(b *testing.B) {
        for b.Loop() {
            _, _ = encodeMsgpZstd(&data)
        }
    })
}

And a separate test to print the actual byte sizes, which is really the number we care about:

func TestOutputSize(t *testing.T) {
    data := generateTestData(1000)

    jsonBytes, _ := encodeJSON(data)
    msgpBytes, _ := encodeMsgp(&data)
    jsonZstdBytes, _ := encodeJSONZstd(data)
    msgpZstdBytes, _ := encodeMsgpZstd(&data)

    t.Logf("JSON:       %d bytes", len(jsonBytes))
    t.Logf("Msgp:       %d bytes", len(msgpBytes))
    t.Logf("JSON+Zstd:  %d bytes", len(jsonZstdBytes))
    t.Logf("Msgp+Zstd:  %d bytes", len(msgpZstdBytes))
}

Here are the results from my machine (Go 1.25, Apple M4 Pro). Sizes first, since that is the whole point:

=== RUN   TestOutputSize
    t_test.go:  JSON:       256243 bytes
    t_test.go:  Msgp:       189709 bytes
    t_test.go:  JSON+Zstd:  22914 bytes
    t_test.go:  Msgp+Zstd:  27517 bytes
--- PASS: TestOutputSize (0.01s)

JSON produces ~256 KB for 1000 records. msgp alone drops that to ~190 KB, a 26% reduction. And here is where it gets interesting: JSON+zstd compresses down to ~23 KB, but msgp+zstd lands at ~27.5 KB. The msgp+zstd combination is larger than JSON+zstd. That wasn't what I expected.

Why msgp+zstd is bigger than JSON+zstd.

This surprised me at first, but it makes sense once you think about how zstd works. zstd is a dictionary-based compressor. It finds repeated byte sequences and replaces them with short back-references. JSON is full of exactly that kind of redundancy: the field names "balance_in_cents":, "idempotency_key":, "is_active": repeat verbatim for every record in the array. With 1000 records, the string "balance_in_cents" appears 1000 times. zstd sees that pattern and after the first occurrence, each repetition costs almost nothing.

msgp has already eliminated that redundancy. Field names are encoded as short binary keys, so there are no long repeated strings left for zstd to find. The msgp output is already compact, which paradoxically means zstd has less to work with. You end up with zstd's frame overhead on top of data that doesn't compress well.

So the tradeoff is not as simple as "stack both optimizations for maximum savings." If your goal is strictly the smallest possible cache footprint and you can afford the CPU cost, JSON+zstd actually wins on size. But that is not the full picture.

The speed side of things.

Here are the encoding benchmarks:

BenchmarkEncode/JSON-14          4254       274337 ns/op     262699 B/op     2 allocs/op
BenchmarkEncode/Msgp-14         37448        31810 ns/op     221185 B/op     1 allocs/op
BenchmarkEncode/JSON+Zstd-14     1960       557230 ns/op     591831 B/op     3 allocs/op
BenchmarkEncode/Msgp+Zstd-14     3906       300849 ns/op     417836 B/op     2 allocs/op

msgp encoding is 8.6x faster than JSON. But JSON+zstd, the size winner, is also the slowest option at 557 us/op, more than twice as slow as plain JSON. You pay for JSON's reflection-based marshaling first, then zstd's compression pass on top of a much larger input buffer.

msgp+zstd at 300 us/op is almost 2x faster than JSON+zstd while producing a comparable cache size (27.5 KB vs 22.9 KB). For a cache layer handling thousands of requests per second, that speed difference matters more than 4.6 KB.

And msgp alone at 31 us/op is almost 9x faster than plain JSON. If your cache data fits comfortably in Redis at 190 KB per user instead of 256 KB, that might be all you need, and you avoid the zstd dependency entirely.

The auto-generated msgp benchmarks confirm the per-struct performance as well:

BenchmarkMarshalMsgLayout-14     32107621     36.08 ns/op    224 B/op    1 allocs/op
BenchmarkAppendMsgLayout-14      76366846     15.95 ns/op      0 B/op    0 allocs/op
BenchmarkUnmarshalLayout-14      18507248     64.46 ns/op      0 B/op    0 allocs/op

36 nanoseconds to marshal a single struct with 13 fields and a nested sub-struct. Zero allocations on the append path. Generated code with no reflection does well here.

So which one should you pick?

It depends on what you are constrained by. Here is how I think about it:

If Redis memory is the bottleneck and you need the absolute smallest stored size, use JSON+zstd. ~23 KB per 1000 records, 91% reduction from plain JSON. You pay for it with 557 us per encode operation, which might be fine if your write rate is low.

If CPU and latency matter and you want fast serialization with a decent size reduction, use msgp alone. ~190 KB per 1000 records, 26% reduction, but 8.6x faster encoding with a single allocation. No compression dependency.

If you want a balance between size and speed, use msgp+zstd. ~27.5 KB per 1000 records, 89% reduction from plain JSON (close to JSON+zstd's 91%) at nearly 2x the encoding speed. This is probably the right default for most applications that need to optimize cache size without introducing a CPU bottleneck.

What does this mean in real numbers? If you are caching data for 100,000 active users with 1000 records each, JSON will eat ~24.4 GB of Redis memory. JSON+zstd brings that to ~2.2 GB. Msgp+zstd lands at ~2.6 GB, but encodes twice as fast. Either way, that is the difference between needing a large Redis cluster and getting by with a single instance.

A note on zstd singleton.

The first thing the klauspost/compress/zstd documentation warns about is not to create a new zstd.NewWriter or zstd.NewReader per request. Both are designed to be reused and are safe for concurrent access. Creating them is expensive because zstd initializes internal lookup tables and allocates buffers on construction. If you put zstd.NewWriter(nil) inside your request handler, you will burn CPU on initialization that has nothing to do with your actual data.

// NOT this per request
func handleRequest(data []byte) []byte {
    encoder, _ := zstd.NewWriter(nil) // expensive, don't do this
    defer encoder.Close()
    return encoder.EncodeAll(data, nil)
}

A cleaner approach is to wrap both into a struct that you initialize once and inject where needed:

type ZstdCompressor struct {
    encoder *zstd.Encoder
    decoder *zstd.Decoder
}

func NewZstdCompressor() (*ZstdCompressor, error) {
    encoder, err := zstd.NewWriter(nil,
        zstd.WithEncoderLevel(zstd.SpeedFastest),
        zstd.WithEncoderConcurrency(1),
        zstd.WithWindowSize(1<<20),
        zstd.WithLowerEncoderMem(true),
    )
    if err != nil {
        return nil, fmt.Errorf("zstd: failed to initialize zstd encoder: %w", err)
    }

    decoder, err := zstd.NewReader(nil,
        zstd.WithDecoderConcurrency(1),
    )
    if err != nil {
        _ = encoder.Close()
        return nil, fmt.Errorf("zstd: failed to initialize zstd decoder: %w", err)
    }

    return &ZstdCompressor{
        encoder: encoder,
        decoder: decoder,
    }, nil
}

func (z *ZstdCompressor) Compress(src []byte) []byte {
    return z.encoder.EncodeAll(src, nil)
}

func (z *ZstdCompressor) Decompress(src []byte) ([]byte, error) {
    result, err := z.decoder.DecodeAll(src, nil)
    if err != nil {
        return nil, fmt.Errorf("zstd: failed to decompress: %w", err)
    }

    return result, nil
}

This way you handle initialization errors properly instead of swallowing them with _, and the compressor can be passed around as a dependency. I'll explain the extra options in the next section.

Taming zstd memory usage.

After deploying the ZstdCompressor above with default options, I noticed RAM usage on the pods grew 2.5x over a couple of hours. Disabling the feature flag stopped new calls, but memory didn't drop. Only a pod restart freed it. That was a red flag.

The cause isn't a memory leak in the traditional sense. It's how the encoder manages its internal buffers. When you create an encoder with zstd.NewWriter(nil), it lazily initializes one set of internal buffers per concurrency slot. The default concurrency is GOMAXPROCS, so on a 4-core pod you get 4 internal encoders, on a 16-core machine you get 16. Each slot allocates window-sized buffers (default 8 MB), plus hash tables and scratch space. All of these are kept alive for reuse after the first EncodeAll call triggers their initialization. The Go garbage collector can't collect them because the singleton encoder holds references.

So if you have a 16-core pod, the encoder alone can hold 16 * ~10 MB = ~160 MB of buffers that will never be released until the encoder itself is garbage collected or the process exits. Under concurrent load, every slot gets initialized and stays initialized.

Three options on the encoder side fix this:

WithEncoderConcurrency(1) limits the encoder to a single internal buffer set. EncodeAll runs each call on one goroutine anyway, so multiple concurrent callers just queue. For a cache layer doing 50 RPS, this is more than enough. This alone can cut encoder memory by 4-16x depending on your CPU count.

WithWindowSize(1<<20) sets the window to 1 MB instead of the default 8 MB. For cache payloads under 200 KB, an 8 MB window is overkill. The window only needs to be larger than your largest input for compression to work at full ratio. 1 MB covers that with room to spare.

WithLowerEncoderMem(true) trades a small amount of compression speed for roughly half the buffer allocations. For SpeedFastest level with small payloads, the speed difference is negligible.

The decoder has a similar concurrency model. Default is min(GOMAXPROCS, 4) decoders, each with its own buffers. WithDecoderConcurrency(1) applies the same fix.

A generic cache helper.

If you want to use this approach across different types, you can write a generic helper using Go's type constraints. The trick is the two-type-parameter pattern: Go generics cannot call pointer-receiver interface methods on T directly, so you need a constraint that ties the pointer type to the value type.

type MsgpCodec[T any] interface {
    MarshalMsg([]byte) ([]byte, error)
    UnmarshalMsg([]byte) ([]byte, error)
    *T
}

func CacheSet[T any, PT MsgpCodec[T]](
    ctx context.Context,
    rdb *redis.Client,
    key string,
    value *T,
    ttl time.Duration,
) error {
    encoded, err := PT(value).MarshalMsg(nil)
    if err != nil {
        return err
    }
    compressed := zstdEncoder.EncodeAll(encoded, nil)
    return rdb.Set(ctx, key, compressed, ttl).Err()
}

func CacheGet[T any, PT MsgpCodec[T]](
    ctx context.Context,
    rdb *redis.Client,
    key string,
) (*T, error) {
    val, err := rdb.Get(ctx, key).Bytes()
    if err != nil {
        return nil, err
    }
    decompressed, err := zstdDecoder.DecodeAll(val, nil)
    if err != nil {
        return nil, err
    }
    result := PT(new(T))
    _, err = result.UnmarshalMsg(decompressed)
    if err != nil {
        return nil, err
    }
    return (*T)(result), nil
}

The call site looks a bit verbose because of Go's generics syntax:

err := CacheSet[Layouts, *Layouts](
    ctx, rdb, "user:123:accounts", &accounts, 10*time.Minute,
)

It's not pretty, but it is type safe and you only write the serialization logic once.

When this does not help.

Before you get excited about 90%+ reductions, I need to be clear about what kind of data benefits from this approach. It only makes sense for serialized JSON stored in Redis. If you are caching a plain string, a boolean flag, a counter, or any other scalar value, there is nothing to optimize. Redis already stores those efficiently. A SET user:123:name "John" is 4 bytes of payload. Running it through msgp + zstd would make it larger, not smaller, because of the encoding headers and compression frame overhead.

The wins come from structured data: arrays of objects, nested JSON documents, anything where encoding/json adds repeated field names, type coercion overhead, and structural characters like {, }, [, ], :, and ,. The more records in your array and the more fields in your struct, the bigger the savings. A single flat struct with 3 fields will barely compress. A thousand records with 13 fields each will compress dramatically, as we saw in the benchmarks above.

So keep your regular SET/GET for simple values. This approach is specifically for the cases where you are serializing Go structs or slices of structs into Redis as JSON blobs.

Caveats.

A few gotchas before you switch everything to msgp + zstd.
First, debugging gets harder. You can't just redis-cli GET a key and read the value anymore. You need a small tool to decode and decompress the data. For development and debugging, I would recommend keeping a fallback to JSON or at least having a CLI utility that can decode your cached values.

Second, msg tags on every field are mandatory. Without them, msgp falls back to Go field names for serialization keys. This works until someone renames a struct field and silently breaks deserialization of all existing cached data. Use explicit msg:"field_name" tags and treat them like database column names: once set, they should not change.

Third, versioning your cache keys is good practice when changing serialization formats. If you switch from JSON to msgp, old cached values will fail to decode. Use a versioned key prefix like v2:user:123:accounts so old and new formats can coexist during rollout.

Fourth, zstd compression ratio depends on your data. Repetitive data like arrays of similar structs compresses well. A single small struct with unique values might not shrink much, and you will pay the CPU cost for no benefit. Test with your actual data before committing.

Fifth, this works because our struct uses simple built-in types. All fields here are int64, int32, float64, bool, and so on. msgp knows how to serialize those out of the box. If your struct contains fields from external libraries, like decimal.Decimalfrom shopspring/decimal or uuid.UUID from google/uuid, msgp won't know how to encode them. You would need to implement the msgp.Marshaler and msgp.Unmarshaler interfaces on those types yourself, or convert the fields to primitive types before serialization (for example, storing a Decimal as a string or as cents in int64). Not a dealbreaker, but worth knowing before you adopt msgp for structs with non-trivial field types.

When to bother with this.

Same as with struct padding optimizations: most applications do not need this. If your Redis usage is well within limits and you are not worried about memory costs, plain JSON works fine and is easier to debug.

But if you are running into Redis memory limits, paying for oversized instances, or caching data for millions of users, a 90% size reduction is hard to ignore. Whether you pick JSON+zstd for maximum compression or msgp+zstd for the speed/size balance depends on your workload. Measure both on your actual data. The answer might surprise you, as it surprised me.

How to read benchmarks and what else you can optimize in your Go application I described in my article Optimization Odyssey: pprof-ing & Benchmarking Golang App.

Sources:

tinylib/msgp: https://github.com/tinylib/msgp
klauspost/compress/zstd: https://github.com/klauspost/compress
MessagePack specification: https://msgpack.org/
Zstandard RFC 8878: https://datatracker.ietf.org/doc/html/rfc8878

High-performance Golang struct optimizations: Paddings and Alignments.

Sam Gozman — Sat, 14 Jun 2025 19:47:01 +0000

When we're creating structs in Golang, we typically think about the fields we need and their types, the readability and maintainability. Sometimes we sort fields by their name, sometimes by their logic, and sometimes we just put them in the order we like. But what if I tell you that the order of fields in Golang struct dramatically affects the memory usage and performance of your application?

In certain scenarios, you can reduce RAM usage by your application by 20-50% and increase allocation and access performance for free. To achieve this, we need to understand how the Golang compiler works with paddings and alignments in structs.

Struct Layout.

The compiler in Golang uses a specified logic for struct layout to make it more efficient in terms of memory access and optimal alignment for target hardware architecture. The compiler will align struct fields to the size of the largest field type in the struct, and it will also add padding bytes between fields to ensure that each field is aligned correctly.

So if your struct has 2 fields, one of type int32 and another of type int64, the compiler will align the int64 field to 8 bytes boundaries, and it will add 4 bytes of padding after the int32 field to ensure that the int64 field is aligned correctly. This 8 byte boundary size applies to 64-bit architecture, which is the most common architecture used today. 64 bits equals 8 bytes, we can say that 64-bit architecture word size is 8 bytes and 32-bit architecture word size is 4 bytes.

However, if the underlying type size is greater than the word size, the compiler will still align it to our architecture word size (8 bytes in our case). For example, if you have a string field of size 30 bytes, the compiler will align it to 8 bytes boundaries making it look like 4 chunks: 3 chunks of 8 bytes and 1 chunk of 6 bytes plus 2 padding bytes to make it 32 bytes in total.

Those padding bytes are not used by the application, but they are still allocated in memory, which can lead to increased memory usage and decreased performance. What can you possibly put in 2 bytes? It can be 2 fields of int8 or byte type, or even bool; one field of int16 and uint16 etc.

So, the order of fields in your structure will affect the number of padding bytes added by the compiler between them and the overall memory usage of your application. The more padding bytes you have, the more memory your application will use, and the slower it will be.

How paddings are born.

Let's take a look at a simple structure example first:

  // NestedLayout represents a nested structure simple example.
  // Size: 24 bytes
  type NestedLayout struct {
    ID    int64 // 8 bytes
    Phone int64 // 8 bytes
    Age   int32 // 4 bytes
    // 4 bytes padding to align the structure to 8 bytes
  }

  // UnoptimizedLayout - not optimized for size and performance.
  // Size: 96 bytes
  type UnoptimizedLayout struct {
    AreaID         int32        // 4 bytes
    IsActive       bool         // 1 byte
    BalanceInCents int64        // 8 bytes
    IsSpecial      bool         // 1 byte
    IdempotencyKey int64        // 8 bytes
    ID             uint32       // 4 bytes
    User           NestedLayout // 24 bytes
    IsMigrated     bool         // 1 byte
    CreatedAt      int32        // 4 bytes
    Status         uint16       // 2 bytes
    Key            float64      // 8 bytes
    TenantID       int8         // 1 byte
    UpdatedAt      int32        // 4 bytes
  }

The structure UnoptimizedLayout may look normal to the untrained eye, but it has a lot of padding bytes between fields. The total size of the structure is 96 bytes after compiler optimizations. But if you manually calculate the sum of all the fields you will get 70 bytes. The size of your structure increased by 26 padding bytes. You can check the size of this structure for yourself using Sizeof function from unsafe package:

  func main() {
    fmt.Printf("UnoptimizedLayout size: %d bytes\n", unsafe.Sizeof(UnoptimizedLayout{}))
  }

UnoptimizedLayout size: 96 bytes

aligo: visualizing paddings of your structure.

To visualize paddings and alignments of your structure, I recommend using aligo tool. It is a simple command line tool that can help you to visualize the layout of your structures in Golang and help you to optimize it. You can install it using go install command:

go install github.com/essentialkaos/aligo/v2@latest

Next, you can run it against your structure:

aligo -s UnoptimizedLayout view main.go

This will output a visual representation of your structure with paddings and alignments:

Paddings marked with red color can be optimized by changing the order of fields in your structure. Let's take a look at another aligo command that will help us to optimize our structure:

aligo -s UnoptimizedLayout check main.go

This will output a sorted struct with optimized paddings and alignments:

You can see that we just changed the order of fields and reduced the size of our structure from 96 bytes to 72 bytes - 25% memory usage reduction.

aligo: comparing structures.

Let's rename our optimized structure to OptimizedLayout and run aligo view to visualize the difference:

You can see that the number of padding bytes is reduced from 26 to 2 bytes. The total size of the structure is reduced from 96 bytes to 72 bytes, which is a 25% memory usage reduction. We still have 2 padding bytes left, we can add 2 more boolean fields in our structure for free, and we will still have the same size of 72 bytes.

If you have a million of those records sitting in your memory, your unoptimized structure will use 96 MB of memory, while the optimized one will use only 72 MB. This is a 24 MB difference, which is a pretty significant number for a million records.

Benchmarks.

Now that we know how to optimize our structures, let's take a look at the performance difference between them.

  func Benchmark(b *testing.B) {
    k := 1000000 // Number of records to allocate and access
    unoptimized := make([]UnoptimizedLayout, k)
    optimized := make([]OptimizedLayout, k)

    b.Run("MemoryAllocation-UnoptimizedLayout", func(b *testing.B) {
      for b.Loop() {
        _ = make([]UnoptimizedLayout, k)
      }
    })

    b.Run("MemoryAllocation-OptimizedLayout", func(b *testing.B) {
      for b.Loop() {
        _ = make([]OptimizedLayout, k)
      }
    })

    b.Run("FieldAccess-UnoptimizedLayout", func(b *testing.B) {
      for i := 0; b.Loop(); i++ {
        for j := range unoptimized {
          unoptimized[j].BalanceInCents = int64(i + j)
        }
      }
    })

    b.Run("FieldAccess-OptimizedLayout", func(b *testing.B) {
      for i := 0; b.Loop(); i++ {
        for j := range optimized {
          optimized[j].BalanceInCents = int64(i + j)
        }
      }
    })
  }

Now run the benchmarks using go test -bench=. -benchmem command and you will get similar output:

Benchmark/MemoryAllocation-UnoptimizedLayout-14  2470  447593  ns/op  96002053 B/op 1 allocs/op
Benchmark/MemoryAllocation-OptimizedLayout-14    3802  314902  ns/op  72007681 B/op 1 allocs/op
Benchmark/FieldAccess-UnoptimizedLayout-14        948  1118201 ns/op  0        B/op 0 allocs/op
Benchmark/FieldAccess-OptimizedLayout-14         1202  927754  ns/op  0        B/op 0 allocs/op

Of course, the numbers may vary depending on your hardware and Go version, but you can see that the optimized structure has better memory allocation performance and field access performance. The memory allocation time for unoptimized structure is 447593 ns/op while for optimized structure it is 314902 ns/op. The field access time for unoptimized structure is 1118201 ns/op while for optimized structure it is 927754 ns/op. This is a 30% and 17% performance improvement respectively for 1 million records. Absolutely for free!

How to read benchmarks and what else you can bench I described in my article Optimization Odyssey: pprof-ing & Benchmarking Golang App.

The elephant in the room.

Let's take a look at the elephant in the room. The elephants, to be precise. Types that I never mentioned in this article on purpose.

Pointer - pointer to the structure in Golang only takes 8 bytes of memory on 64-bit architecture. But it obscures the real size of the structure it points to and adds additional job for the garbage collector as well.

Interface - the interface type in Golang is a reference type, which means that it does not have a fixed size and can be used to store any type that implements the interface. We can't say for sure ho much memory it will use because it depends on the implementation. But we can say that the interface pointer in Golang is 16 bytes. Why 16 bytes and not 8? Because Golang uses pointer tagging to store additional information about the pointer, like the type of the value it points to. This is done to make the garbage collector more efficient and to allow for more flexible memory management. So, if you have a structure with an interface field, it will use 16 bytes for the interface and additional memory for the implementation it points to.

String - the string type in Golang is also a reference type, which means that it does not have a fixed size and can be used to store any string value. The string type in Golang is implemented as a struct with two fields: a pointer to the string data and the length of the string. The pointer is 8 bytes (uintptr type) and the length is 8 bytes (int type), so the minimum size of the string type is 16 bytes. This means that if you have a structure with a string field, it will use 16 bytes for the pointer and the length of the string, plus additional memory for the string data itself.

Slice - the slice type in Golang is also a reference type, which means that it does not have a fixed size and can be used to store any slice value. Similar to the string type, the slice type in Golang is implemented as a struct with three fields: a pointer to the slice data, the length of the slice, and the capacity of the slice.

The same goes for the map type in Golang and others. I avoided them here on purpose because it is hard to say how much memory they will use in this educational article. It will depend on the implementation and the data you put in them, and it opens doors to more optimizations and considerations.

Some tips and considerations.

Now, when you know how to optimize your structures in Golang, you probably wonder why the compiler does not do it for you automatically? Don't we have a compiler flag for that? The answer is quite simple: the compiler does not know the context of your application and the logic behind your structures. You can use your structures for serialization and deserialization, for database operations, for network communication, etc. The compiler cannot know if the order of the fields in your structure is important for your application logic.

At this time, I need to stop you to think. Should you really optimize your structures? In most cases, the answer is no. The performance gain will be more visible only in certain cases, like working with large datasets, parallel batch processing with workflows, or high-performance applications. If you are working on a simple HTTP router - most likely you don't notice any difference in performance.

But let's say you want to make a habit of optimizing your structures for some reason. The rule of thumb here is to group the fields by their size, moving the largest fields to the top of the structure and the smallest fields to the bottom. This might be enough for most cases. You can also use the help of the govet linter in golangci-lint tool that you probably already use in your project. Just enable fieldalignment option for govet in your .golangci.yml configuration file. But it might be overkill.

You can also call this linter manually, since it is using a tool from the Golang standard library under the hood:

go install golang.org/x/tools/go/analysis/passes/fieldalignment/cmd/fieldalignment@latest
fieldalignment -fix main.go

I hope this article inspired you to take a look at your code and find some places where you can benefit from padding optimizations. If you are working with large datasets - every byte counts.

Cuckoo and Bloom filters: probabilistic efficient caching.

Sam Gozman — Sun, 25 May 2025 08:16:15 +0000

In this article, I wanted to tell you about a highly efficient cache design using probabilistic data structures.
This technique is used in many modern applications, including web browsers, databases, and distributed systems.

Cuckoo and Bloom filters are two popular types of probabilistic data structures that provide very space-efficient ways to store and lookup data that answers whether an element is a member of a set or not. The idea is to use optimized hashing for the elements instead of storing the elements themselves.

Those data structures are probabilistic in their nature, meaning that they can return false positives but never false negatives. Even though both algorithms allow less than 1% chance of false positives, this limits the algorithm's usefulness in many use cases. This is the price to pay for significant space efficiency and high lookup speeds.

The Cuckoo filter has one big advantage over the Bloom filter: it allows deletion of elements from the set, which can be very useful in numerous instances. It also provides better lookup performance and better space efficiency for applications requiring low false positive rates (< 3%). But before we dive into the details of the algorithms, let's take a look at some use cases where they can be used.

Use cases:

Financial fraud detection. To check whether a recipient was already flagged or not. Or to check whether his card was marked as stolen or not. An additional benefit of using a probabilistic filter in this case is that financial organizations can exchange their lists of stolen or blocked credit card numbers without revealing the numbers themselves. Many other security-related checks can be cached by this algorithm.
Check if username is taken or not. Checks where username/email/domain/etc. is already taken or not.
Handling a large amount of data requests that return "not found" errors. To check if data is not found before checking in a database or 3rd party service. Imagine you have an API that returns "not found" errors for 75% of requests. This is where probabilistic filters shine because they don't produce false negatives.
Ad campaigns. To show an ad based on if the user has signed up for a campaign or not. This is a good use case for the Cuckoo filter because it allows to delete users from the campaign if they unsubscribe.
Firewall network security, DDoS protection systems, etc. to efficiently check whether an IP address is already blocked or not.
Referee codes, discount/promo coupons. To check if a user has already used a code or not or if a code is valid or not. This is a good use case for the Cuckoo filter because it allows to delete codes from the set if they are used.
CDNs and caching. To optimize storage and request efficiency. For example, using Bloom filters to prevent caching of resources that are requested only once.
Blockchain events / logs searches. Ethereum is known to use Bloom filters to optimize the search for relevant logs in a block.

Those algorithms are mainly used with large data sets. The key takeaway is that you can use them to answer whether an element is a member of a set only for the cases where you can afford to have ~1% false positives. But you can use them to answer if an element is not a member of a set with 100% accuracy if you will be able to sync and maintain this set properly.

So if you need to have 100% accuracy in answering whether an element is a member of a set, you should use regular hash tables or sets instead. But you will pay the price of space and time complexity for that.

Choosing the right algorithm or tool is always a trade-off.

The Bloom filter.

A Bloom filter is a probabilistic data structure that is used to check if an element might exist in a set or definitely not. So it can guarantee only the absence of an element in the set but not its presence. Even though the false positive rate can be drastically reduced by increasing the number of hash functions, it can never be eliminated completely. However, Bloom filters do not permit deletion of items from the set.

When an item is passed through a fast hashing function, bits from the hash are sampled and set from 0 to 1 at specific intervals in a bitfield. This is how bloom filters operate. The same bits are sampled to see if they are present in a Bloom filter. Since a relational hash function generates unique identifiers, even though many items may have bits that overlap, we can be certain that a single bit from the hash hasn't been added before if it remains a 0.

A Bloom filter is an array of many bits. When an element is added to a bloom filter, the element is hashed (corresponding bits are set to 1). To check whether an item is present or not, the hash is computed and the filter sees whether the corresponding bit is set or not. Of course, this is subject to collisions. If a collision occurs, the filter will return a false positive.

To reduce the risk of collisions, an entry may use more than one bit per element. Generally, the more bits per element, the lower the likelihood of false positives. Another value affecting the accuracy of a Bloom filter is its fill ratio, or how many bits in the filter are actually set. The more bits that are set, the higher the chance of a false positive.

Choosing the right number of bits per element and the target fill ratio will be determined case by case. It depends on the data size, data added rate, acceptable false positive rate, size of filter limitation, etc. The internet is full of formulas to calculate the optimal configuration for the Bloom filter. I want to talk more about practical implementations!

Bloom filter usage in Go.

You would expect me here to show you a full implementation of a Bloom filter from scratch, but it rarely makes sense for production web applications. Most likely, you would like to use it with Redis or some Redis-like solution that already has it implemented and optimized for you.

Redis has a built-in support for Bloom filters. It was implemented as a separate module called RedisBloom and it is available on GitHub for Redis versions below 7. Starting from Redis 8 (May 2025), it is included in the core Redis distribution (not only the Bloom filter but a vast collection of probabilistic data structures as well).

This is a code example of how to use Redis Bloom filters in Golang:

// Creates an empty Bloom filter with a single sub-filter for the initial specified capacity(1000) and with an upper bound error_rate(0.01).
res1, err := rdb.BFReserve(ctx, "bikes:models", 0.01, 1000).Result()
if err != nil {
  panic(err)
}

fmt.Println(res1) // >>> OK

res2, err := rdb.BFAdd(ctx, "bikes:models", "Smoky Mountain Striker").Result()
if err != nil {
  panic(err)
}

fmt.Println(res2) // >>> true

res3, err := rdb.BFExists(ctx, "bikes:models", "Smoky Mountain Striker").Result()
if err != nil {
  panic(err)
}

fmt.Println(res3) // >>> true

This example is taken from official Redis documentation: https://redis.io/docs/latest/develop/data-types/probabilistic/bloom-filter/. With functions like BFReserveExpansion, BFReserveNonScaling and BFReserveWithArgs you can control not only the initial capacity and error rate but also the scaling behavior of the filter.

The required number of bits per item, given the desired error_rate and the optimal number of hash functions, is -ln(error_rate) / ln(2)^2. Hence, the required number of bits in the filter is capacity * -ln(error_rate) / ln(2)^2.

1% error rate requires 7 hash functions and 9.585 bits per item.
0.1% error rate requires 10 hash functions and 14.378 bits per item.
0.01% error rate requires 14 hash functions and 19.170 bits per item.

So the total size of the filter with 1000 items and 0.01% error rate is 1000 * 19.170 = 19170 bits = 2396 bytes = 2.4 kilobytes. And the total size of the filter with 1000 items and 1% error rate is 1000 * 9.585 = 9585 bits = 1198 bytes = 1.2 kilobytes. Only requires twice more space for 0.01% vs 1% false positive rate difference.

Once again, the key tradeoff is between the size of the filter and the false positive rate. Depending on your data set size, available memory, and acceptable false positive rate, you can choose the right configuration for your Bloom filter.

The Cuckoo filter.

The Cuckoo filter is a more advanced probabilistic data structure used for high-speed set membership testing (like the Bloom filter). The main advantage over the Bloom filter is that Cuckoo allows deletion of items from the set, which is not possible with the Bloom filter. The Cuckoo filter also provides better lookup performance and space efficiency compared to the Bloom filter.

I must mention that my analysis of the Cuckoo filter is primarily based on the original research paper from 2014: Cuckoo Filter: Practically Better Than Bloom. I recommend you read it as well if you want to understand the algorithm in more detail and check its performance against various Bloom filter variations.

The Cuckoo filter is an array of buckets, storing fingerprints of the values in one of the buckets at positions decided by the two hash functions. A membership query for item x looks for x's fingerprint in all potential buckets and returns true if an identical fingerprint is found. The size of a cuckoo filter's fingerprint directly affects the false positive rate.

Cuckoo filter behavior suggests that the fill rate will probably never reach 100% since the filter is likely to declare itself full before capacity is reached. The filter will automatically create more sub-filters when it declares itself full, resulting in decreased performance and an increased error rate. The size of the previous sub-filter is multiplied by expansion rate to create the new sub-filter. The error rate increases linearly with the number of sub-filters, just like bucket size.

Cuckoo filter usage in Go.

Yet again, I would like to provide you with the real usage example, not the full implementation of the algorithm. The Cuckoo filter is also available in Redis in the same RedisBloom module, which is part of Redis 8 core distribution.

The usage is similar to the Bloom filter, but I would recommend using CFReserveWithArgs at all times. Choosing the right bucket size, max iterations, and expansion multiplier is crucial for the performance of the Cuckoo filter. It is a more complex filter to set up than Bloom if you want to achieve performance gains.

res1, err := rdb.CFReserveWithArgs(ctx, "bikes:models", &redis.CFReserveOptions{
  // The filter will not fill up to 100%, make sure to reserve extra capacity if you want to avoid expansions.
  Capacity:      1000,
  // The minimal false positive error rate is 2/255 ≈ 0.78% when bucket size of 1 is used.
  BucketSize:    1,
  // Number of attempts to swap items between buckets before declaring filter as full and creating an additional filter.
  MaxIterations: 20,
   // When a new filter is created, its size is the size of the current filter multiplied by expansion.
  Expansion:     2,
}).Result()

if err != nil {
  panic(err)
}

fmt.Println(res1) // >>> OK

res2, err := rdb.CFAdd(ctx, "bikes:models", "Smoky Mountain Striker").Result()
if err != nil {
  panic(err)
}

fmt.Println(res2) // >>> true

res3, err := rdb.CFExists(ctx, "bikes:models", "Smoky Mountain Striker").Result()
if err != nil {
  panic(err)
}

fmt.Println(res3) // >>> true

res4, err := rdb.CFDel(ctx, "bikes:models", "Smoky Mountain Striker").Result()

if err != nil {
  panic(err)
}

fmt.Println(res4) // >>> true

This example is mostly taken from official Redis documentation: https://redis.io/docs/latest/develop/data-types/probabilistic/cuckoo-filter/. The CFReserveWithArgs function allows you to specify the bucket size, max iterations, and expansion multiplier, which are crucial for the performance of the Cuckoo filter.

A wholly different approach in filter setup makes it harder to compare with the Bloom filter directly. Which is why I once again refer you to the original paper for more details on the algorithm and its linear performance change based on the number of sub-filters and bucket size.

Cache fill control.

When using probabilistic filters, it is important to understand that they are not a replacement for a regular cache. They are a complementary tool that can be used to improve the performance of a cache or other data sources.

Since those filters will not fill themselves with hashes, we need to talk about it. In this part, I want to list popular cache fill patterns that you might use.

Write-Through Pattern. In this pattern, all write operations to the data source are first written to the cache (and filters), and then to the data source. Adds overhead for write, update, and delete operations but ensures that the cache is always in sync.
In-Memory Pattern. Load all data hashes into the filter once (e.g., at application startup, with CRON jobs, workflows, etc.) and keep it in memory. This is the simplest pattern, but it requires that all data fits into memory and can be loaded at once.
~~Cache-aside Pattern.~~ This is the most common caching pattern, where the application code is responsible for checking the cache first and then fetching the data from the data source if it is not found in the cache. This pattern is almost useless for probabilistic filters. I just wanted to mention it here for completeness.

There are more other patterns as well, but I wanted to focus your attention first on the two most popular for probabilistic cache: Write-Through and In-Memory. The choice of the pattern depends on the use case, data size, and performance requirements.

Conclusion.

While Cuckoo filters and Bloom filters are both used in common applications, they have different strengths and weaknesses. Bloom filters are simpler to implement and use, but they do not allow deletion of items from the set. Cuckoo filters are more complex, but they provide better performance and allow deletion.

In my opinion, I would rather use the Bloom filter over Cuckoo in most cases, even now, 11 years after the Cuckoo filter paper was published. My main reasons are those:

Bloom filter has gained widespread adoption and has been supported by many libraries and tools for years now
Bloom filter is significantly simpler to implement and use compared to Cuckoo filter (if you need to implement it from scratch)
Bloom filter has more predictable false positive rates and performance characteristics. It is easier to calculate the required size of the filter and the number of hash functions needed for a given error rate
Bloom filter is already significantly faster and more space-efficient than regular hash tables or sets, which is the main goal of using it in the first place. Picking up the Cuckoo filter over the Bloom filter should be done with careful consideration and benchmarks based on real data size and access patterns
Bloom filters typically exhibit better performance and scalability when inserting items (if you need to add data frequently)
Deletion feature of the Cuckoo filter might be a huge advantage in some cases, but there are some workarounds to achieve similar behavior with the Bloom filter. For example, using Counting Bloom filter allows you to delete items from the set for the price of 3-4x space overhead against the classic Bloom filter. It is not currently supported by Redis out of the box, but you can implement it yourself or use a library that supports it

So, in short, if you need a simple and efficient way to check if an element is a member of a set, use the Bloom filter. If you need to delete items from the set and can afford the complexity overhead, use the Cuckoo filter. If your main goal is to use the most efficient algorithm, still start with the Bloom filter. You will know in practice if you need to switch to the Cuckoo filter later on based on your data size and access patterns.

Probabilistic data structures proved to be very useful and efficient over the years, and they are widely used in applications working with big data sets or high RPS requirements. They are a great tool to have in your toolbox, but I have to emphasize one thing:

Never engage in premature optimizations.

If you implement something new, start small. Make it happen first; you can always optimize it later. Use hash tables or sets to cache some steps, and then if you see that you have a problem with space or performance of your cache, consider using Bloom filters or Cuckoo filters.

Sources:

Cuckoo filter paper: https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf
Redis Bloom official documentation: https://redis.io/docs/latest/develop/data-types/probabilistic/bloom-filter/
Redis Cuckoo official documentation: https://redis.io/docs/latest/develop/data-types/probabilistic/cuckoo-filter/
Bloom Filter Python implementation in great details: https://www.geeksforgeeks.org/bloom-filters-introduction-and-python-implementation/

Optimization Odyssey: pprof-ing & Benchmarking Golang App

Sam Gozman — Mon, 06 May 2024 07:39:58 +0000

In this post, I want to guide you through the whole process of profiling and benchmarking the Golang app. I will create a simple server app with the most naive implementation of it and will show you the step-by-step process of finding slow spots in your application with the help of the pprof profiling tool.

But before we get started, I want to emphasize something for you:

Never engage in premature optimizations.

Usually, the naive approach to the problem is the best and safest. In the real world, outside Leetcode, your employer wants you to ship fast, not spending much time on optimizing nanosecond executions. Yes, there are some optimizations you still need to perform (like solving N+1 mysteries), but they are most likely to be business logic optimizations not covered in this post.

Do fine-tune optimizations and benchmarking only when necessary or if you expect to see some massive load on this part of the code. Your 0.1 RPS app does not need this, believe me 🙂

Stage 1: The Naive Approach.

OK, so before we deep dive into the pprof profiling, let's create our elementary project first. I will create some server app with just one route that will iterate over the large JSON array to return some summary. The dummy data array will consist of some fictional data objects of users' transactions and bank accounts; the app will need to iterate over the array and sum up those numbers. For the server, I will choose gin since it is the most popular tool currently, and for calculating money amounts I will choose shopspring/decimal.

Everything I mentioned here in this article is available for you in the GitHub repository: samgozman/golang-optimization-stages. At this point, you can safely skip Stage 1 and go directly to the next chapter.

The main function will be ServeApp and you can find the source code for it here: samgozman/golang-optimization-stages/blob/main/stage1/main.go. Nothing special about it, just a classic gin server with context shutdown. Let's keep attention on the part that will do some work, in our case the router handler GetJSONHandler:

// GetJSONHandler is a simple handler that returns a JSON response with a message
func GetJSONHandler(c *gin.Context) {
  // 1. Read the content of the file dummy_data.json
  data := ReadFile("../object/dummy_data.json")

  // 2. Return a JSON response with a message
  users := ParseJSON(data)

  // 3. Calculate the total balances
  currents, pendings := BalancesTotals(users)

  // 4. Calculate the total transactions
  transactionsSum, transactionsCount := TransactionsTotals(users)

  c.JSON(200, gin.H{
    "current":            currents.String(),
    "pending":            pendings.String(),
    "transactions_sum":   transactionsSum.String(),
    "transactions_count": transactionsCount,
  })
}

It consists of 4 major functions: ReadFile, ParseJSON, BalancesTotals and TransactionsTotals.

// ReadFile reads a file and returns its content as a string
func ReadFile(filePath string) []byte {
  content, err := os.ReadFile(filePath)
  if err != nil {
    panic(fmt.Errorf("failed to read file: %w", err))
  }
  return content
}

func ParseJSON(data []byte) []object.User {
  var users []object.User
  if err := json.Unmarshal(data, &users); err != nil {
    panic(fmt.Errorf("failed to unmarshal JSON: %w", err))
  }

  return users
}

// BalancesTotals calculates the total balances of the users.
func BalancesTotals(users []object.User) (currents decimal.Decimal, pendings decimal.Decimal) {
  for _, user := range users {
    current, _ := decimal.NewFromString(user.Balance.Current)
    currents = currents.Add(current)

    pending, _ := decimal.NewFromString(user.Balance.Pending)
    pendings = pendings.Add(pending)
  }

  return
}

// TransactionsTotals calculates the total transactions of the users.
func TransactionsTotals(users []object.User) (sum decimal.Decimal, count int) {
  var transactionsSum decimal.Decimal
  var transactionsCount int

  for _, user := range users {
    for _, transaction := range user.Transactions {
      amount, _ := decimal.NewFromString(transaction.Amount)
      transactionsSum = transactionsSum.Add(amount)
      transactionsCount++
    }
  }

  return transactionsSum, transactionsCount
}

Doesn't really matter what fields are in object object.User for this purpose. The main thing you should know is that it is a big structure and the array size is ~1.8 MB on the disk.

I think that you can find already just a few spots that could be done more efficiently, like using a pointer to the structure, combining BalancesTotals and TransactionsTotals, reducing the Big O complexity (nested for-loops), or counting transactions with len().

It would be fantastic if you could spot those things. However, it is not the best approach for an optimization process because you are not sure which changes will have the biggest impact. And do they have a positive impact at all? Will this be enough?

How to Im'pprof' our code: the guide into Golang profiling tool.

To start the optimization process, we first need to prioritize our targets. To do so, we need to find the slowest parts of our application and the most memory-inefficient parts of it. This where the pprof tool comes in.

pprof, or performance profiler, is a tool that helps you to collect CPU profiles, traces, and memory heap profiles of your Golang application. It also comes with features to analyze and visualize generated pprof reports (but this will require graphviz installed).

There are a few techniques of how you can capture profiles from your app:

By manually calling the pprof tool in your code (in the benchmarking process)
By running pprof API router alongside of your application, which is just one line of code with the help of standard library net/http/pprof.

I will go with the first approach for this article because it can show you more of how you can use this tool. And you will not always be able to set up a pprof API comfortably in your application (if you are building some desktop application or a CLI tool for instance).

Now, back to the code! We need to create some very basic tests for the app, but in the case of profiling, I would recommend going with the benchmark test type.

func BenchmarkServeApp(b *testing.B) {
  // Start pprof profiling
  if err := utils.StartPprof(); err != nil {
    log.Fatal(err)
  }

  // Create a context
  ctx, cancel := context.WithCancel(context.Background())
  defer cancel()

  // Run the server
  go ServeApp(ctx)

  // Run the benchmark
  for i := 0; i < b.N; i++ {
    _, err := http.Get("http://localhost:8080/json")
    if err != nil {
      b.Fatal(err)
    }
  }

  // Cancel the context
  cancel()
}

I have moved pprof initialization to the utils.StartPprof, we will return to it in a bit. Normally, we would have to create a benchmark just for the router handler function GetJSONHandler, since it holds all the logic we are trying to optimize. But, for the educational approach, I will test the whole ServeApp - because you might want to replace gin router with something else to see if it improves any numbers.

To run the benchmark in Go, all you have to do, besides creating a test with b *testing.B as a parameter, is to run a console command:

go test -bench=.

As simple as that, but we will add some more parameters to it later on. Back to the profiling initialization function that I moved to have fewer repetitions between the steps in the repository for that post.

var cpuprofile = flag.String("cpuprofile", "", "write cpu profile to `file`")
var memprofile = flag.String("memprofile", "", "write memory profile to `file`")

// StartPprof starts pprof profiling
func StartPprof() error {
  flag.Parse()
  if *cpuprofile != "" {
    f, err := os.Create(*cpuprofile)
    if err != nil {
      return fmt.Errorf("could not create CPU profile: %w", err)
    }
    if err := pprof.StartCPUProfile(f); err != nil {
      _ = f.Close()
      return fmt.Errorf("could not start CPU profile: %w", err)
    }
    pprof.StopCPUProfile()
    if err := f.Close(); err != nil {
      return fmt.Errorf("could not close CPU profile file: %w", err)
    }
  }

  if *memprofile != "" {
    f, err := os.Create(*memprofile)
    if err != nil {
      return fmt.Errorf("could not create memory profile: %w", err)
    }
    runtime.GC() // get up-to-date statistics
    if err := pprof.WriteHeapProfile(f); err != nil {
      _ = f.Close()
      return fmt.Errorf("could not write memory profile: %w", err)
    }
    if err := f.Close(); err != nil {
      return fmt.Errorf("could not close memory profile file: %w", err)
    }
  }

  return nil
}

Ok, that's quite a lot of code! Remember, you can avoid writing this manually if you can "afford" to run a profiling router alongside your application. We are doing it this way not because we want just to optimize the app, but also to measure the results with controlled benchmarks.

Let's take a look at this code. First, we are getting the pprof options from the CLI. In the StartPprof we are creating 2 separate files for binary output data from pprof (btw, it is not just binary, it is basically a compiled Protobufs). In this case, we are trying to capture CPU and memory heap profiles. Pprof has a few more options you can trace, but those are the most common and important in my opinion to start with.

Remember that Golang has a garbage collector? Garbage collection is the process of finding and reclaiming memory that is no longer in use by your program. In Go, this is typically managed automatically, but here we are calling runtime.GC(), which is not the thing you will normally do in your application. The reason runtime.GC() is called before writing the memory profile is to ensure that the memory statistics are up-to-date. If there is any memory that can be reclaimed, runtime.GC() will do so, and then the memory profile will reflect the current state of memory usage after that cleanup.

In contrast, CPU profiling is about tracking where time is spent, rather than resources that are used. It records the function call stack at regular intervals during the period of profiling. This is why runtime.GC() is not called in the CPU profiling section - it's not relevant to the data being collected.

Capturing & reading performance profiles with pprof.

Everything is set up and ready for the first test! Let's proceed with the actual bech & profiling:

go test -v -cpuprofile=cpu.pprof -memprofile=mem.pprof -benchmem -bench=. -benchtime=1s -count=10

What it does is running our benchmark test with performance profile writing into mem.pprof and cpu.pprof files. The part I want you to keep attention to is those 2 flags:

-benchtime=1s will ensure we will run as many iterations as possible within 1 second. This will help to get fast results regarding your function RPS capabilities. You can also lock the number of iterations of your code to run instead of a timeout by setting 5x as an argument (to run 5 times, for example).
-count=10 is how many times we will repeat our benchmark. For fast 1s benchmarks, I recommend you to run at least 10 times to get some average values. Depending on many things in your system, each run will not be the same. This is why it is recommended to run benchmarks in some dedicated server environment with as few background processes as possible to get reliable results.

After running this command, you will get this output (your actual numbers will vary):

BenchmarkServeApp-10   75   15359607 ns/op   5022236 B/op   63424 allocs/op
BenchmarkServeApp-10   79   15283220 ns/op   5019885 B/op   63420 allocs/op
...

It is a lot of information and is not that readable at the first glance (I don't know why the go-team forgot to add some headers here, but anyway). You can manually average down those values or to run longer tests (also benchmark iterations can be compared automatically by the CLI tool, but I will address it later). Let's take a look at how we can read those values:

BenchmarkServeApp-10 - The -10 indicates the env GOMAXPROCS value, which is the maximum number of CPUs that can be executing simultaneously.
75, 79, etc.: These are the number of iterations that the benchmark was able to run in the time specified by -benchtime (1 second in our case).
15359607 ns/op, 15283220 ns/op, etc.: This is the average time taken per operation, in nanoseconds. An "operation" is one iteration of your benchmark function.
5022236 B/op, 5019885 B/op, etc.: This is the average amount of memory allocated per operation, in bytes. This includes both heap and stack memory.
63424 allocs/op, 63420 allocs/op, etc.: This is the average number of memory allocations made per operation.

Now that you understand how to read Golang benchmark values, it is time for the next part: reading mem.pprof and cpu.pprof profiles.

Small reminder - graphviz installation required.

I will start from the mem.pprof, since it is impossible to open them both with one command. To do so, run this command and head into the browser:

go tool pprof -http=':8081' mem.pprof

Open the URL provided in the console to access pprof UI. Before you will be overwhelmed with the chart, let's look at the menu. In the "Sample" section, there are 4 options available: alloc_objects, alloc_space, inuse_objects and inuse_space. I think it is obvious from the name what each sample group represents. The group that is interested the most to me is alloc_space - click on it.

What you see here is a call stack of functions used in our app and the libraries inside it. Just by looking at this graph view, you can already notice what functions allocated the most space during execution. We had only 4 functions in GetJSONHandler where ParseJSON with ReadFile is draining most of the memory.

Focus on optimizing function in the descending order of the resources they use.

Next, in the "View" section of the menu, choose "Flame Graph" view. The default graph was great, but I prefer to use "Flame Graph" here (I found it more compact and readable).

You have probably noticed an alarming memory usage value, almost 4 GB in our case! Oh, that's a lot! But don't be frightened, it is not a constant RAM pressure.

The pprof tool in Go provides a visualization of the memory usage of your program. When you see large numbers like 4 GB, it's showing the cumulative size of the objects allocated in your program, not the current memory usage. And, since we were running the command with the -count=10 flag for 1 second each iteration, you see the cumulative size for 10 runs.

The pprof tool records every allocation made by your program. If your program allocates plenty of small objects that are quickly garbage collected, the total size of those allocations can be much larger than the actual memory usage of your program at any given moment.

For example, if your program allocates 1 MB of memory 4000 times, and each time it frees that memory before the next allocation, your program would never use more than 1 MB of memory at a time. However, pprof would report that your program allocated 4 GB of memory.

So, the large numbers you're seeing in pprof are not a measure of your program's memory usage, but rather a measure of its memory allocation. It's a useful tool for identifying parts of your code that are creating a lot of garbage for the garbage collector to clean up, which can slow down your program.

Ok, that was a lot of information. But we are not done, not yet! Remember that we are still having the cpu.pprof to process? Stop the current pprof execution and open the next file:

go tool pprof -http=':8081' cpu.pprof

From the first seconds, you can notice that we are having a larger and more complex graph. And you can see that your application was used ~51% of the CPU time… And the rest was utilized by Golang runtime and garbage collector. Let's switch the view to "Flame Graph" to look at our program (the sample setting is "cpu"):

See how it differs from memory usage? Most of the CPU time was utilized on the ParseJSON function. Every other function that we wrote are having so insignificant an impact on the CPU, that we can't even set their names on the chart without clicking on the items. But is it a correct conclusion? No, it is not.

While you can't see ReadFile, BalancesTotals & TransactionsTotals functions here, they are impacting your application performance in the most hidden way: through the runtime and garbage collector (GC). Our root application only occupies about 51% of CPU time. Every unnecessary memory allocation in nested loops, pointers misuse, redundant goroutines etc. will lead to pressure on the CPU via the runtime process and GC. It is the rest 49% of our program and there is no easy way to measure the impact here (and some processes we can't control), but we will use this information to compare the impact of our future optimizations.

Stage 2: Optimizing JSON parsing in Golang.

Before we continue, there is something I need you to modify in the go test command. Add at the end of it > stage1.txt to save the output to the text file instead of the console. We will use it later to assess our results between stages.

go test -v -cpuprofile=cpu.pprof -memprofile=mem.pprof -benchmem -bench=. -benchtime=1s -count=10 > stage1.txt

You also will need to install benchstat cmd if you would rather not compare benchmark results manually every single time https://cs.opensource.google/go/x/perf.

go install golang.org/x/perf/cmd/benchstat@latest

From every performance metric that we've seen so far: JSON parsing is our main bottleneck. It is a well-known problem for applications that need to operate under high RPS load. Our first step here is to find a more efficient library to handle JSON tasks. I will save you some time and will go on with go-json, you will only need to replace import. You can also go with sonic library, but its performance is very dependent on the target system.

-import "encoding/json"
+import "github.com/goccy/go-json"

Now it's time to run our performance test for the second time, but this time we will save it into stage2.txt file.

go test -v -cpuprofile=cpu.pprof -memprofile=mem.pprof -benchmem -bench=. -benchtime=1s -count=10 > stage2.txt

Just by looking at the stage2.txt output, I can already tell that we made some huge difference in terms of performance. The number of iterations we were able to perform in one second significantly increased by 3-3.5x times!

BenchmarkServeApp-10   229  4959000 ns/op  6174943 B/op   31765 allocs/op
BenchmarkServeApp-10   247  4957436 ns/op  6168185 B/op   31759 allocs/op
...

It is time to compare results with our new tool. To run the benchstat we only need to provide those files:

benchstat -table .fullname ./stage1/stage1.txt ./stage2/stage2.txt

Now let's look at the results

benchstat -table .fullname ./stage1/stage1.txt ./stage2/stage2.txt
.fullname: ServeApp-10
            │ ./stage1/stage1.txt │         ./stage2/stage2.txt         │
            │       sec/op        │   sec/op     vs base                │
ServeApp-10          15.569m ± 2%   5.221m ± 5%  -66.47% (p=0.000 n=10)

            │ ./stage1/stage1.txt │         ./stage2/stage2.txt          │
            │        B/op         │     B/op      vs base                │
ServeApp-10          4.785Mi ± 0%   5.732Mi ± 3%  +19.77% (p=0.000 n=10)

            │ ./stage1/stage1.txt │         ./stage2/stage2.txt         │
            │      allocs/op      │  allocs/op   vs base                │
ServeApp-10           63.42k ± 0%   31.75k ± 0%  -49.94% (p=0.000 n=10)

The benchmark results show that after replacing the standard JSON decoder with go-json, the execution time per operation (sec/op) decreased by approximately 66.47%, indicating a significant improvement in performance. However, the memory usage per operation (B/op) increased by about 19.77%, and the number of allocations per operation (allocs/op) decreased by nearly 49.94%, suggesting that while the operation became faster, it also became slightly more memory-intensive.

After looking at the updated pprof flame graph, you can see that ParseJSON function still takes up a lot of space here. You can also notice that the total cumulative memory usage skyrocketed from 4 GB to almost 20 GB! But keep in mind, that we were able to increase the number of operations per second by roughly 3 times and, well, B/op is up 19.77%, which was a tradeoff. This explains such a difference in cumulative memory usage.

Overall, this was a giant step in our optimization journey! Let's continue forward.

Stage 3: Optimizing ReadFile operation.

Our next step is to increase ReadFile performance. Instead of relying on os.ReadFile we can use the built-in bufio package. Buffio implements a buffered reader and writer of the same interface.

func ReadFile(filePath string) (io.Reader, func()) {
  file, err := os.Open(filePath)
  if err != nil {
    panic(fmt.Errorf("failed to open file: %w", err))
  }

  return bufio.NewReader(file), func() {
    _ = file.Close()
  }
}

func ParseJSON(data io.Reader) []object.User {
  var users []object.User

  if err := json.NewDecoder(data).Decode(&users); err != nil {
    panic(fmt.Errorf("failed to unmarshal JSON: %w", err))
  }

  return users
}

func GetJSONHandler(c *gin.Context) {
  data, cancel := ReadFile("../object/dummy_data.json")

  users := ParseJSON(data)
  // Close the file reader after reading the content.
  // Note: it can be deferred, but it's better to close it as soon as possible.
  cancel()

  ...
}

The primary distinction between buffered I/O and standard I/O lies in how data is processed. Buffered I/O operates by reading or writing data in blocks, rather than handling one byte at a time.

Let's run our tests and compare the results!

BenchmarkServeApp-10     189   5937594 ns/op 4593636 B/op   31772 allocs/op
BenchmarkServeApp-10     190   6154874 ns/op 4559697 B/op   31766 allocs/op
...

We can see that something is not right. The number of iterations decreased (from ~250 in Stage 2 to 190). Next, I will run a benchstat to compare results further:

benchstat -table .fullname ./stage2/stage2.txt ./stage3/stage3.txt
.fullname: ServeApp-10
            │ ./stage2/stage2.txt │         ./stage3/stage3.txt         │
            │       sec/op        │   sec/op     vs base                │
ServeApp-10           4.795m ± 2%   6.155m ± 3%  +28.37% (p=0.000 n=10)

            │ ./stage2/stage2.txt │         ./stage3/stage3.txt          │
            │        B/op         │     B/op      vs base                │
ServeApp-10          5.699Mi ± 2%   4.235Mi ± 3%  -25.70% (p=0.000 n=10)

            │ ./stage2/stage2.txt │        ./stage3/stage3.txt         │
            │      allocs/op      │  allocs/op   vs base               │
ServeApp-10           31.75k ± 0%   31.76k ± 0%  +0.04% (p=0.012 n=10)

The execution time per operation (sec/op) increased by 28% (this explains the significant drop in iterations), however, the memory usage per operation (B/op) decreased by almost the same amount! So we are using less memory, but need more CPU to achieve this.

What we have learned here is that each optimization has its own drawbacks. You need to consider what you are trying to achieve - whether it's speeding up your application or increasing its performance. These objectives do not always go hand in hand.

I will continue to focus on speeding up the application here, so we should devise a different approach suitable for our case. Since we are consistently serving only one file, and it remains intact at all times, we can cache it.

The best approach here will be to use sync.Once struct from the standard package. It will ensure that ReadFile saves the data just once.

var (
  once    sync.Once
  content []byte
)

func ReadFile(filePath string) []byte {
    once.Do(func() {
        var err error
        content, err = os.ReadFile(filePath)
        if err != nil {
            panic(fmt.Errorf("failed to read file: %w", err))
        }
    })
    return content
}

Neat and simple. We should run our benchmarks to see the impact:

BenchmarkServeApp-10     280   4208854 ns/op 4188958 B/op   31745 allocs/op
BenchmarkServeApp-10     286   4146523 ns/op 4165174 B/op   31741 allocs/op
...

OK, that was better, really better! The number of iterations increased from Stage 2 by about 10-12%. Let's look at the benchstat as we did before:

benchstat -table .fullname ./stage2/stage2.txt ./stage3/stage3.txt
.fullname: ServeApp-10
            │ ./stage2/stage2.txt │        ./stage3/stage3.txt         │
            │       sec/op        │   sec/op     vs base               │
ServeApp-10           4.795m ± 2%   4.329m ± 3%  -9.71% (p=0.000 n=10)

            │ ./stage2/stage2.txt │         ./stage3/stage3.txt          │
            │        B/op         │     B/op      vs base                │
ServeApp-10          5.699Mi ± 2%   3.928Mi ± 2%  -31.07% (p=0.000 n=10)

            │ ./stage2/stage2.txt │        ./stage3/stage3.txt         │
            │      allocs/op      │  allocs/op   vs base               │
ServeApp-10           31.75k ± 0%   31.74k ± 0%  -0.03% (p=0.000 n=10)

After caching the file, the benchmarking results show a significant performance improvement. The operation time decreased by approximately 9.71%, the memory usage declined by around 31.07%, and the number of allocations per operation remained almost the same.

Now you can't even find the ReadFile function from the pprof UI! Furthermore, good to mention that the total cumulative memory usage has noticeably reduced from 20 GB to 15 GB. I'm pleased with the results and will continue further!

Stage 4: Pointers & Loops.

I think that the optimizations described in this chapter were the most obvious from the start. The first reason I kept them for so long is that you are most likely to use pointers automatically and can spot unnecessary loops on your own. The second reason is that while this approach matters, it will only help us just a tiny bit in comparison with the previous things.

I will change 3 things in the code:

Replace all []object.User with []*object.User. If you are new to this, you might want to do something like *[], but it will make things worse in any use case. Copying pointers is a cheap operation.
Combine BalancesTotals & TransactionsTotals into one function to remove redundant loops. Sidenote: you can also flatten the data, so you can remove a nested loop, but that will require a bit more time to cook.
Count transactions with len()

I think that is enough of the code being shown here already. If you're interested, you can see the full example in the GitHub repository: samgozman/golang-optimization-stages/tree/main/stage3. Let's move to benchmarking once again…

BenchmarkServeApp-10     277   3953330 ns/op 3697193 B/op   31744 allocs/op
BenchmarkServeApp-10     306   3929780 ns/op 3693402 B/op   31739 allocs/op
...

Just a slight difference, benchstat will show us a bit more:

benchstat -table .fullname ./stage3/stage3.txt ./stage4/stage4.txt
.fullname: ServeApp-10
            │ ./stage3/stage3.txt │      ./stage4/stage4.txt      │
            │       sec/op        │   sec/op     vs base          │
ServeApp-10           4.329m ± 3%   4.264m ± 7%  ~ (p=0.684 n=10)

            │ ./stage3/stage3.txt │         ./stage4/stage4.txt          │
            │        B/op         │     B/op      vs base                │
ServeApp-10          3.928Mi ± 2%   3.519Mi ± 0%  -10.42% (p=0.000 n=10)

            │ ./stage3/stage3.txt │      ./stage4/stage4.txt      │
            │      allocs/op      │  allocs/op   vs base          │
ServeApp-10           31.74k ± 0%   31.73k ± 0%  ~ (p=0.467 n=10)

The memory usage declined by 10% from the previous iteration, everything else is the same. Most of the impact here comes from switching to pointers, the loop optimization hasn't affected the performance much.

Stage 5: Cache Cheating.

And now it is the time for a little cheat which is only applicable to our code example! Remember that our router only serves one file every single time? It is not that uncommon, especially if your server does the job of serving front-end texts. In Stage 3 I introduced sync.Once for the ReadFile function. But we can use it for the whole handler logic!

var (
  once     sync.Once
  response gin.H
)

func GetJSONHandler(c *gin.Context) {
  once.Do(func() {
    // 1. Read the content of the file dummy_data.json
    data := ReadFile("../object/dummy_data.json")

    // 2. Return a JSON response with a message
    users := ParseJSON(data)

    // 3. Calculate the total balances and transactions
    currents, pendings, transactionsSum, transactionsCount := GetTotals(users)

    response = gin.H{
      "current":            currents.String(),
      "pending":            pendings.String(),
      "transactions_sum":   transactionsSum.String(),
      "transactions_count": transactionsCount,
    }
  })

  c.JSON(200, response)
}

I have moved once.Do from ReadFile into GetJSONHandler. Now we only create this response once, which makes all our previous optimizations kinda useless. But I wanted to show them because this particular cheat can only be used in some very specific examples. In more complicated scenarios, you will probably cache responses in some Redis-like key-value storage via router middleware. Nevertheless, the global idea is the same: use cache.

BenchmarkServeApp-10    3146    466019 ns/op   24553 B/op     174 allocs/op
BenchmarkServeApp-10    2426    511220 ns/op   22710 B/op     171 allocs/op
...

Astonishing results!

benchstat -table .fullname ./stage4/stage4.txt ./stage5/stage5.txt
.fullname: ServeApp-10
            │ ./stage4/stage4.txt │          ./stage5/stage5.txt          │
            │       sec/op        │    sec/op      vs base                │
ServeApp-10          4264.5µ ± 7%   805.6µ ± 340%  -81.11% (p=0.000 n=10)

            │ ./stage4/stage4.txt │          ./stage5/stage5.txt          │
            │        B/op         │     B/op       vs base                │
ServeApp-10        3603.50Ki ± 0%   27.12Ki ± 12%  -99.25% (p=0.000 n=10)

            │ ./stage4/stage4.txt │        ./stage5/stage5.txt         │
            │      allocs/op      │ allocs/op   vs base                │
ServeApp-10          31733.5 ± 0%   174.0 ± 5%  -99.45% (p=0.000 n=10)

Now this router barely uses anything, it is just crazy! Yes, it is cheating, but the legal one in the case of this example. I suggest to look at our alloc_space graph for the last time, just for fun:

It is difficult to notice any function on this graph that we've been optimizing for all this time! The total cumulative memory usage is now 1/8 of its original value. The same applies to the CPU profile as well.

Summary.

It was too much information for one blog post, but I'm glad I didn't split it in half! We moved from our first naive approach, which could hold up to 78 requests per second, to the optimal approach in stage 4 which helped to increase RPS up to 279 on average. And in stage 5 we've used some illegal cache cheating to move the plank even higher to up to 1487 RPS.

Everything that we've covered here is about how to approach the optimization process, what steps to take, and how to use pprof and benchmarks in Golang.

Some tips and notes:

Never engage in premature optimizations
Read more about pprof in the README file: github.com/google/pprof
You can optimize work with large objects by using sync.Pool method. It has its drawbacks, but will help to reduce the number of allocations in some cases
The sync.Once function will lead us to the usage of global variables, which is an antipattern in most languages. Use this thing wisely and never export those variables to the outside
If you want to bench some part of the application inside the test, be sure not to use any mocks in it, as it will lead to some troubles: testing.B interface is not expected by most mocking techniques (testing.T)
If your benchmark setup process takes significant time before the actual testing - use b.ResetTimer() function to reset timer statistics
Use net/http/pprof package to run pprof API alongside your application (this can help to "debug" some problems on production)
When optimizing your code, strive not to sacrifice readability

I hope you found this information useful and see you in the next post! Subscribe for future updates :)

The State of Dependency Injections in Golang

Sam Gozman — Mon, 22 Apr 2024 16:00:33 +0000

Dependency injection (DI) is a well-established concept in programming. This is the design pattern in software engineering where dependencies are injected into a class rather than the class creating them itself, promoting loose coupling and easier testing.

However, when it comes to Golang, where classic object-oriented programming (OOP) constructs like classes are absent (struct's don’t count, sorry mates), implementing dependency injection poses some slight differences. But you will soon see that this practice is quite easy to implement in Golang, even without third-party dependencies. In fact, I'm 100% certain that you're already using DI in your Golang applications because this pattern is almost unavoidable.

Later in this article, I will demonstrate you how you can enforce your experience with DI in Go by using a simple tool from Google called wire.

DI without dependencies.

If you have ever written unit tests in Go, then you are likely already familiar with this practice. It is the only way to do a unit test when you have some database connections or any kind of third party external service used in your function. You will need to mock this service. To achieve this, you will have to provide it (either to the function directly, or into the struct your method assigned to). You will also need to have an interface for your third-party service and both your mock and your service should implement the methods of an interface.

Accept interfaces, return structs.

For example, you will have a structure like this in some service.go file:

// Some service your App depends on
type ThirdPartyService struct {
    baseURL string
}

func NewThirdPartyService(baseURL string) *ThirdPartyService {
    return &ThirdPartyService{
        baseURL: baseURL,
    }
}

func (t *ThirdPartyService) FindSomething(query string) (string, error) {
    resp, err := http.Get(fmt.Sprintf("%s%s", t.baseURL, query))
    if err != nil {
        return "", err
    }
    defer func(Body io.ReadCloser) {
        _ = Body.Close()
    }(resp.Body)

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        return "", err
    }

    return string(body), nil
}

type ThirdPartyServiceInterface interface {
    FindSomething(query string) (string, error)
}

type App struct {
    service ThirdPartyServiceInterface
}

func NewApp(service ThirdPartyServiceInterface) *App {
    return &App{
        service: service,
    }
}

func (a *App) DoSomething(query string) (string, error) {
    return a.service.FindSomething(query)
}

This is the way how you can create a dependency injections in Golang, and it also allows you to mock your dependencies in the unit tests.

To use this App in your code, you will need to construct all its dependencies first and then proceed with the NewApp.

In order to unit test your App you will need to mock this ThirdPartyService. There are various methods to generate mocks from the interface, but I will not discuss it here and will stick a manual solution with classic testify/mock and testify/assert which are commonly used in many mockery libraries under the hood.

Here is an example of your test with mock:

import (
    "testing"

    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/mock"
)

type MockThirdPartyService struct {
    mock.Mock
}

func (m *MockThirdPartyService) FindSomething(query string) (string, error) {
    args := m.Called(query)
    return args.String(0), args.Error(1)
}

func TestApp_DoSomething(t *testing.T) {
    mockThirdPartyService := new(MockThirdPartyService)
    app := NewApp(mockThirdPartyService)

    mockThirdPartyService.On("FindSomething", "test").Return("response", nil)

    result, err := app.DoSomething("test")
    assert.NoError(t, err)
    assert.Equal(t, "response", result)
}

What we just did is created a mock for the ThirdPartyService and injected it as a dependency into your app.

See? You are already employing this design pattern even if you weren't consciously aware of it. There is no way you can escape from dependency injection in Go since you can’t stub and spy on your method as easy as it was in Node.js with a sinonjs library, for example.

That was easy, but what is the challenge?

The challenge comes when your app grows bigger. In fact, you are having multiple binaries using dozens of internal and external dependencies, and it becomes harder to organize and reuse your components and to maintain low coupling.

Now, imagine you have a dozen App structures, each represents the struct of its own binary. And each App is having a unique sets of components, although there may be intersections between them. And some components will have their very own dependencies etc.

Well, it is not a big challenge; you can manage this manually, no doubt. However, this approach will require consistency from you and is likely to be error-prone. Additionally, you'll need to monitor unused dependencies in your components and apps. A preferable approach is to enforce this practice with an external tool to mitigate common mistakes.

There are many dependency injection tools on the market, for example: samber/do, or uber/fx. However, they all share one common trait - it is not just about dependency injection, it is more like a DI framework. Lots of features, necessity to learn specific syntax and write your services and components in the prescribed manner. And sometimes those tools can introduce breaking changes. Furthermore, if you intend to share your components with other services, you'll need to use the same DI tool, or alternatively, write a wrapper.

I want to show you the different approach (my favourite among others) - google/wire. It will be much, much simpler and will not force you to stick with some specific syntax. Your components can be shared with others without forcing them to use your tool.

Wire up with Google.

google/wire is an external tool for Golang made by Google to generate code that helps us to connect component using DI. There is no specific syntax of how you should write your components. It can be a structure of any kind, like you already have. As a result, the learning curve is minimal, as you only need to write a wire.go file.

//go:build wireinject

//go:generate go run github.com/google/wire/cmd/wire

package service

import "github.com/google/wire"

func initApp(baseURL string) (*App, error) {
    wire.Build(
        // All providers needed to create the App or other providers that are needed in between
        NewThirdPartyService,
        // NewApp is the function that creates the final App struct
        NewApp,

        // Bind the interface to the implementation so Wire knows which one to use.
        // Accept interfaces, return structs. Remember?
        wire.Bind(new(ThirdPartyServiceInterface), new(*ThirdPartyService)),
    )

    return &App{}, nil
}

Well, that's basically it. You can run “go generate” to produce the new code:

go generate ./...

This will should generate wire_gen.go file with app/component constructor:

// Code generated by Wire. DO NOT EDIT.

//go:generate go run -mod=mod github.com/google/wire/cmd/wire
//go:build !wireinject
// +build !wireinject

package service

// Injectors from wire.go:

func initApp(baseURL string) (*App, error) {
    thirdPartyService := NewThirdPartyService(baseURL)
    app := NewApp(thirdPartyService)
    return app, nil
}

What it does is try to match all dependencies that you provided with their independent components. One dependency can be used in different components at the same time. There are no strict ordering rules for the dependencies to be processed in wire.Build. If you have redundant dependencies, the wire tool will notify you. Similarly, if you're missing dependencies from your build configuration, it will also alert you.

Now, you can simply run initApp(...) in your main () to access the built App. There's no need to manually create an initialization function for each dependency in every app ever again. This becomes very convenient when you have dozens of components within the same app. For instance, let's consider a more elaborate wire build:

wire.Build(
    ProvideLogs,
    ProvideConfig,
    ProvideServer,
    ProvideHandler,
    ProvideEmailService,
    ProvideAuthService,
    ProvideTemporalService,
    ProvideAwsStorage,
    ProvideAwsSns,
    ProvideGrpcClient,
    NewApp,
)

Logs can be used in any components, so as Config and others. Wire will connect all of them, so you don’t have to worry about unreadable main() or init() functions. Remember to add wire.Bind to associate interfaces with implementations for Wire.

Since one component can have multiple dependencies, you can organize them into ProviderSet by using wire.NewSet() function and storing alongside your component:

var ProviderSet = wire.NewSet(
    ProvideConfig,
    ProvideService,
    wire.Bind(new(types.ServiceInterface), new(*Service)),
)

No magic is done here, no special syntax to learn - it will just make your wire.go files look cleaner by grouping all the things in one place:

func initApp(ctx context.Context, cfg *config.Config) (*serverApp, error) {
    wire.Build(
        db.ProviderSet,
        github.ProviderSet,
        jwt.ProviderSet,
        captcha.ProviderSet,
        mailer.ProviderSet,
        server.ProvideServer,
        handler.ProviderSet,

        newServerApp,
    )

    return &serverApp{}, nil
}

You can find this code example of google/wire usage in my go-bloggy repository. In fact, this blog is built with Wire :)

Wrap up the Wire.

In my opinion, Wire is the easiest and simplest dependency injection tool I’ve seen in my life, not only in Golang exclusively. It just gets the job done and nothing else. No learning curve, no special syntax; you can seamlessly add it to your project, without even touching the code of your existing components. This means your code and components remain highly reusable, even in repositories that don't use Wire. Love it 🧡

Some tips and tricks:

Use Provide verb instead of New for components initialization. ProvideService, ProvideClient etc.
Combine multiple inner provider dependencies into ProviderSet.
Create provider.go file in each component to hold all component initializations and ProviderSets in one place.
When you use wire.Bind(new(ServiceInterface), new(*Service)) don’t forget about *Service. It should be a pointer in most cases (if your methods are attached like func (s *Service)).
If you see the error “no provider found” - double check that component has all declared dependencies in wire.Build. Also double-check the bindings of interfaces with structs and pointers.
There are some uncertainties regarding how to manage configuration. The dependency injection pattern often leads to a scenario where each component has its own sources of configuration (with each component reading only its own environment variables). While this approach helps maintain low coupling, I find it cumbersome to have configuration scattered across multiple places and prefer to consolidate it into a single file. Therefore, my ProvideConfig function will accept the core Config as a parameter and retrieve only the necessary configuration values from it. However, I'm still undecided on the best approach here, so I'm open to suggestions.

Long-term pitfalls of using Protobuf for Apache Kafka

Sam Gozman — Sun, 31 Mar 2024 08:06:52 +0000

In my new blog post, I would like to discuss the nuances I encountered while working on a legacy project that uses Protobuf for messages in Apache Kafka. Specifically, the issues that arose years later from this decision, how the diversity of programming languages in the project complicated things, and how the untimely updating of gRPC tools further magnified the problem. The events and issues described here occurred between 2022 and 2023.

Input data.

We had two monoliths in NodeJS & Ruby, a few dozen containerized apps in Golang and Python, five thousand outdated Protobuf schemas, a salt shaker half full of gRPC protocols, and a whole galaxy of multi-colored serializers, deserializers, structured and unstructured data… and also a quart of Kafka queues, a quart of proto-Kafka messages, a case of critical bugs, a pint of raw VSOP protoc and two dozen highly outdated dependencies glued with DevOps love to over-complication.

Not that we needed all that for the team of 40 devs, but once you get locked into a serious legacy code collection, the tendency is to push it as far as you can.

What was the task?

We needed to update the protoc versions for five reasons:

For NodeJS, protoc and other gRPC tools were marked as deprecated and hadn't been updated in several years. New approaches have moved away from creating auto-generated files to working with Protobuf on the fly.
In NodeJS services, developers wanted to type their work in Protobuf using TypeScript, and the old solution was incapable of generating types.
We needed to access the option syntax in Protobuf (option <option_name> = <value>;) for Golang services (yes, our protoc was that old).
Our version of protoc didn't work on ARM processors, which made development challenging for employees using Macs.
It was one of the bottlenecks the company wanted to resolve in the process of updating old dependencies to reduce the number of CVEs in the images.

The root of the evil lay in the fact that the builder for Go binaries and the builder for NodeJS were moved out of the repositories into an external dependency controlled by DevOps. A dependency without proper versioning that was connected as a git submodule.

So, what does Kafka have to do with it?

The fact is that Protobuf was not only used for describing gRPC messages but also for Kafka messages. Often, the same contract applied in gRPC could be reused in Apache Kafka.

But there are a few key differences between how gRPC and Kafka handle messages:

gRPC:

Typical gRPC messages are always binary, serialized using Protocol Buffers by default.
gRPC tools (like protoc) generate code to serialize and deserialize messages automatically based on the defined Protobuf message types.
While binary is the default and highly recommended format for gRPC messages due to its efficiency, it's possible to use other serialization formats like JSON if needed. However, JSON serialization would require additional manual handling or third-party libraries.

Kafka:

Kafka messages are generally treated as binary payloads.
Kafka itself does not enforce any specific data format on the messages.
While Kafka itself is agnostic to the format of the messages it processes and stores, choosing an appropriate serialization format is crucial. Although Kafka messages are commonly serialized to binary for efficiency (to optimize for both size and speed) - alternate formats like JSON, Avro, or even plain text are entirely viable.

The flexibility in working with Kafka messages leads us to the fact that it does not have its own serializers in the way we are accustomed to with gRPC. The process of serialization and even the decision about its necessity lies entirely with the developers.

The naive approach.

Of course, the most naive approach would have been to simply update the protoc version and see what breaks. This was the quickest and most obvious option, and so we did just that.

What broke? Pretty much everything 🙂

Since I worked exclusively with NodeJS and Golang, I will speak about the problems that surfaced in my stack.

Golang

The syntax for generating Protobuf files changed significantly.
Old syntax example:

  protoc --proto_path="$dir" --go_out=plugins="grpc:$dir" "$file"

New syntax example:

  protoc --go_out="$dir" --go_opt=paths=source_relative --go-grpc_out="$dir" --go-grpc_opt=paths=source_relative "$file"

Now, the grpc-go plugin is used for generating Protobuf. To generate Protobuf with a new plugin, it's now necessary to specify options for go, for example:

  option go_package = "github.com/your_company/your_package";

Some methods from the generated proto files have changed, so code modifications were inevitable.

Without specifying go_package, Protobuf generates with an error. Adding option broke compatibility with the old static generator in NodeJS (it didn't understand the syntax), significantly complicating the task. There was an alternative way to specify the go package through the protoc console option:

  --go_opt=Mprotos/buzz.proto=github.com/your_company/your_package

But no one could get it to work; the command constantly failed with a syntax error. Likely, we made a mistake somewhere, but after unsuccessful attempts, we moved on.

NodeJS

It was necessary to replace the npm library grpc with @grpc/grpc-js. The grpc-tools for NodeJS had not been maintained. It contains an outdated version of protoc.js and protobuf v3.19.1. All replacements for this utility used the same outdated version of protoc & protobuf under the hood, which at the time did not have an ARM build and had several other differences from the latest versions.

We started looking for a current replacement for grpc-tools for NodeJS and concluded that modern approaches in JS no longer offer file generation based on proto. Instead, they allow working on the fly, for example, protobuf.js and @grpc/proto-loader.

Static generation, using grpc-tools and protoc for generation and grpc or @grpc/grpc-js in runtime, is basically deprecated by underlying outdated binaries and lack of GitHub activity. Dynamic generation, on the other hand, using @grpc/proto-loader and protobuf.js has become an industry standard.

After several unsuccessful attempts to painlessly update grpc-tools for NodeJS to stick with the Static approach, we began exploring the Dynamic approach. Initially, everything went smoothly, working with protobuf.js was easy and straightforward. But after a few days of rewriting code to work with gRPC, we suddenly remembered… that our Apache Kafka also uses Protobuf alongside gRPC.

Specifically, the deserializeBinary and serializeBinary methods from the generated files. The amount of code that needed to be rewritten to move from a static to a dynamic approach exceeded ten thousand lines of code, which was beyond our capabilities given the allocated time and the general reluctance to move in that direction. In all the languages that we supported on the project and in all the languages the team had previously worked with, the static approach of generating code based on proto contracts was always applied.

What we have learned.

Updating the approach to working with gRPC on NodeJS will take time, but it doesn't pose any complexity. I've rewritten one contract, and the rest will go smoothly. However, the task with Kafka-protobuf seems to be more challenging.

Serializers and deserializers will have to be invented on our own. Those that can be borrowed from the generated code are not very readable and not at all versatile. As a lazy approach, of course, one could simply copy them and leave them somewhere inside the project, but then any changes to the existing structures would become complicated. We need to create a universal solution. The team did not consider moving away from protobuf in Kafka.

With Golang, things also did not go as smoothly. Due to our failures with passing the package name in go_opt, it will be necessary to add option go_package in all existing contracts, with subsequent updates for consumers (contracts were connected as git submodules). An extremely tedious and monotonous task.

Conclusion.

Untimely updating of dependencies and neglected technical debt lead to long-lasting complex consequences. There are no perfect projects or products without technical debt, but that doesn't mean it shouldn't be addressed.

For a development team that grew from ~20 people to ~40, the decision to develop everything on top of gRPC seems excessive. Despite all my love for gRPC, I believe that maintaining and developing service interactions through gRPC for a small team constantly prolonged the completion time for tasks. Using a simple HTTP router and describing the interaction schema through OpenAPI (e.g., Swagger) would have significantly simplified the development and support of the API for a small team, while also generating excellent documentation for internal consumers.

The latency between services was never a key problem in the speed of our product. The problems were more obvious and expected - a huge amount of N+1 issues, inefficient database operations etc.

As for the use of Protobuf within Apache Kafka, I don't think it was a mistake. Serializing messages into compact binary data had advantages for us, as we had more than once hit Kafka topic size limits. Protobuf proved itself well as a unified format for describing structures between different programming languages.

However, it's worth noting that the team avoided using types in Protobuf messages. All, or nearly all, fields were of type string. So, in the end, I believe we could have just used JSON and sent it as binary, without the hassle (although it would have been less compact than Protobuf). JSON schemas can also be described and stored separately. JSON can be validated against a schema in all programming languages we use with predictable results.

A key lesson from this experience, beyond the well-understood issues surrounding technical debt, is the importance of preparation when choosing to use Protobuf for Kafka messages. It's vital to anticipate the need for custom serialization and deserialization mechanisms well ahead of implementation. Even though Protobuf is not strictly tied to gRPC, most developers only apply it there, which undoubtedly leaves its mark on public libraries. Any use of Protobuf outside the context of gRPC should be considered unconventional, meaning you need to avoid the bottleneck of relying solely on conventional gRPC-associated tools.