DEV Community: Prasad Ekke

Determinism Is a Core Contract in Build Caching

Prasad Ekke — Tue, 04 Aug 2026 14:30:00 +0000

If you have set up a remote build cache and watched the hit rate sit stubbornly low, nondeterministic outputs are one important suspect — but not the only one. Source churn, changing flags, platform differences, toolchain changes, eviction, and incomplete input declarations also cause misses. Nondeterminism is particularly dangerous because it can quietly reduce reuse or make cached results untrustworthy without producing an obvious build error. It is worth understanding exactly how that happens and how to isolate it.

Why determinism is the contract, not a nice-to-have

A build cache works by fingerprinting an action's inputs and using that fingerprint to look up a previously computed output. The entire scheme rests on one assumption: identical inputs produce identical outputs. That is the contract. Break it and one of two bad things happens.

If the cache keys only on inputs — which is how these systems work — and your action is nondeterministic, then the cache will happily hand back an output for those inputs, but not necessarily the one a fresh run would produce. Best case, the variation is cosmetic and harmless. Worst case, you have just served a subtly wrong artifact from cache and you will spend a very confusing afternoon. Either way, the cache is now a source of doubt rather than speed.

Another operational outcome is cascading misses, and that is where determinism stops being abstract.

One bad action poisons everything downstream

Builds are graphs. The output of one action is the input to the next. So consider what happens when a single early action — say, a code generator — embeds the current timestamp in its output. Each time that generator actually runs, the file it produces is byte-for-byte different, so its content hash changes.

Whenever that generator has to execute again — because its entry was evicted, the cache was bypassed, or another declared input changed — it produces a new content digest. Every action that consumes the generated file then sees a different input and misses its previous cache entry. Their outputs feed further actions, so the effect can cascade through the subgraph. You did not merely lose reuse for one action; you invalidated downstream keys even when the semantic content did not need to change.

Where nondeterminism actually comes from

The sources are mundane, which is what makes them easy to miss. The usual suspects:

Timestamps baked into outputs — __DATE__ and __TIME__ macros, build IDs, and the modification times stored inside archive formats like .a, .tar, and .zip.
Absolute paths leaking into outputs — debug info that records the full build directory, __FILE__ expansions, paths embedded by the linker. Build the same code in /home/alice and /home/bob and you get different bytes.
Iteration order — generating code by walking a hash map or set, whose iteration order is unspecified, so the output lines come out shuffled run to run.
Concurrency — actions that assemble output from parallel work without imposing a stable order on the result.
Randomness — UUIDs, unseeded PRNGs, anything that reaches for entropy at build time.
The ambient environment — locale, hostname, username, and stray environment variables that the action reads but never declared as inputs.
Filesystem ordering — relying on readdir order, which varies across filesystems and runs.

None of these throws an error. Each just makes one action's output wiggle, and the cache does the rest of the damage.

How to hunt them down

The basic diagnostic is to execute the same build twice from clean, isolated directories with cache reuse disabled, then compare the outputs bit for bit. Anything that differs deserves investigation. diffoscope is the right tool here — it understands archives, binaries, and debug information and can show not just that two outputs differ but where.

The fixes are well-trodden, because the reproducible-builds community has been fighting this war for years:

Timestamps: honor SOURCE_DATE_EPOCH so embedded dates are fixed rather than "now"; build archives in deterministic mode (tar --sort=name --mtime=..., ar in its deterministic D mode).
Absolute paths: use -ffile-prefix-map / -fdebug-prefix-map to rewrite build paths to a stable placeholder, so debug info does not depend on where you built.
Iteration order: sort before you emit. If codegen walks a map, sort the keys first. Never let an unordered container decide output order.
Environment: pin the locale (LC_ALL=C), and run actions in a sandbox that only exposes declared inputs, so an undeclared environment dependency fails loudly instead of silently varying.
Randomness: seed it, or design it out. A build is not the place for entropy.

Hermeticity is how you keep it fixed

Fixing the determinism bugs you have is half the job. The other half is preventing new ones, and hermetic, sandboxed builds remove many sources of ambient input. A filesystem sandbox can hide undeclared files and tools, and a controlled environment can pin locale and environment variables. Sandboxing alone does not necessarily hide the wall clock, randomness, hostname, kernel details, or every network source; those must be blocked, virtualized, or deliberately normalized. The goal is to make undeclared dependencies fail loudly or become stable rather than silently changing outputs.

Measure it, because it regresses

Determinism is not a one-time cleanup. It rots. Someone adds a code generator, someone embeds a build timestamp "just for diagnostics," and reuse slides without anyone noticing. Treat cache hit rate as a first-class metric, but segment it by action type, platform, branch, and miss reason because determinism is only one contributor. Track reproducibility separately with scheduled rebuild-and-compare tests. When hit rate drops and action keys churn without meaningful input changes, reach for build-twice-and-diff before blaming the cache infrastructure.

That is the real point. Determinism is not purity for its own sake. Along with complete input declarations, compatible platforms and toolchains, and retained cache entries, it is a precondition for trustworthy reuse. Get it wrong and the cache may return questionable outputs or lose reuse across a large subgraph. Get it right and equivalent actions can reliably share results across developers and CI. Determinism is not the whole cache system, but it is part of its correctness contract.

Further reading: the Reproducible Builds documentation

Remote Execution, Explained for People Who Just Run `make`

Prasad Ekke — Tue, 28 Jul 2026 12:02:00 +0000

Most engineers think of the build as something that happens on their laptop. You type a command, the fans spin up, and a few minutes later you have a binary. That model is so ingrained that we rarely question it — until the codebase gets big enough that it quietly collapses.

At a certain scale, the laptop build is too big for one machine, too slow to repeat, and too wasteful to run from scratch on every change. Ten engineers rebuild the same unchanged files ten times before lunch. A clean build takes forty minutes. CI is the bottleneck for the whole team. Remote execution is the answer large codebases reach for, and once you understand it, a lot of otherwise-baffling build-system design suddenly makes sense. Here is the mental model, minus the vendor jargon.

The core idea: the build is a graph of pure functions

Stop thinking of a build as a sequence of commands and start thinking of it as a graph. Each node is an action: a single command plus the exact set of inputs it reads and outputs it produces. Compiling one file is an action. Linking is an action. Generating code from a schema is an action. The edges are dependencies — the linker action consumes the object files the compile actions produced.

Here is the key insight that everything else rests on. If you treat each action as a pure function — same inputs always produce the same outputs — then two things become possible that are not possible with a pile of shell scripts. You can run independent actions anywhere, on any machine, in parallel. And you can cache the result of any action, because identical inputs mean you already know the output.

Remote execution is what you get when you take those two possibilities seriously.

What an action actually is

Concretely, an action is described by its command, the content of every input it depends on, and the environment it runs in. Crucially, inputs are identified by the hash of their content, not their path or timestamp:

Action {
  command:       ["clang", "-c", "parser.c", "-o", "parser.o"]
  input_digests: { "parser.c": sha256(abc1...),
                   "parser.h": sha256(def2...) }
  platform:      { os: linux, arch: x86_64, ... }
}

action_digest = sha256(serialize(Action))

That action_digest is the core lookup key. It fingerprints everything the build system has declared as relevant to the output. Correctness therefore depends on complete input and environment declarations: an undeclared dependency is invisible to the key. The build system computes the digest and asks a shared service one question: have you seen this exact declared action before?

Cache hit: the service returns the output blobs immediately. No compilation happens at all — not locally, not remotely. You just fetched the answer.
Cache miss: the service schedules the action on a remote worker, the worker runs the command in a clean sandbox, uploads the resulting parser.o to a content-addressed store, and records the action-to-output mapping so the next person gets a hit.

Multiply that across a team and a CI fleet and the effect is dramatic. The first engineer to build a given commit pays the cost; everyone else, and every CI run on that commit, gets cache hits. Work is done once and shared.

The two pieces of machinery

Underneath, remote execution rests on two services that are simpler than they sound.

The first is a content-addressed store (the CAS). It is a blob store keyed by the hash of the content. Put bytes in, get a hash back; present a hash, get the bytes. Because the key is the content's fingerprint, identical files can be deduplicated and a retained blob cannot become stale under the same digest. Storage policies may evict blobs, so clients still have to handle missing content. Source files, intermediate objects, and final binaries can all live here.

The second is the action cache: a map from action_digest to the set of output digests it produced. That is the lookup that turns "compile this" into "you already compiled this, here are the bytes."

Bazel and Buck2 both support the common Remote Execution API for these services, which allows compatible deployments to point either client at the same class of remote backend. You do not need to know the protocol to use it, but it helps to know it exists: the build tool on your laptop coordinates the graph while remote workers and shared caches handle much of the expensive execution and storage.

Why this reshapes how build systems are designed

Once you see the build as a graph of cacheable pure functions, the design decisions of modern build systems stop looking arbitrary.

Why does Bazel make you declare every input and output explicitly, instead of letting a command read whatever files it wants? Because the action digest has to capture every input. An undeclared input is an input the cache cannot see, which means the cache will hand you a stale result. Why the obsession with hermetic, sandboxed actions? Because a worker on the other side of the cluster has to be able to run your action with nothing but the declared inputs and get the same answer your laptop would. The strictness that feels like bureaucracy when you are writing a BUILD file is the exact thing that makes the action a pure function — and a pure function is the thing you can distribute and cache.

Where it gets hard

It is not free, and the failure modes are worth knowing before you adopt it.

The network becomes the new bottleneck. You have traded local CPU for cache lookups and blob transfers, so a slow link or a far-away CAS can make a "cached" build feel slower than a local one. Large inputs and outputs strain the CAS and the network; build graphs with giant artifacts need care. And the whole edifice rests on one assumption that deserves its own discussion: that actions are actually deterministic. The moment an action produces different output from identical inputs, the cache starts handing out wrong or useless answers, and the value proposition quietly inverts.

That last point is important enough that it is the subject of its own post. For now, the mental model is the takeaway: a build is a graph of actions, each action is a pure function fingerprinted by its inputs, and remote execution is just the infrastructure for running those functions anywhere and remembering their answers. Once that clicks, the rest of the modern build world — the explicit inputs, the sandboxing, the content hashing — reads as the obvious consequence it is.

Go Slice Initialization Gotchas: The Hole-Creating Bug I Keep Seeing (And a Linter That Catches It)

Prasad Ekke — Sun, 26 Jul 2026 02:30:00 +0000

Go slices are simple on the surface. Under the hood, the interaction between length, capacity, and append has a specific failure mode I've been bitten by more than once — and seen in enough code reviews to write about. The bug is invisible at compile time, often invisible in tests, and can corrupt data silently.

The three ways to initialize a slice

Before the bug, the mechanics. make for slices takes up to three arguments:

make([]T, length)
make([]T, length, capacity)

These behave very differently:

// Length 0, capacity 100
// Content: []
// append() fills from index 0
a := make([]int, 0, 100)

// Length 5, capacity 5
// Content: [0 0 0 0 0]
// Indexed assignment fills existing slots
b := make([]int, 5)

// Length 5, capacity 10
// Content: [0 0 0 0 0]
// Indexed assignment fills existing slots 0–4
// append() fills from index 5 onward
c := make([]int, 5, 10)

The key rule: append always starts after the last element, at index len(slice). It does not fill from the beginning. It does not know or care what's in the existing slots.

s := make([]int, 5, 10)
fmt.Println(s)       // [0 0 0 0 0]
fmt.Println(len(s))  // 5
fmt.Println(cap(s))  // 10

s = append(s, 42)
fmt.Println(s)       // [0 0 0 0 0 42]  ← 42 is at index 5, not 0

This is correct, expected behavior. The problem arises when you mix the two initialization patterns during a refactor.

The bug: optimization that creates holes

Here's the sequence of events. You have code that initializes a slice with a fixed length and fills it by index:

// Version 1: works correctly
results := make([]int, 5)

for i, job := range jobs {
    results[i] = process(job)
}
// results: [r0 r1 r2 r3 r4]

This is fine. Five jobs, five slots, indexed assignment, everything lines up.

Later, someone (possibly future-you) decides to optimize memory allocation by pre-allocating capacity for growth:

// Version 2: "optimization"
results := make([]int, 5, 10)  // added capacity

The loop still works. Indexed assignment into results[i] fills positions 0–4. No change in behavior yet.

Now, a separate change: the indexed loop gets refactored. Maybe the loop structure changes, maybe someone decides append is more idiomatic, maybe the jobs slice changes to a channel. The replacement looks like this:

// Version 3: refactored loop using append
results := make([]int, 5, 10)

for _, job := range jobs {
    results = append(results, process(job))
}

This compiles. This runs. And this produces:

// results: [0 0 0 0 0 r0 r1 r2 r3 r4]

Five zero-value holes at the front, followed by the actual results. The slice now has length 10 instead of 5. Any code downstream that iterates results processes 10 elements instead of 5, with the first 5 being garbage zeros.

If the downstream code is a JSON response, you send extra zeros to clients. If it's an aggregation, your totals are wrong. If it's a database write, you insert rows you didn't intend to.

Why this is hard to catch

No compile error. Both indexed assignment and append are valid operations on a []int. The compiler has nothing to complain about.

Tests often miss it. If your test asserts len(results) == 5, it fails — but if it only checks specific indices or iterates the whole slice, the zeros look like valid data. A test against a jobs list where process legitimately returns 0 for some inputs will pass completely.

The change looks like a safe refactor. Changing results[i] = x to results = append(results, x) seems equivalent to anyone who hasn't internalized how append interacts with a non-zero initial length. The diff is small and the intent looks the same.

The capacity change seems unrelated. The make([]int, 5) to make([]int, 5, 10) change is in a different commit, possibly by a different person. The connection between that change and the later append refactor isn't obvious.

The three clean patterns — pick one and be consistent

The root cause is mixing patterns that don't compose. Each of these is correct on its own:

Pattern 1: Length 0, append only

// Correct: length 0, append fills from the start
results := make([]int, 0, len(jobs))

for _, job := range jobs {
    results = append(results, process(job))
}
// results: [r0 r1 r2 r3 r4]

Use this when you're building a slice incrementally and don't need indexed access during construction. The capacity hint len(jobs) avoids reallocations without creating holes.

Pattern 2: Length N, indexed assignment only

// Correct: length N, indexed assignment fills all slots
results := make([]int, len(jobs))

for i, job := range jobs {
    results[i] = process(job)
}
// results: [r0 r1 r2 r3 r4]

Use this when you're processing a collection with known size and need random-access writes (out-of-order filling, parallel writes with sync, etc.).

Pattern 3: Length N with capacity, indexed assignment only

// Correct: length N, capacity M, indexed assignment for first N slots
// append for subsequent elements
results := make([]int, len(jobs), len(jobs)*2)

for i, job := range jobs {
    results[i] = process(job)
}

// Later appends go to indices N onward — intentional
results = append(results, extraResults...)

Use this only when you genuinely need both: a pre-filled region AND growth capacity. This is the rarest case and the one that most needs a comment explaining the intent.

The linter: makezero

The makezero linter catches exactly this class of mistake. It flags append calls on slices that were initialized with non-zero length, giving you a chance to examine whether the initialization and the append are both intentional.

go install github.com/ashanbrown/makezero@latest
makezero ./...

Or via golangci-lint (recommended — runs alongside your other linters):

# .golangci.yml
linters:
  enable:
    - makezero

What it flags:

results := make([]int, 5, 10)
results = append(results, 42)
// makezero: append to slice results initialized with non-zero length at line N

What it does NOT flag (both are intentional patterns):

// Pattern 1: length 0, append is fine
a := make([]int, 0, 10)
a = append(a, 42)  // ✅ no warning

// Pattern 2: length N, no append
b := make([]int, 5)
b[0] = 42  // ✅ no warning

The linter doesn't tell you your code is wrong — it tells you that you have a combination that might be wrong and deserves a look. That's the right level of signal. The version 3 code from the bug scenario above would have triggered this warning, prompting a review of the initialization before the append refactor shipped.

Adding an explicit comment when you genuinely mix patterns

If you have a legitimate reason to initialize with non-zero length and later append, say so:

// Pre-fill the first slot with a sentinel value.
// Actual results are appended starting at index 1.
results := make([]Result, 1, len(jobs)+1)
results[0] = Result{sentinel: true}

for _, job := range jobs {
    results = append(results, process(job)) //nolint:makezero // intentional: sentinel at index 0
}

A comment here, paired with a linter suppression (//nolint:makezero), tells reviewers and future maintainers that the mixed pattern is deliberate. Without it, the next person to read the code — or the next refactor — will make the same assumption the bug relied on.

Quick mental checklist for slice initialization

Before writing make([]T, n, m) ask:

Will I use indexed assignment or append? Pick one. If indexed, length = final size. If append, length = 0.
Do I need pre-filled slots AND append? Only then use make([]T, n, m) with n > 0. Add a comment.
Is the capacity hint right? make([]T, 0, len(input)) is almost always the right pattern for transforming one slice into another.

The simplest version of this rule: if you're using append, start with length 0.

Summary

make([]T, 0, cap) — Use with: append only. Risk if mixed: none — this is the safe default.

make([]T, n) — Use with: indexed assignment only. Risk if mixed: none if consistent.

make([]T, n, m) — Use with: indexed assignment for the first n, append after. Risk if mixed: high — easy to introduce holes during a refactor.

The bug isn't exotic. It's the interaction of two individually correct operations that become incorrect when combined without awareness of how append uses len, not cap, to decide where to write. The makezero linter closes the gap that the compiler leaves open.

This is part of a series on Go patterns and production gotchas. Previous posts cover concurrency mistakes, typed constants, context propagation, and profiling.

Why I Always Specify Types for Constants in Go (And Name the Unit Too)

Prasad Ekke — Thu, 23 Jul 2026 02:30:00 +0000

Go's untyped constants are one of the language's more elegant features — until they silently do something you didn't intend. This post is about a specific class of bug that untyped numeric constants can introduce, why the fix is simpler than you might think, and a naming convention that makes the intent impossible to misread.

What untyped constants actually are

When you declare a constant in Go without an explicit type, it becomes an untyped constant. It doesn't have a fixed type yet — it carries a kind (integer, float, string, bool) and a value, but its actual type is deferred until it's used.

const Timeout = 5        // untyped integer constant
const Ratio  = 0.95      // untyped float constant
const Label  = "worker"  // untyped string constant

This is genuinely useful. An untyped integer constant can be assigned to any integer type without an explicit conversion:

var a int     = Timeout  // fine
var b int64   = Timeout  // fine
var c float64 = Timeout  // fine

The compiler resolves the type at the point of use. This is why you can write make([]byte, 1024) without casting — 1024 is an untyped integer constant that fits whatever integer type make needs.

So far so good. The problem is that this implicit resolution extends to named types built on top of numeric primitives — and some of those named types carry semantic meaning that a bare number does not.

The bug: time.Duration and the silent conversion

Here's the situation I ran into. I had a constant for a worker pool job timeout:

const JobTimeout = 5

And a function that used it:

func startWorker(timeout time.Duration) {
    timer := time.NewTimer(timeout)
    // ...
}

startWorker(JobTimeout)

This compiles. No warning. No error. Go happily passes 5 as a time.Duration.

The problem: time.Duration is defined as type Duration int64, measured in nanoseconds. So JobTimeout = 5 becomes a 5-nanosecond timeout. The worker timer fires almost instantly. In testing against a fast local environment, the timeout path was easy to miss. Under load with a slow downstream, every job timed out almost immediately.

The intent was 5 minutes. What actually ran was 5 nanoseconds.

// What I meant
5 * time.Minute   // 300,000,000,000 nanoseconds

// What Go silently did
time.Duration(5)  // 5 nanoseconds

The compiler had no way to know my intent because I gave it no information about units. I just said "5". Go resolved it to time.Duration(5) and moved on.

Why this specific bug is hard to catch

Two things make it dangerous:

It compiles cleanly. There's no type mismatch because untyped integer constants are assignable to any integer-based named type. The type system doesn't protect you here.

It's invisible in fast environments. A 5-nanosecond timer almost never fires before the work completes on a modern CPU. The bug only manifests under conditions (slow network, slow DB, high load) that don't appear in unit tests running against local mocks.

This is the worst class of bug: no compile error, no test failure, silent wrong behavior in production.

This is not just a `time.Duration` problem

While revisiting this post, I found several examples of the same underlying issue showing up in different forms. They are useful because they make the problem feel less like a personal mistake and more like a real edge in Go's constant model.

The Go spec is precise about this behavior: operations on untyped constants keep producing untyped constants, and when different untyped numeric kinds are mixed, the result follows the spec's kind-ordering rules. That is elegant, but it means expression shape matters.

One Reddit thread demonstrates this with a tiny expression:

0.5 * 1/2 == 1/2 * 0.5

Those look mathematically equivalent, but in Go they do not behave the same way. In 1/2 * 0.5, the 1/2 part is evaluated as an untyped integer constant first, so it becomes 0 before the multiplication by 0.5. The original poster described hitting the same pattern inside a numeric algorithm, where a small-looking constant mistake created a much larger real-world error.

That is the floating-point cousin of the time.Duration bug. In both cases, the compiler is following the language rules. The code is the thing that failed to say what it meant.

There is also an open golangci-lint issue requesting an untypedconst linter. The requested check would warn when an untyped constant is passed where a defined type is expected. That is exactly the boundary this post is about:

startWorker(5)        // untyped constant crosses into time.Duration
startWorker(5 * time.Minute)

The first call is legal because 5 can become a time.Duration. The second call is clear because the unit is part of the expression.

Other issues show adjacent sharp edges:

A Go compiler issue discusses a huge untyped integer constant that eventually produced a constant overflow error, even though untyped constants are often described as living in an "ideal" numeric space.
An older go/types issue involved subtle behavior around an untyped integer constant in a shift and conversion expression.
A go-ora issue shows a real dependency hitting linter failures for 2147483648 (untyped int constant) overflows int.
Another Reddit thread calls out loss of precision when using untyped constants, which is the same family of problem: the value is legal, but the intended numeric semantics are easier to lose than the code suggests.

The lesson I take from these examples is not "avoid untyped constants everywhere." That would be an overcorrection. Untyped constants are one of the reasons Go numeric literals are pleasant to use.

The lesson is narrower:

When a constant crosses a semantic boundary, make the semantics explicit.

Named types like time.Duration, enum-like custom types, byte counts, ports, limits, and unit-bearing values are semantic boundaries. A bare number crossing one of those boundaries deserves suspicion.

The fix: give the constant the correct type — time.Duration

The fix is to declare the constant with the type it is meant to have — time.Duration — and to write its value with an explicit unit:

// ❌ Untyped int — coerced to time.Duration(5), i.e. 5 nanoseconds
const JobTimeout = 5

// ✅ Typed as time.Duration, value written with its unit
const JobTimeout time.Duration = 5 * time.Minute

There is a subtlety here that is the whole point: two separate things are happening on that second line, and you need both.

First, the type is time.Duration. That is what stops the constant from being silently accepted where an int or some other numeric type is expected.

Second, the value is 5 * time.Minute, not 5. That is what makes the magnitude correct — 300,000,000,000 nanoseconds instead of 5.

Specifying the type alone does not save you. const JobTimeout time.Duration = 5 compiles, is type-safe, and is still 5 nanoseconds — the original bug, now wearing a type. For a unit-bearing type like time.Duration, "specify the correct type" and "write the unit" are the same discipline: the only sensible way to express a time.Duration magnitude is with a unit, and once you do, the constant is already a time.Duration whether or not you add the annotation.

So why annotate at all, if 5 * time.Minute is already a time.Duration? For intent at the declaration and to enforce the discipline in review. const JobTimeout time.Duration = 5 * time.Minute states the contract plainly — this is a duration. It does not make = 5 a compile error, but it makes a bare number visibly suspicious because the type and the missing unit no longer agree with the intent.

Now pass the typed constant to a function that does not accept time.Duration:

func startWorker(timeout time.Duration) { ... }
func processJob(count int) { ... }

startWorker(JobTimeout)  // ✅ fine
processJob(JobTimeout)   // ❌ compile error: cannot use JobTimeout (constant 300000000000 of type time.Duration) as int value in argument to processJob

The typed constant carries its semantic meaning into every use site, and the compiler enforces it.

Going one step further: encode the unit in the name

Even with a typed constant, consider what happens when someone reads just the constant name at a call site:

startWorker(JobTimeout)

They know it's a time.Duration from the type, but they have to jump to the declaration to know what 5 minutes means in context. Is that a reasonable timeout? Too long? Too short?

The convention I've adopted: put the unit in the constant name when the value is a pure magnitude.

// Unit in the name — intent readable at every call site
const JobTimeoutMinutes time.Duration = 5 * time.Minute
const RetryDelaySeconds time.Duration = 30 * time.Second
const CacheTTLHours     time.Duration = 24 * time.Hour

Now every call site is self-explanatory:

startWorker(JobTimeoutMinutes)
scheduleRetry(RetryDelaySeconds)
setCache(key, value, CacheTTLHours)

A reader doesn't need to look at the declaration. They know the unit from the name, and if they do check the declaration, the type and the multiplier confirm it.

The pattern generalises beyond time.Duration

time.Duration is the most common place this matters, but the same principle applies to any named type that wraps a primitive:

// Byte sizes
const MaxPayloadBytes  int64  = 10 * 1024 * 1024  // 10 MB
const ChunkSizeKB      int64  = 64 * 1024

// Counts that could be confused with durations or sizes
const WorkerPoolSize   int    = 10
const MaxRetryCount    int    = 3

// Custom named types in your own code
type Port      int
type TimeoutMS int64

const HTTPSPort    Port      = 443
const QueryTimeout TimeoutMS = 500

The rule of thumb: if a constant is a magnitude of something — time, bytes, counts tied to a specific resource — give it an explicit type and encode the unit in the name. If it's a dimensionless scalar (a ratio, a flag, a pure count), the type is less critical but still worth being explicit about.

What about typed constants and iota?

One more case where explicit types matter: iota-based enumerations. Without a type, the values are just untyped integers and can be mixed accidentally.

// ❌ Untyped iota — nothing prevents mixing State and Priority values
const (
    StateIdle    = iota // 0
    StateRunning        // 1
    StateDone           // 2
)

const (
    PriorityLow    = iota // 0
    PriorityNormal        // 1
    PriorityHigh          // 2
)

func setWorkerState(s int) { ... }

setWorkerState(PriorityHigh)  // compiles, silently wrong

With explicit types, the compiler catches the mix-up:

// ✅ Typed iota — compiler enforces correct usage
type WorkerState int
type Priority    int

const (
    StateIdle    WorkerState = iota
    StateRunning
    StateDone
)

const (
    PriorityLow    Priority = iota
    PriorityNormal
    PriorityHigh
)

func setWorkerState(s WorkerState) { ... }

setWorkerState(PriorityHigh)  // ❌ compile error: cannot use PriorityHigh (constant 2 of type Priority) as WorkerState value in argument to setWorkerState

The type system now enforces semantic correctness, not just structural compatibility.

Summary

const Timeout = 5 passed to a time.Duration param — Risk: silent nanosecond conversion. Fix: const TimeoutMinutes time.Duration = 5 * time.Minute.

const Size = 1024 passed to a byte-count param — Risk: ambiguous unit. Fix: const MaxSizeBytes int64 = 1024.

Untyped iota constants mixed across enumerations — Risk: silent wrong value. Fix: an explicit named type per enumeration.

Go's untyped constants exist for good reasons — they make numeric literals flexible and reduce casting noise. But that flexibility comes with a cost: the compiler can't infer your intent, only your value. When a constant represents a quantity with a unit, make the unit explicit in both the type and the name. The compiler becomes your ally instead of a silent bystander.

The bug that prompted this post took me an embarrassingly long time to track down. A 5-nanosecond timer that "worked fine in testing" is the kind of thing that ages you. Explicit types and unit-named constants are cheap insurance against it.

References and related examples

This is part of a series on Go patterns and production gotchas. Previous posts cover concurrency mistakes, context propagation, interface design, and profiling with pprof.

Container-Aware Resource Management in Go: The Problem, Go 1.25’s Fix, and What’s Still Missing

Prasad Ekke — Tue, 21 Jul 2026 16:35:24 +0000

If you’ve deployed a Go service to Kubernetes and set careful CPU and memory limits on your pod, there’s a good chance your Go runtime was ignoring them entirely — spawning too many threads, running GC on the wrong schedule, and getting throttled or OOM-killed as a result.

Go 1.25 fixed half of this problem natively. This post explains both halves: what was broken, what’s fixed, what’s still not fixed, and what to do if you can’t upgrade yet.

The problem: Go doesn’t know it’s in a container

When a Go program starts, the runtime reads the host machine’s CPU count and sets GOMAXPROCS — the number of OS threads that can run Go code in parallel — to that number. On a 64-core Kubernetes node, your Go service thinks it has 64 CPUs available, even if your pod’s resource limit is cpu: "2".

The same problem exists for memory. Go’s garbage collector uses a target heap growth ratio (controlled by GOGC) to decide when to trigger collection. By default, it allows the heap to double before collecting. If your container has a 512 MB memory limit and the Go runtime doesn’t know about it, the GC may allow heap growth that pushes your process past the cgroup limit before it triggers a collection. The result: an OOM kill.

These are two distinct problems with different severities.

Problem 1: GOMAXPROCS ignores cgroup CPU limits

What actually happens

Container runtimes use Linux cgroups to enforce CPU limits. A Kubernetes cpu: "2" limit translates into a cgroup CPU bandwidth quota: the container gets 2 CPU-seconds of time per second, spread across all threads.

If GOMAXPROCS is set to 64 (the node’s core count), your Go service creates up to 64 OS threads competing for 2 CPUs worth of time. The kernel enforces the quota by throttling — when the cgroup exhausts its CPU time budget for the current period (typically 100ms), the kernel pauses all threads in the container until the next period begins.

Throttling is brutal. It’s not gentle scheduling slowdown. It’s a hard pause of your entire process. Under this scenario:

A p99 latency that should be 10ms can spike to hundreds of milliseconds
The Go GC, which runs its own goroutines, can exhaust the CPU budget and cause application threads to stall
Context switching overhead multiplies — 64 threads competing for 2 CPUs thrash the scheduler

Node: 64 cores
Pod CPU limit: 2 cores
GOMAXPROCS (before 1.25): 64   ← runtime sees the host
Effective CPU available: 2     ← kernel enforces this

Result: 64 threads, 2 CPUs, aggressive throttling

Real-world benchmarks have shown CPU wait times reaching tens of seconds under this misconfiguration on high-core-count nodes.

The Go 1.25 fix

Go 1.25 makes GOMAXPROCS container-aware by default on Linux. At startup, the runtime reads the cgroup CPU bandwidth limit and sets GOMAXPROCS to the lower of the usual non-container default and the container’s CPU limit. Fractional CPU quotas are rounded up because GOMAXPROCS is an integer.

Node: 64 cores
Pod CPU limit: 2 cores
GOMAXPROCS (Go 1.25): 2   ← runtime reads the cgroup

Two additional behaviors from the official release notes:

Dynamic updates: GOMAXPROCS is now periodically re-evaluated at runtime. If Kubernetes adjusts your pod’s CPU limit on the fly (via VPA or manual edit), the Go runtime will pick up the change automatically — no restart needed.

Opt-out: both behaviors can be disabled via GODEBUG:

# Disable cgroup CPU awareness at startup
GODEBUG=containermaxprocs=0

# Disable periodic GOMAXPROCS updates
GODEBUG=updatemaxprocs=0

Manual override still works: if you set GOMAXPROCS via the environment variable or a runtime.GOMAXPROCS() call, Go 1.25’s automatic behavior is completely disabled. Your explicit value takes precedence.

One edge case worth noting: Go 1.25 respects cpu: limits in Kubernetes but not cpu: requests. The cgroup CPU quota corresponds to the limit, not the request. If you set requests but not limits (a common pattern), Go 1.25’s container awareness has nothing to read and defaults to the host core count.

resources:
  requests:
    cpu: "0.5"    # Go 1.25 ignores this
  limits:
    cpu: "2"      # Go 1.25 reads this ✅

Problem 2: GOMEMLIMIT still doesn’t read cgroup memory limits (not fixed in 1.25)

This is the important accuracy note. Go 1.25 only fixed CPU (GOMAXPROCS). Memory awareness via GOMEMLIMIT is a separate proposal (golang/go issue #75164) that has not yet shipped. As of Go 1.25, GOMEMLIMIT still defaults to “unlimited” — the Go runtime has no built-in mechanism to read the container’s memory limit from cgroups and apply it automatically.

GOMEMLIMIT (introduced in Go 1.19) is a soft memory limit for the Go runtime. When heap usage approaches the limit, the GC becomes more aggressive to stay within it. Without it set to your container’s memory limit, the GC may allow heap growth that triggers the kernel OOM killer before the GC intervenes.

Container memory limit: 512 MB
GOMEMLIMIT default: math.MaxInt64 (unlimited)

Go heap target (GOGC=100): doubles before collecting
If live heap grows past 512 MB: OOM kill before GC triggers

With GOMEMLIMIT=450MB:
GC triggers more aggressively as heap approaches 450 MB
OOM kills become rare

Workaround 1: Set GOMEMLIMIT as an environment variable

The simplest approach. Set it in your Kubernetes deployment to slightly below your memory limit — leaving headroom for non-heap memory (stack space, runtime overhead, cgo memory):

# Kubernetes pod spec
env:
  - name: GOMEMLIMIT
    value: "450MiB"   # 512MB limit minus ~12% headroom
resources:
  limits:
    memory: "512Mi"

Or use the Kubernetes Downward API to derive it from the actual limit:

env:
  - name: GOMEMLIMIT
    valueFrom:
      resourceFieldRef:
        containerName: my-service
        resource: limits.memory
        divisor: "1"   # bytes

Caveat: the Downward API injects the raw memory limit, giving the runtime 0% headroom. For most services this is fine — GOMEMLIMIT is a soft limit; the runtime won’t OOM itself trying to hit it. But memory-mapped files, cgo allocations, and runtime overhead sit outside the Go heap and don’t count against GOMEMLIMIT. Leave 10–15% headroom if your service uses any of these.

Workaround 2: Set GOMEMLIMIT in code at startup

import "runtime/debug"

func main() {
    // Set to 90% of container memory limit
    // Replace with your actual limit or read from env
    const memLimitBytes = 450 * 1024 * 1024 // 450 MB
    debug.SetMemoryLimit(memLimitBytes)

    // ... rest of main
}

This works but hardcodes the value, which is brittle across environments.

Third-party libraries: pre-1.25 CPU fix and memory fix

If you can’t upgrade to Go 1.25, or if you need memory awareness that 1.25 doesn’t provide, two libraries solve these problems cleanly.

uber-go/automaxprocs — CPU (pre-1.25)

The library that Go 1.25 made redundant for most users. A single blank import sets GOMAXPROCS to the container’s CPU limit at startup by reading cgroup v1 and v2 files.

import _ "go.uber.org/automaxprocs"

That’s the entire integration. The init() function runs at startup, reads the cgroup CPU quota, calculates the effective CPU count, and calls runtime.GOMAXPROCS(). If no cgroup limit is found, it leaves GOMAXPROCS at the default.

When to still use it: if you’re on Go 1.24 or earlier. Go 1.25 reads both cgroup v1 and v2, so on Go 1.25+ you’re covered natively regardless of cgroup version.

Migration path: upgrade to Go 1.25 and remove the import. Several major projects have done exactly this.

KimMachineGun/automemlimit — Memory (still relevant on all versions)

Since GOMEMLIMIT container awareness isn’t in Go 1.25, this library fills the gap. It reads the cgroup memory limit and sets GOMEMLIMIT automatically at startup, with configurable headroom.

import _ "github.com/KimMachineGun/automemlimit"

Default behavior: sets GOMEMLIMIT to 90% of the cgroup memory limit, leaving 10% headroom for non-heap allocations. The ratio is configurable:

import "github.com/KimMachineGun/automemlimit/memlimit"

func main() {
    // Set to 85% of container memory limit
    memlimit.SetGoMemLimitWithOpts(
        memlimit.WithRatio(0.85),
        memlimit.WithProvider(memlimit.FromCgroup),
    )
    // ...
}

Supports cgroup v1 and v2. Falls back gracefully if no cgroup limit is found (leaves GOMEMLIMIT unchanged).

tprasadtp/go-autotune — Both CPU and memory, cgroup v2 + Windows

A newer library that handles both GOMAXPROCS and GOMEMLIMIT in a single import, with support for Windows via the Job Objects API in addition to Linux cgroups:

import _ "github.com/tprasadtp/go-autotune"

Constraint: cgroup v2 only on Linux. If you’re on older infrastructure still running cgroup v1 (pre-Kubernetes 1.25, RHEL 8 and below), use automaxprocs + automemlimit instead.

What to use and when

Go 1.25+, Linux, Kubernetes 1.25+ — CPU: built-in. Memory: use automemlimit or set GOMEMLIMIT manually.

Go 1.24 or earlier — CPU: automaxprocs. Memory: automemlimit or the GOMEMLIMIT env var.

Go 1.25+, cgroup v2, need memory automation — CPU: built-in. Memory: automemlimit or go-autotune.

Go 1.25+, want explicit control — CPU: GODEBUG=containermaxprocs=0 plus a manual value. Memory: GOMEMLIMIT env var.

Non-Linux (Windows containers) — CPU: go-autotune. Memory: go-autotune.

Practical Kubernetes setup for a Go 1.25 service

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: my-service
          image: my-service:latest
          resources:
            requests:
              cpu: "500m"
              memory: "256Mi"
            limits:
              cpu: "2"       # Go 1.25 reads this for GOMAXPROCS
              memory: "512Mi"
          env:
            # GOMAXPROCS: handled automatically by Go 1.25
            # GOMEMLIMIT: still needs to leave headroom for non-heap memory
            - name: GOMEMLIMIT
              value: "450MiB"  # 512Mi limit minus roughly 12% headroom

Alternatively, in main.go, use automemlimit instead of setting GOMEMLIMIT manually:

package main

import (
    _ "github.com/KimMachineGun/automemlimit" // sets GOMEMLIMIT to 90% of cgroup limit
)

func main() {
    // GOMAXPROCS handled by Go 1.25 runtime automatically.
    // GOMEMLIMIT handled by automemlimit with its default 10% headroom.
    // ...
}

Summary

The container resource problem in Go has two parts: CPU throttling from a mismatched GOMAXPROCS, and OOM kills from a GC that doesn’t know its memory budget. Go 1.25 solves the CPU side natively and dynamically. The memory side remains a manual step — either via the GOMEMLIMIT environment variable, the Kubernetes Downward API, or automemlimit.

If you’re running Go 1.25 on Kubernetes with CPU limits set, you can remove automaxprocs and get the same behavior for free. For memory, automemlimit remains the cleanest solution until the Go team ships the cgroup memory proposal.

This post is part of a series on Go in production. Previous posts cover goroutine leaks, context propagation, and profiling with pprof.
Next: Async Rust vs Go goroutines — the mental model shift.

Profiling a Go Service in Production: pprof in 10 Minutes

Prasad Ekke — Tue, 30 Jun 2026 15:09:51 +0000

Go ships with a profiler built in. No third-party tools, no agents to install, no instrumentation to add upfront. If your service imports net/http/pprof, you can profile it live in production right now — CPU usage, memory allocations, goroutine counts, blocking operations. The data is available over HTTP, readable with standard Go tooling.

Most engineers know pprof exists. Fewer have actually used it under pressure, on a real service, to find a real problem. This post walks through the mechanics — how to enable it, how to collect a profile, how to read a flame graph — and then shows the three classes of problems it catches most often.

Enabling pprof

For an HTTP service, one import is all it takes:

import (
    "log"
    "net/http"

    _ "net/http/pprof" // registers handlers on DefaultServeMux
)

func main() {
    // Your service setup...

    // pprof endpoints are now available on DefaultServeMux
    go func() {
        log.Println(http.ListenAndServe("localhost:6060", nil))
    }()

    // ... rest of main
}

The blank import registers the pprof HTTP handlers automatically. They’re available at:

GET /debug/pprof/ — index of available profiles
GET /debug/pprof/goroutine — all current goroutines
GET /debug/pprof/heap — memory allocations
GET /debug/pprof/profile?seconds=30 — CPU profile (30-second sample)
GET /debug/pprof/block — goroutine blocking events
GET /debug/pprof/mutex — mutex contention

Security note: bind pprof to localhost only, or a private interface. Never expose it on your public port. If your service runs in Kubernetes, use kubectl port-forward to reach it.

For services not using DefaultServeMux, register the handlers explicitly:

mux := http.NewServeMux()
mux.HandleFunc("/debug/pprof/", pprof.Index)
mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
mux.HandleFunc("/debug/pprof/trace", pprof.Trace)

Collecting a CPU profile

With the service running under load (profiling a quiet service tells you nothing useful), collect a 30-second CPU sample:

go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

This downloads the profile and drops you into an interactive shell. Or skip the shell and go straight to a flame graph:

# Collect the profile to a file
curl -o cpu.prof "http://localhost:6060/debug/pprof/profile?seconds=30"

# Open the flame graph in a browser
go tool pprof -http=:8080 cpu.prof

The -http flag serves a web UI with multiple views. The flame graph is the most useful starting point.

Reading a flame graph

A flame graph shows where your program spends time. Each horizontal bar is a function. Width represents time — wider means more CPU. The vertical axis is the call stack — functions lower down called the ones above them.

|          processJob          |     |  encode  |
|       fetchFromDB      |other|
|         sql.Query            |
|          net.Read            |

You’re looking for wide bars near the top — those are the functions consuming the most CPU. If json.Marshal is unexpectedly wide, you’re spending a lot of time serializing. If runtime.mallocgc is wide, you’re allocating heavily. If sync.Mutex.Lock is wide, you have lock contention.

The three most common findings in worker pool services:

Problem 1: Excessive allocations

Symptom: runtime.mallocgc is prominent in the CPU profile. Heap profile shows high allocation rate with short-lived objects.

# Collect a heap profile
curl -o heap.prof "http://localhost:6060/debug/pprof/heap"
go tool pprof -http=:8080 heap.prof

In the heap profile, switch to “alloc_objects” or “alloc_space” view (not “inuse” — that shows what’s live, not what’s being allocated). Look for hot allocation sites.

Common cause in worker pools: allocating a new buffer or slice on every job iteration.

// ❌ Allocates a new byte slice for every job
func processJob(ctx context.Context, job Job) error {
    buf := make([]byte, 4096)
    // use buf...
    return nil
}

// ✅ Reuse buffers with sync.Pool
var bufPool = sync.Pool{
    New: func() interface{} {
        b := make([]byte, 4096)
        return &b
    },
}

func processJob(ctx context.Context, job Job) error {
    bufPtr := bufPool.Get().(*[]byte)
    defer bufPool.Put(bufPtr)
    buf := *bufPtr
    // use buf...
    return nil
}

sync.Pool maintains a pool of reusable objects. Objects not retrieved before the next GC cycle are collected, so it doesn’t prevent garbage collection — it reduces the rate of allocation, which reduces GC pressure.

Problem 2: Goroutine leak

Symptom: goroutine count grows over time and never decreases.

# Check current goroutine count
curl "http://localhost:6060/debug/pprof/goroutine?debug=1"

The debug=1 parameter returns a text listing of all goroutines with stack traces. Look for hundreds or thousands of goroutines all blocked at the same location — that’s your leak.

goroutine 1042 [chan receive, 3 minutes]:
main.worker()
    /app/worker.go:45 +0x68
created by main.startWorkers
    /app/pool.go:23 +0x4c

goroutine 1043 [chan receive, 3 minutes]:
...
(997 more like this)

A thousand goroutines blocked on chan receive for 3 minutes means the jobs channel was never closed. The fix is covered in the concurrency mistakes post in this series — the producer must own close(jobs) via defer.

You can also check the goroutine count programmatically:

// In your health check endpoint
func healthHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "goroutines: %d\n", runtime.NumGoroutine())
}

If that number grows continuously over hours of runtime and never stabilizes, you have a leak.

Problem 3: Lock contention

Symptom: throughput is lower than CPU utilization suggests it should be. Workers are running but not making progress.

Mutex profiling is off by default because it has overhead. Enable it from inside the process only while investigating contention:

import "runtime"

runtime.SetMutexProfileFraction(1)
defer runtime.SetMutexProfileFraction(0)

Then collect the mutex profile:

curl -o mutex.prof "http://localhost:6060/debug/pprof/mutex"
go tool pprof -http=:8080 mutex.prof

Common cause in worker pools: a shared results map protected by a single mutex, written to by all workers simultaneously.

// ❌ All workers contend on one mutex
type ResultCollector struct {
    mu      sync.Mutex
    results map[string]Result
}

func (c *ResultCollector) Add(id string, result Result) {
    c.mu.Lock()
    c.results[id] = result
    c.mu.Unlock()
}

Solutions, in order of preference:

// ✅ Option 1: Use a results channel instead of shared state
results := make(chan Result, 1000)

// Workers send to channel (no lock)
go func() {
    for job := range jobs {
        result := processJob(ctx, job)
        results <- result
    }
}()

// Single collector goroutine reads from channel (no contention)
go func() {
    for result := range results {
        collected[result.ID] = result
    }
}()

// ✅ Option 2: sync.Map for concurrent read-heavy workloads
var results sync.Map
results.Store(id, result)
val, ok := results.Load(id)

The channel approach is idiomatic Go — share memory by communicating rather than communicating by sharing memory. The sync.Map approach is better when you need random access to results across goroutines.

A quick reference for which profile to use

High CPU, unclear where — CPU profile: pprof .../profile?seconds=30.

Memory growing over time — heap profile (inuse_space): pprof .../heap.

High GC pressure — heap profile (alloc_objects): pprof .../heap.

Goroutine count growing — goroutine profile: pprof .../goroutine?debug=1.

Low throughput despite low CPU — mutex or block profile: pprof .../mutex or .../block. Enable mutex profiling with runtime.SetMutexProfileFraction; enable block profiling with runtime.SetBlockProfileRate.

One workflow for production incidents

When something is wrong and you need to understand it quickly:

# 1. Check goroutine count (cheap, immediate)
curl localhost:6060/debug/pprof/goroutine?debug=2 | head -50

# 2. If CPU is high, get a 30s CPU profile
curl -o cpu.prof localhost:6060/debug/pprof/profile?seconds=30

# 3. If memory is growing, get a heap snapshot
curl -o heap.prof localhost:6060/debug/pprof/heap

# 4. Open whichever profile is relevant
go tool pprof -http=:8080 cpu.prof

The whole process takes under 5 minutes. You don’t need to reproduce the problem locally, you don’t need to redeploy with extra instrumentation, and you don’t need to restart the service. The profiler runs against the live process.

Summary

pprof is built into Go’s standard library and available with a single import. The three problems it catches most reliably in worker pool services are excessive allocations (fix: sync.Pool), goroutine leaks (fix: close channels, use context), and mutex contention (fix: channel-based result collection or sync.Map). The flame graph view in go tool pprof -http is the fastest path from profile to finding.

The only prerequisite is running the service under realistic load when you collect the profile. A profile from a quiet service tells you where your code could spend time, not where it does.

Previous in this series: context.Context Is Not Optional
Next: Container-aware resource management in Go.

context.Context Is Not Optional: A Practical Guide to Cancellation in Go Services

Prasad Ekke — Tue, 23 Jun 2026 12:47:35 +0000

Every Go service that does I/O — database calls, HTTP requests, queue polling, file reads — should be passing a context.Context through every layer. In practice, a large number of codebases treat context.Context as optional plumbing that gets added later, or as something only the HTTP handler layer needs to worry about.

That’s the mistake. By the time you realize you need cancellation deep in a worker pool, retrofitting it is painful. This post shows what context propagation looks like in a real worker pool, what the failure modes are when you skip it, and how to do it correctly from the start.

What context.Context actually does

A context.Context carries three things:

A cancellation signal — a channel that’s closed when the work should stop
A deadline or timeout — an absolute time after which the context is automatically cancelled
Key-value pairs — request-scoped values like trace IDs (use sparingly)

When a context is cancelled, any blocking operation that’s watching that context should stop and return an error. The context propagates down the call stack — a parent context cancelled cancels all children. It never propagates up.

The critical point: the context has to be passed all the way down to the code that actually blocks. Holding it at the HTTP handler layer and never passing it to your database call means cancellation never reaches the work.

What happens without context

Here is a worker pool that ignores context:

func runWorkers(jobs <-chan Job) {
    var wg sync.WaitGroup
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for job := range jobs {
                processJob(job) // no context
            }
        }()
    }
    wg.Wait()
}

func processJob(job Job) error {
    result, err := fetchFromDatabase(job.ID) // no context
    if err != nil {
        return err
    }
    return sendToDownstream(result) // no context
}

Now consider what happens when:

The calling service shuts down
A client disconnects
A deadline passes
An operator sends SIGTERM

The workers keep running. fetchFromDatabase keeps waiting. sendToDownstream keeps trying. The process can’t exit cleanly because goroutines are blocked on operations that have no way of knowing they should stop.

In the best case, the process takes 30 seconds to shut down waiting for goroutines to unblock naturally. In the worst case, it never terminates and has to be killed, potentially leaving in-flight work in an inconsistent state.

The correct structure: context flows in, errors flow out

The pattern is simple. Every function that does I/O or blocks takes a context as its first parameter:

func runWorkers(ctx context.Context, jobs <-chan Job) error {
    var wg sync.WaitGroup
    errs := make(chan error, 10)

    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for {
                select {
                case <-ctx.Done():
                    return // context cancelled, stop cleanly
                case job, ok := <-jobs:
                    if !ok {
                        return // channel closed, no more jobs
                    }
                    if err := processJob(ctx, job); err != nil {
                        select {
                        case errs <- err:
                        default:
                        }
                    }
                }
            }
        }()
    }

    wg.Wait()
    close(errs)

    // Return first error if any
    return <-errs
}

func processJob(ctx context.Context, job Job) error {
    result, err := fetchFromDatabase(ctx, job.ID)
    if err != nil {
        return err
    }
    return sendToDownstream(ctx, result)
}

Now when the context is cancelled — whether by timeout, operator signal, or parent cancellation — each worker’s select unblocks on ctx.Done() and returns cleanly. fetchFromDatabase and sendToDownstream propagate the context to their underlying I/O calls (database drivers, HTTP clients), which respect it.

Wiring context to OS signals for clean shutdown

The context that flows into your worker pool should ultimately be rooted at main, wired to OS signals:

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    jobs := make(chan Job, 100)

    // Producer
    go func() {
        defer close(jobs)
        if err := loadJobs(ctx, jobs); err != nil {
            log.Printf("producer stopped: %v", err)
        }
    }()

    // Workers
    if err := runWorkers(ctx, jobs); err != nil {
        log.Printf("workers stopped with error: %v", err)
        os.Exit(1)
    }

    log.Println("shutdown complete")
}

signal.NotifyContext (added in Go 1.16) returns a context that’s cancelled when SIGINT or SIGTERM arrives. That cancellation propagates through loadJobs and runWorkers automatically. When the signal arrives, the producer stops generating new jobs, in-flight I/O observes cancellation through the propagated context, workers exit, and the process shuts down cleanly.

Timeouts: per-job vs. pool-wide

There are two distinct timeout requirements in a worker pool, and they need separate contexts:

Pool-wide timeout: how long the entire batch is allowed to run. Use context.WithTimeout or context.WithDeadline on the root context.

Per-job timeout: how long a single job is allowed to take. Use a derived context inside the worker loop.

func runWorkers(ctx context.Context, jobs <-chan Job) error {
    // ctx already carries the pool-wide deadline from the caller

    var wg sync.WaitGroup
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for {
                select {
                case <-ctx.Done():
                    return
                case job, ok := <-jobs:
                    if !ok {
                        return
                    }

                    // Per-job timeout: 30 seconds max per job
                    jobCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
                    err := processJob(jobCtx, job)
                    cancel() // always call cancel to release resources

                    if err != nil {
                        log.Printf("job %s failed: %v", job.ID, err)
                    }
                }
            }
        }()
    }

    wg.Wait()
    return ctx.Err()
}

Two things to note. First, jobCtx is derived from ctx — if the pool-wide context is cancelled, the per-job context is also cancelled automatically. The hierarchy works in your favor. Second, cancel() is called immediately after processJob returns, not deferred. Deferring a context cancel inside a loop means you accumulate uncancelled contexts until the goroutine exits. Call cancel immediately after the work is done.

Checking context in CPU-bound work

The ctx.Done() pattern works for I/O that blocks, because the underlying library propagates the context. But what about CPU-bound work — a job that does heavy computation without any I/O?

func processJob(ctx context.Context, job Job) error {
    chunks := split(job.Data)

    for i, chunk := range chunks {
        // Check context periodically in long-running CPU work
        if i%100 == 0 {
            select {
            case <-ctx.Done():
                return ctx.Err()
            default:
            }
        }

        if err := processChunk(chunk); err != nil {
            return err
        }
    }

    return nil
}

The select with a default branch is non-blocking — it checks whether the context is done and moves on immediately if not. Checking every 100 iterations (or every N milliseconds using a ticker) is a reasonable balance between responsiveness and overhead.

The three rules

1. Context is always the first parameter.

func doWork(ctx context.Context, ...) error

Not a struct field, not a global, not injected via closure. First parameter, every time. This is a Go convention strong enough to be in the standard library style guide.

2. Never store context in a struct.

// ❌
type Worker struct {
    ctx context.Context
}

// ✅
func (w *Worker) Process(ctx context.Context, job Job) error

Contexts are request-scoped. A struct outlives any single request. Storing context in a struct means you’re using the wrong context for future requests, or holding a cancelled context longer than you should.

3. Always call cancel.

ctx, cancel := context.WithTimeout(parent, 5*time.Second)
defer cancel() // even if the timeout fires first

If the timeout fires before you call cancel, calling cancel is a no-op. But if the work finishes before the timeout, calling cancel releases the timer resources immediately rather than waiting for the timeout to expire. It costs nothing and prevents a resource leak.

Summary

Clean shutdown on SIGTERM — signal.NotifyContext at main, propagate down.

Stop workers on cancellation — select { case <-ctx.Done(): return } in the worker loop.

Per-job timeout — context.WithTimeout(ctx, duration) inside the loop, cancel immediately after.

CPU-bound work — periodic select { case <-ctx.Done(): return ctx.Err(); default: }.

Function signatures — ctx context.Context is always the first parameter.

Context propagation is not boilerplate. It is the mechanism that lets your service behave correctly under pressure — when deadlines pass, when clients disconnect, when operators need to restart the service without waiting for goroutines to unblock naturally. The cost of adding it from the start is a few extra parameters. The cost of retrofitting it later is much higher.

Previous in this series: Three Go Concurrency Mistakes I See in Almost Every Worker Pool
Next: Profiling a Go service in production with pprof.

Three Go Concurrency Mistakes I See in Almost Every Worker Pool

Prasad Ekke — Tue, 16 Jun 2026 12:42:10 +0000

Go makes concurrency feel easy. Goroutines are cheap, channels are built into the language, and the standard library gives you everything you need to spin up a worker pool in under 30 lines. That ease is genuinely valuable — but it also means bugs sneak in quietly. The program compiles, the tests pass, and the mistake only surfaces under load or after hours of runtime.

After working on high-throughput backend systems in Go, I keep seeing the same three mistakes. None of them are obscure. All of them are fixable in minutes once you know what to look for.

Mistake 1: Goroutine leaks from channels that never close

Here is a worker pool that looks completely reasonable:

func startWorkers(jobs <-chan Job) {
    for i := 0; i < 10; i++ {
        go func() {
            for job := range jobs {
                process(job)
            }
        }()
    }
}

Each worker ranges over the jobs channel and processes whatever comes in. Clean, idiomatic Go. The problem: if jobs is never closed, every one of those goroutines blocks forever.

for range on a channel blocks until the channel is closed or a value arrives. If the producer stops sending but never calls close(jobs), your 10 workers sit there permanently — holding memory, holding stack space, invisible to your metrics unless you’re explicitly tracking goroutine count.

In a long-running service, this compounds. Every time you restart the worker pool (say, on a config reload or a new batch) without draining the old one, you accumulate leaked goroutines.

The fix: always close the jobs channel when the producer is done, and own that responsibility clearly.

func runBatch(jobList []Job) {
    jobs := make(chan Job, len(jobList))

    // Producer owns the channel and closes it when done
    go func() {
        defer close(jobs) // guaranteed even if we panic
        for _, job := range jobList {
            jobs <- job
        }
    }()

    var wg sync.WaitGroup
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for job := range jobs {
                process(job)
            }
        }()
    }

    wg.Wait()
}

Two things worth noting here. First, defer close(jobs) in the producer goroutine means the channel closes even if the producer exits early or panics while sending jobs. Second, the sync.WaitGroup gives us a clean join point — wg.Wait() blocks until every worker has finished draining.

Quick diagnostic: add runtime.NumGoroutine() to your health endpoint. If that number grows over the lifetime of your service and never comes back down, you have a leak.

Mistake 2: WaitGroup counter manipulation in the wrong place

sync.WaitGroup has one sharp edge: wg.Add must be called before the goroutine starts, not inside it.

This version has a race condition:

var wg sync.WaitGroup

for _, job := range jobs {
    go func(j Job) {
        wg.Add(1)        // ❌ too late
        defer wg.Done()
        process(j)
    }(job)
}

wg.Wait()

If the loop spawns goroutines quickly and the scheduler runs slowly, wg.Wait() can be reached before any goroutine has called wg.Add(1). The counter is zero, Wait() returns immediately, and your main function exits while workers are still running — silently dropping work.

This is a classic race condition. It won’t always reproduce, which makes it worse.

The fix is simple: call wg.Add in the loop body, before launching the goroutine.

var wg sync.WaitGroup

for _, job := range jobs {
    wg.Add(1) // ✅ happens-before the goroutine starts
    go func(j Job) {
        defer wg.Done()
        process(j)
    }(job)
}

wg.Wait()

Now the counter increment is guaranteed to happen before wg.Wait() could possibly observe a zero. The goroutine carries the Done() responsibility via defer, so it fires even on panic.

A secondary pattern worth knowing: if you know the count upfront (say, you’re launching exactly N workers), call wg.Add(N) once before the loop. That’s even cleaner and harder to get wrong.

const numWorkers = 10
var wg sync.WaitGroup
wg.Add(numWorkers) // add all at once

for i := 0; i < numWorkers; i++ {
    go func() {
        defer wg.Done()
        for job := range jobs {
            process(job)
        }
    }()
}

wg.Wait()

Mistake 3: Unidirectional channel types ignored on function boundaries

Go lets you declare channels as send-only (chan<- Job) or receive-only (<-chan Job). Most codebases I’ve seen don’t use this consistently — functions just accept chan Job everywhere. This is technically fine, but it loses something real: the compiler can no longer tell you when your data flow is wrong.

Here is a worker pool where the direction confusion causes a real bug:

func spawnWorkers(jobs chan Job, results chan Result) {
    for i := 0; i < 10; i++ {
        go func() {
            for job := range jobs {
                result := process(job)
                jobs <- result.nextJob  // ❌ worker feeding back into the input channel
                results <- result
            }
        }()
    }
}

A worker accidentally wrote back into jobs instead of results. With bidirectional channels, this compiles without complaint. With directional types, it fails at compile time:

func spawnWorkers(jobs <-chan Job, results chan<- Result) {
    for i := 0; i < 10; i++ {
        go func() {
            for job := range jobs {
                result := process(job)
                jobs <- result.nextJob  // ✅ compile error: cannot send to receive-only channel
                results <- result
            }
        }()
    }
}

The directional types act as documentation that’s enforced by the compiler. The producer only sees chan<- Job. The consumer only sees <-chan Job. Neither can accidentally do the wrong thing.

A complete worker pool using this pattern correctly:

func produce(jobs chan<- Job, jobList []Job) {
    defer close(jobs)
    for _, job := range jobList {
        jobs <- job
    }
}

func consume(jobs <-chan Job, results chan<- Result, wg *sync.WaitGroup) {
    defer wg.Done()
    for job := range jobs {
        results <- process(job)
    }
}

func run(jobList []Job) []Result {
    jobs := make(chan Job, 100)
    results := make(chan Result, 100)

    var wg sync.WaitGroup
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go consume(jobs, results, &wg)
    }

    go produce(jobs, jobList)

    // Close results once all workers are done
    go func() {
        wg.Wait()
        close(results)
    }()

    var out []Result
    for result := range results {
        out = append(out, result)
    }
    return out
}

Every function now only sees what it’s supposed to see. The compiler enforces the data flow. And the pattern is easy to read: produce feeds in, consume drains, the closer goroutine shuts down results when all workers finish.

Summary

Channel never closed — What goes wrong: goroutines leak, memory grows. Fix: the producer owns close(ch), always via defer.

wg.Add inside the goroutine — What goes wrong: work silently dropped, races at shutdown. Fix: call wg.Add before go func().

Bidirectional channels everywhere — What goes wrong: wrong-direction sends compile silently. Fix: use chan<- and <-chan at function boundaries.

None of these require clever solutions. They require habit. Once you internalize that the producer owns close, that Add precedes launch, and that channel direction is free documentation, they stop appearing.

Go 1.25’s go vet includes a WaitGroup analyzer that flags mistake 2 when Add is called from inside a goroutine. On older Go versions, test the behavior directly by asserting that all launched work completes; go test -race is not reliable for this case because WaitGroup uses internal synchronization. For mistake 1, runtime.NumGoroutine() in a test that runs your pool repeatedly and checks for growth is the simplest approach. Mistake 3 is pure compiler enforcement — no test needed once the types are right.

If this was useful, the next post in this series covers context.Context propagation in worker pools — specifically what happens when you need to cancel in-flight jobs cleanly without leaking the workers themselves.

Go Interfaces: Why Less Is Almost Always More

Prasad Ekke — Mon, 08 Jun 2026 16:28:10 +0000

Go Interfaces: Why Less Is Almost Always More

If you’re coming to Go from Java or C++, interfaces look familiar at first glance. You define a set of methods, types implement them, you write code against the interface. Same idea, right?

Not quite. Go interfaces have one property that changes everything about how you should design them: a type implements an interface implicitly, without declaring it. There’s no implements keyword. If your type has the right methods, it satisfies the interface — whether it knows about the interface or not.

That single property pushes you toward a design philosophy that trips up most engineers coming from other languages: interfaces should be small, and they should be defined by the consumer, not the producer.

What the standard library is trying to tell you

Look at the most-used interfaces in Go’s standard library:

type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

type Closer interface {
    Close() error
}

One method each. That’s not laziness — that’s a deliberate design decision. Because interfaces are satisfied implicitly, a one-method interface can be satisfied by an enormous range of types: files, network connections, buffers, custom types you haven’t written yet. The smaller the interface, the more things satisfy it, the more reusable your code becomes.

io.ReadWriter combines two:

type ReadWriter interface {
    Reader
    Writer
}

And io.ReadWriteCloser combines three. Notice that the standard library builds up complexity by composing small interfaces, not by defining large ones upfront.

This is the pattern worth internalizing.

The fat interface trap

Here is how engineers from Java or C++ backgrounds tend to define interfaces in Go. Say you’re building a job processing system and you want to abstract over different job stores:

// ❌ The fat interface
type JobStore interface {
    AddJob(job Job) error
    GetJob(id string) (Job, error)
    DeleteJob(id string) error
    ListJobs(filter Filter) ([]Job, error)
    UpdateJob(job Job) error
    CountJobs(filter Filter) (int, error)
    MarkComplete(id string) error
    MarkFailed(id string, reason error) error
}

This feels complete. It covers everything the store can do. But now ask: how many types in your codebase will ever implement all 8 methods? Probably one — your real database implementation. And your test mock has to implement all 8 even if the function under test only calls GetJob.

// Every test mock must implement all 8 methods even if irrelevant
type mockStore struct{}

func (m *mockStore) AddJob(job Job) error                         { return nil }
func (m *mockStore) GetJob(id string) (Job, error)               { return Job{}, nil }
func (m *mockStore) DeleteJob(id string) error                   { return nil }
func (m *mockStore) ListJobs(filter Filter) ([]Job, error)       { return nil, nil }
func (m *mockStore) UpdateJob(job Job) error                     { return nil }
func (m *mockStore) CountJobs(filter Filter) (int, error)        { return 0, nil }
func (m *mockStore) MarkComplete(id string) error                { return nil }
func (m *mockStore) MarkFailed(id string, reason error) error    { return nil }

Eight methods of boilerplate for a test that calls one. And if you add a 9th method to the interface, every mock in the entire codebase breaks.

The Go way: consumer-defined, minimal interfaces

Instead of defining one big interface at the store level, define small interfaces at each use site — the function that actually needs the behavior.

// ✅ Each function declares exactly what it needs

type JobGetter interface {
    GetJob(id string) (Job, error)
}

type JobAdder interface {
    AddJob(job Job) error
}

type JobCompleter interface {
    MarkComplete(id string) error
    MarkFailed(id string, reason error) error
}

func processJob(id string, store JobGetter) error {
    job, err := store.GetJob(id)
    if err != nil {
        return err
    }
    // process...
    return nil
}

func submitJob(job Job, store JobAdder) error {
    return store.AddJob(job)
}

Now processJob only depends on JobGetter. Its test mock is one method:

type mockGetter struct {
    job Job
    err error
}

func (m *mockGetter) GetJob(id string) (Job, error) {
    return m.job, m.err
}

Your real PostgresJobStore still implements all 8 methods. It satisfies every one of these small interfaces automatically, without any change, because Go interfaces are implicit. The store doesn’t know or care about JobGetter or JobAdder — it just has the methods, and that’s enough.

Accepting interfaces, returning structs

There’s a corollary rule in Go that follows directly from this: accept interfaces, return concrete types.

// ✅ Accept an interface — flexible, testable
func drainQueue(queue <-chan Job, store JobAdder) error {
    for job := range queue {
        if err := store.AddJob(job); err != nil {
            return err
        }
    }
    return nil
}

// ❌ Return an interface — forces callers to type-assert, hides information
func newJobStore(cfg Config) JobStore {
    return &PostgresJobStore{...}
}

// ✅ Return a concrete type — callers get the full picture
func newJobStore(cfg Config) *PostgresJobStore {
    return &PostgresJobStore{...}
}

When you return a concrete type, callers can always choose to assign it to a narrower interface themselves. When you return a fat interface, you’ve already made that decision for them, and they’re stuck with it.

The one exception: returning error is an interface, and that’s intentional — it lets you return any error type, including custom ones. But error is a one-method interface, so the rule still holds in spirit.

Composing small interfaces when you need more

Sometimes a function genuinely needs multiple behaviors. Rather than reaching for a fat interface, compose:

type JobProcessor interface {
    JobGetter
    JobCompleter
}

func runWorker(id string, store JobProcessor) error {
    job, err := store.GetJob(id)
    if err != nil {
        return err
    }

    if err := doWork(job); err != nil {
        return store.MarkFailed(id, err)
    }

    return store.MarkComplete(id)
}

JobProcessor is defined at the call site, composed from two smaller interfaces, and PostgresJobStore satisfies it without modification. The test mock now only needs three methods.

Summary

Interface size — Avoid: 8-method interfaces defined upfront. Prefer: 1–3 method interfaces defined at the use site.

Where to define — Avoid: the producer side (the package that has the struct). Prefer: the consumer side (the package that calls the function).

What to return — Avoid: interfaces from constructors. Prefer: concrete types from constructors.

Complexity — Avoid: a single large interface. Prefer: composition of small interfaces.

Go’s implicit interface satisfaction is not just a syntactic convenience — it’s the mechanism that makes all of this work without coordination between packages. A type in one package can satisfy an interface defined in a completely different package, written years later, with no coupling between them.

The standard library’s one-method interfaces aren’t minimal because Go is simple. They’re minimal because the designers understood what implicit satisfaction makes possible.

Next in this series: Three Go concurrency mistakes I see in almost every worker pool.

DEV Community: Prasad Ekke

Determinism Is a Core Contract in Build Caching

Why determinism is the contract, not a nice-to-have

One bad action poisons everything downstream

Where nondeterminism actually comes from

How to hunt them down

Hermeticity is how you keep it fixed

Measure it, because it regresses

Remote Execution, Explained for People Who Just Run `make`

The core idea: the build is a graph of pure functions

What an action actually is

The two pieces of machinery

Why this reshapes how build systems are designed

Where it gets hard

Go Slice Initialization Gotchas: The Hole-Creating Bug I Keep Seeing (And a Linter That Catches It)

The three ways to initialize a slice

The bug: optimization that creates holes

Why this is hard to catch

The three clean patterns — pick one and be consistent

Pattern 1: Length 0, append only

Pattern 2: Length N, indexed assignment only

Pattern 3: Length N with capacity, indexed assignment only

The linter: makezero

Adding an explicit comment when you genuinely mix patterns

Quick mental checklist for slice initialization

Summary

Why I Always Specify Types for Constants in Go (And Name the Unit Too)

What untyped constants actually are

The bug: time.Duration and the silent conversion

Why this specific bug is hard to catch

This is not just a time.Duration problem

The fix: give the constant the correct type — time.Duration

Going one step further: encode the unit in the name

The pattern generalises beyond time.Duration

What about typed constants and iota?

Summary

References and related examples

Container-Aware Resource Management in Go: The Problem, Go 1.25’s Fix, and What’s Still Missing

The problem: Go doesn’t know it’s in a container

Problem 1: GOMAXPROCS ignores cgroup CPU limits

What actually happens

The Go 1.25 fix

Problem 2: GOMEMLIMIT still doesn’t read cgroup memory limits (not fixed in 1.25)

Workaround 1: Set GOMEMLIMIT as an environment variable

Workaround 2: Set GOMEMLIMIT in code at startup

Third-party libraries: pre-1.25 CPU fix and memory fix

uber-go/automaxprocs — CPU (pre-1.25)

KimMachineGun/automemlimit — Memory (still relevant on all versions)

tprasadtp/go-autotune — Both CPU and memory, cgroup v2 + Windows

What to use and when

Practical Kubernetes setup for a Go 1.25 service

Summary

Profiling a Go Service in Production: pprof in 10 Minutes

Enabling pprof

Collecting a CPU profile

Reading a flame graph

Problem 1: Excessive allocations

Problem 2: Goroutine leak

Problem 3: Lock contention

A quick reference for which profile to use

One workflow for production incidents

Summary

context.Context Is Not Optional: A Practical Guide to Cancellation in Go Services

What context.Context actually does

What happens without context

The correct structure: context flows in, errors flow out

Wiring context to OS signals for clean shutdown

Timeouts: per-job vs. pool-wide

Checking context in CPU-bound work

The three rules

Summary

Three Go Concurrency Mistakes I See in Almost Every Worker Pool

Mistake 1: Goroutine leaks from channels that never close

Mistake 2: WaitGroup counter manipulation in the wrong place

Mistake 3: Unidirectional channel types ignored on function boundaries

Summary

Go Interfaces: Why Less Is Almost Always More

Go Interfaces: Why Less Is Almost Always More

What the standard library is trying to tell you

The fat interface trap

This is not just a `time.Duration` problem