If you run Go on Kubernetes, there's a good chance two things have bitten you without leaving a single error in your logs: latency spikes you can't explain, and pods that die with OOMKilled while your heap graphs looked fine.
Those are two different problems with two different fixes. Recent Go versions quietly solved one of them. The other is still your job.
One quick correction before we start, because I've seen it repeated in a few places: container-aware GOMAXPROCS didn't arrive in a late patch release. It shipped in Go 1.25, and everything newer (1.26, 1.27.x) inherits it. What matters in practice is not just which toolchain you build with, but what your go.mod says. More on that trap in a minute.
The problem: Go counted the node, not your pod
Say your pod has limits.cpu: "2" and lands on a 32-core node. Before Go 1.25, the runtime asked the OS how many CPUs exist, got 32, and set GOMAXPROCS=32. So you had a scheduler happily running up to 32 threads of Go code in parallel, inside a container that was only paid for 2 CPUs' worth of time.
Linux enforces CPU limits with CFS bandwidth control. Your limit becomes a quota: for cpu: 2, that's 200 ms of CPU time per 100 ms period. Thirty-two busy threads can burn that 200 ms in about 6 ms of wall-clock time. After that the kernel does the blunt thing: it pauses the whole container until the next period starts.
From the outside this looks like random tail latency. No panic, no error, no log line. Your p50 is great and your p99 is haunted.
What Go 1.25 changed
The runtime now reads the cgroup CPU quota at startup and sets the default GOMAXPROCS to the lower of the machine's CPU count and your container's CPU limit. A few details worth knowing, because they're where people get surprised:
-
It reads the limit, not the request.
requests.cpuis just a scheduling hint and a weight under contention. Go ignores it. -
Fractions round up. A limit of
0.5gives youGOMAXPROCS=1, and2.5gives you3. -
It keeps watching. The runtime re-checks the limit about once a second, so if something resizes your pod's CPU limit in place,
GOMAXPROCSfollows without a restart. -
Any manual setting wins. If the
GOMAXPROCSenv var is set, or your code callsruntime.GOMAXPROCS(n), the automatic behavior steps aside. -
No limit, no change. With no CPU limit set, you get the usual
NumCPUdefault.
If you were pulling in go.uber.org/automaxprocs for this, you can most likely delete it.
The go.mod trap
This is the one I'd put in a code review checklist. Go gates new runtime behavior behind the language version in go.mod. If your module still says:
go 1.24
then building with a shiny new toolchain does not turn this on. The documented defaults for containermaxprocs and updatemaxprocs are the old behavior for language versions 1.24 and below. Bump the go line to 1.25 or later, or set the GODEBUG settings explicitly.
And if you ever need the old behavior back, you have the escape hatches:
GODEBUG=containermaxprocs=0 # ignore the cgroup limit at startup
GODEBUG=updatemaxprocs=0 # don't re-check the limit periodically
What it doesn't fix
Don't over-credit it. GOMAXPROCS limits how many threads run Go code in parallel. It doesn't cap everything your process does with CPU. Cgo calls, threads blocked in syscalls, and background work still draw from the same quota. And GOMAXPROCS equal to your limit is a good default, not a guarantee of zero throttling: a bursty service can still hit the quota inside a single period.
So measure. The number I'd put on a dashboard is the throttled-period ratio:
sum(rate(container_cpu_cfs_throttled_periods_total{container="my-service"}[5m]))
/
sum(rate(container_cpu_cfs_periods_total{container="my-service"}[5m]))
If that sits near zero, you're fine. If it's climbing under load, either the limit is too tight for the work or something other than Go code is eating your quota.
Now the part Go did not fix: memory
Here's the asymmetry that trips people up. Go learned to read the CPU limit. It did not learn to read the memory limit.
GOMEMLIMIT defaults to effectively unlimited. The Go runtime documentation still describes the default as math.MaxInt64, which disables the limit, and the 1.27 docs say the same. So your garbage collector paces itself against heap growth (that's GOGC's job), with no idea that a cgroup wall stands at 512 MiB.
The result is the classic story: under load the heap grows, nothing tells the GC to work harder, the container crosses its limit, and the kernel's OOM killer ends the process. No Go panic. Just exit code 137 and OOMKilled in kubectl describe pod.
GOMEMLIMIT is a soft limit. As the runtime approaches it, the GC runs more aggressively and the runtime gets more eager to return memory to the OS. It's a pressure valve, not a wall. Two honest caveats:
- It covers memory the Go runtime manages (heap, stacks, runtime structures). It excludes things like cgo allocations and the binary's own mappings, but those still count toward the container limit. That's why you set it below the limit rather than at it.
- If your live data genuinely exceeds the limit, no setting helps. The GC will work harder and harder and you'll just trade an OOM kill for a slow, GC-bound death. The fix there is a bigger limit or less memory.
How to set it
The rule of thumb I follow: GOMEMLIMIT at roughly 85 to 90 percent of the container memory limit, leaving headroom for everything the runtime doesn't track.
The simplest option is to be explicit and keep the two numbers next to each other in your manifest:
resources:
requests:
cpu: "1"
memory: 512Mi
limits:
cpu: "2"
memory: 512Mi
env:
- name: GOMEMLIMIT
value: "460MiB" # ~90% of the 512Mi limit
You might be tempted to use the Kubernetes downward API (resourceFieldRef: limits.memory) so the value follows the limit automatically. It gives you the raw limit, though, and there's no way to multiply it by 0.9 there. Setting GOMEMLIMIT equal to the limit leaves the GC no room to act before the kernel does.
If you want it derived automatically, a small library handles the ratio and cgroup v1/v2 for you:
import _ "github.com/KimMachineGun/automemlimit" // sets GOMEMLIMIT from the cgroup limit, 90% by default
(It's a no-op if GOMEMLIMIT is already set. AUTOMEMLIMIT adjusts the ratio.) Either approach is fine. Pick one and be consistent, because the failure mode of forgetting is silent.
A 10-line sanity check
Don't trust configuration you haven't observed. Log this at startup and look at it in a real pod:
import (
"log"
"runtime"
"runtime/debug"
)
func logRuntimeLimits() {
log.Printf("NumCPU=%d GOMAXPROCS=%d GOMEMLIMIT=%d",
runtime.NumCPU(),
runtime.GOMAXPROCS(0),
debug.SetMemoryLimit(-1), // negative = just report the current limit
)
}
Healthy output in a pod limited to 2 CPUs on a big node: GOMAXPROCS=2 while NumCPU shows the node's count, and a GOMEMLIMIT that's a real number rather than 9223372036854775807. If you see that giant number, your memory limit isn't set and the GC is flying blind.
My checklist
- Set the
goline ingo.modto 1.25 or later, so container-awareGOMAXPROCSactually turns on. - Set a CPU limit on latency-sensitive services. Go can only match a limit that exists. (Whether every workload should have one is a long-running Kubernetes debate. For Go services where tail latency matters, I'd rather have a predictable limit than none.)
- Set
GOMEMLIMITto about 90% of the memory limit, explicitly or via a library. - Set memory
requestsequal tolimits, so memory is never over-committed on the node. - Alert on the throttled-period ratio and on
OOMKilledrestarts, not just on CPU and heap graphs. - Log
GOMAXPROCSand the memory limit at startup so the config is visible in every environment.
The short version
Go 1.25 closed a decade-old gap: your scheduler now sizes itself to the CPU you were actually given. That removes a whole class of mysterious tail latency. Memory is the one the runtime still leaves to you, and it costs one environment variable to fix. Set it, check it, and spend your debugging time on something more interesting than exit code 137.
If you've seen different behavior on your own clusters, especially around throttling after the upgrade, I'd like to hear about it in the comments.



Top comments (0)