DEV Community

Cover image for Go Now Reads Your Container's CPU Limit. It Still Can't Read Your Memory Limit.
Meso Francis
Meso Francis

Posted on

Go Now Reads Your Container's CPU Limit. It Still Can't Read Your Memory Limit.

If you run Go on Kubernetes, there's a good chance two things have bitten you without leaving a single error in your logs: latency spikes you can't explain, and pods that die with OOMKilled while your heap graphs looked fine.

Those are two different problems with two different fixes. Recent Go versions quietly solved one of them. The other is still your job.

One quick correction before we start, because I've seen it repeated in a few places: container-aware GOMAXPROCS didn't arrive in a late patch release. It shipped in Go 1.25, and everything newer (1.26, 1.27.x) inherits it. What matters in practice is not just which toolchain you build with, but what your go.mod says. More on that trap in a minute.

The problem: Go counted the node, not your pod

Say your pod has limits.cpu: "2" and lands on a 32-core node. Before Go 1.25, the runtime asked the OS how many CPUs exist, got 32, and set GOMAXPROCS=32. So you had a scheduler happily running up to 32 threads of Go code in parallel, inside a container that was only paid for 2 CPUs' worth of time.

GOMAXPROCS before and after: 32 versus 2

Linux enforces CPU limits with CFS bandwidth control. Your limit becomes a quota: for cpu: 2, that's 200 ms of CPU time per 100 ms period. Thirty-two busy threads can burn that 200 ms in about 6 ms of wall-clock time. After that the kernel does the blunt thing: it pauses the whole container until the next period starts.

CFS throttling timeline

From the outside this looks like random tail latency. No panic, no error, no log line. Your p50 is great and your p99 is haunted.

What Go 1.25 changed

The runtime now reads the cgroup CPU quota at startup and sets the default GOMAXPROCS to the lower of the machine's CPU count and your container's CPU limit. A few details worth knowing, because they're where people get surprised:

  • It reads the limit, not the request. requests.cpu is just a scheduling hint and a weight under contention. Go ignores it.
  • Fractions round up. A limit of 0.5 gives you GOMAXPROCS=1, and 2.5 gives you 3.
  • It keeps watching. The runtime re-checks the limit about once a second, so if something resizes your pod's CPU limit in place, GOMAXPROCS follows without a restart.
  • Any manual setting wins. If the GOMAXPROCS env var is set, or your code calls runtime.GOMAXPROCS(n), the automatic behavior steps aside.
  • No limit, no change. With no CPU limit set, you get the usual NumCPU default.

If you were pulling in go.uber.org/automaxprocs for this, you can most likely delete it.

The go.mod trap

This is the one I'd put in a code review checklist. Go gates new runtime behavior behind the language version in go.mod. If your module still says:

go 1.24
Enter fullscreen mode Exit fullscreen mode

then building with a shiny new toolchain does not turn this on. The documented defaults for containermaxprocs and updatemaxprocs are the old behavior for language versions 1.24 and below. Bump the go line to 1.25 or later, or set the GODEBUG settings explicitly.

And if you ever need the old behavior back, you have the escape hatches:

GODEBUG=containermaxprocs=0   # ignore the cgroup limit at startup
GODEBUG=updatemaxprocs=0      # don't re-check the limit periodically
Enter fullscreen mode Exit fullscreen mode

What it doesn't fix

Don't over-credit it. GOMAXPROCS limits how many threads run Go code in parallel. It doesn't cap everything your process does with CPU. Cgo calls, threads blocked in syscalls, and background work still draw from the same quota. And GOMAXPROCS equal to your limit is a good default, not a guarantee of zero throttling: a bursty service can still hit the quota inside a single period.

So measure. The number I'd put on a dashboard is the throttled-period ratio:

sum(rate(container_cpu_cfs_throttled_periods_total{container="my-service"}[5m]))
/
sum(rate(container_cpu_cfs_periods_total{container="my-service"}[5m]))
Enter fullscreen mode Exit fullscreen mode

If that sits near zero, you're fine. If it's climbing under load, either the limit is too tight for the work or something other than Go code is eating your quota.

Now the part Go did not fix: memory

Here's the asymmetry that trips people up. Go learned to read the CPU limit. It did not learn to read the memory limit.

GOMEMLIMIT defaults to effectively unlimited. The Go runtime documentation still describes the default as math.MaxInt64, which disables the limit, and the 1.27 docs say the same. So your garbage collector paces itself against heap growth (that's GOGC's job), with no idea that a cgroup wall stands at 512 MiB.

The result is the classic story: under load the heap grows, nothing tells the GC to work harder, the container crosses its limit, and the kernel's OOM killer ends the process. No Go panic. Just exit code 137 and OOMKilled in kubectl describe pod.

Heap growth with and without GOMEMLIMIT

GOMEMLIMIT is a soft limit. As the runtime approaches it, the GC runs more aggressively and the runtime gets more eager to return memory to the OS. It's a pressure valve, not a wall. Two honest caveats:

  1. It covers memory the Go runtime manages (heap, stacks, runtime structures). It excludes things like cgo allocations and the binary's own mappings, but those still count toward the container limit. That's why you set it below the limit rather than at it.
  2. If your live data genuinely exceeds the limit, no setting helps. The GC will work harder and harder and you'll just trade an OOM kill for a slow, GC-bound death. The fix there is a bigger limit or less memory.

How to set it

The rule of thumb I follow: GOMEMLIMIT at roughly 85 to 90 percent of the container memory limit, leaving headroom for everything the runtime doesn't track.

The simplest option is to be explicit and keep the two numbers next to each other in your manifest:

resources:
  requests:
    cpu: "1"
    memory: 512Mi
  limits:
    cpu: "2"
    memory: 512Mi
env:
  - name: GOMEMLIMIT
    value: "460MiB"   # ~90% of the 512Mi limit
Enter fullscreen mode Exit fullscreen mode

You might be tempted to use the Kubernetes downward API (resourceFieldRef: limits.memory) so the value follows the limit automatically. It gives you the raw limit, though, and there's no way to multiply it by 0.9 there. Setting GOMEMLIMIT equal to the limit leaves the GC no room to act before the kernel does.

If you want it derived automatically, a small library handles the ratio and cgroup v1/v2 for you:

import _ "github.com/KimMachineGun/automemlimit" // sets GOMEMLIMIT from the cgroup limit, 90% by default
Enter fullscreen mode Exit fullscreen mode

(It's a no-op if GOMEMLIMIT is already set. AUTOMEMLIMIT adjusts the ratio.) Either approach is fine. Pick one and be consistent, because the failure mode of forgetting is silent.

A 10-line sanity check

Don't trust configuration you haven't observed. Log this at startup and look at it in a real pod:

import (
    "log"
    "runtime"
    "runtime/debug"
)

func logRuntimeLimits() {
    log.Printf("NumCPU=%d GOMAXPROCS=%d GOMEMLIMIT=%d",
        runtime.NumCPU(),
        runtime.GOMAXPROCS(0),
        debug.SetMemoryLimit(-1), // negative = just report the current limit
    )
}
Enter fullscreen mode Exit fullscreen mode

Healthy output in a pod limited to 2 CPUs on a big node: GOMAXPROCS=2 while NumCPU shows the node's count, and a GOMEMLIMIT that's a real number rather than 9223372036854775807. If you see that giant number, your memory limit isn't set and the GC is flying blind.

My checklist

  1. Set the go line in go.mod to 1.25 or later, so container-aware GOMAXPROCS actually turns on.
  2. Set a CPU limit on latency-sensitive services. Go can only match a limit that exists. (Whether every workload should have one is a long-running Kubernetes debate. For Go services where tail latency matters, I'd rather have a predictable limit than none.)
  3. Set GOMEMLIMIT to about 90% of the memory limit, explicitly or via a library.
  4. Set memory requests equal to limits, so memory is never over-committed on the node.
  5. Alert on the throttled-period ratio and on OOMKilled restarts, not just on CPU and heap graphs.
  6. Log GOMAXPROCS and the memory limit at startup so the config is visible in every environment.

The short version

Go 1.25 closed a decade-old gap: your scheduler now sizes itself to the CPU you were actually given. That removes a whole class of mysterious tail latency. Memory is the one the runtime still leaves to you, and it costs one environment variable to fix. Set it, check it, and spend your debugging time on something more interesting than exit code 137.

If you've seen different behavior on your own clusters, especially around throttling after the upgrade, I'd like to hear about it in the comments.

Top comments (0)