DEV Community

Nerav Doshi
Nerav Doshi

Posted on

The Same Memory Number That Worked in Podman Got OOMKilled in Kubernetes

OOMKilled is what Kubernetes says when a container tries to use more memory than its limit allows and the kernel steps in and kills it. Going into this one, my assumption was straightforward: Entry 09 measured 1.663GB under Podman, so setting a Kubernetes limit to that exact number should be enough. Same model, same host, same measurement. That assumption was wrong, and finding out why turned into the most interesting result in the series so far.

Deployed the same image to a fresh kind cluster with requests.memory and limits.memory both set to the precise Podman figure — 1663Mi. Pulled the model, tried to load it:

kubectl exec -it ollama-sized -- ollama pull llama3.2:1b
kubectl exec -it ollama-sized -- ollama run llama3.2:1b
Enter fullscreen mode Exit fullscreen mode

Pull worked. Loading gave me command terminated with exit code 137 — no real error message, just a dead session. kubectl describe pod filled in what actually happened:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
Restart Count:  2
Enter fullscreen mode Exit fullscreen mode

Killed twice, for exceeding the exact number that had worked cleanly under Podman on the same machine, for the same model.

I don't have a confirmed explanation yet, and I'd rather say that plainly than make one up. A few real possibilities: the Podman number was a snapshot taken after the model had already loaded and settled, not a peak — memory during actual loading (KV cache setup, context buffers) might spike higher than that steady-state figure ever captured. Or cgroup memory accounting just differs between runtimes — Kubernetes' limit on containerd typically counts page cache against you, Podman's VM might not count it the same way. Or the two VMs underneath — Podman's applehv machine, kind's node — have different baseline overhead that was never separately measured. Any of these could be true. None of them are confirmed.

What is confirmed: a real, measured number from one container runtime doesn't automatically carry over to another, even on identical hardware running the identical model. "It worked under Podman" turned out to be necessary, not sufficient. The honest next move isn't guessing a bigger number and hoping — it's actually finding the real ceiling under kind and seeing how far off 1663Mi was.

Top comments (0)