DEV Community

Cover image for 512m Is Half a Byte. It Will Pass Your Code Review.
Kalislav Smirnov
Kalislav Smirnov

Posted on

512m Is Half a Byte. It Will Pass Your Code Review.

#devops #discuss #kubernetes #cloud

Three resource specs that look the same in code review.

memory: 512Mi   # 536,870,912 bytes
memory: 512M    # 512,000,000 bytes, about 488 Mi
memory: 512m    # 0.512 bytes
Enter fullscreen mode Exit fullscreen mode

All three pass kubectl apply. Only one of them is what you meant.

I used to explain this from the docs. This time I spun up a throwaway cluster (kind, Kubernetes 1.37, containerd, cgroup v2) and applied every bad unit I could think of, to see what actually happens. The answer is weirder than "OOMKilled".

Expanding brain: memory 512Mi, memory 512M, memory

 

 

Mi and M are different units

Kubernetes quantities take both binary and decimal suffixes. Ki, Mi, Gi are powers of two; k, M, G are powers of ten. The gap grows with the unit: M is 4.6% smaller than Mi, G is 6.9% smaller than Gi.

Put a 4G limit on a JVM sized for 4Gi and you've removed 281 MiB of headroom without noticing. That's the OOMKill that only shows up under load, on a Friday, in the one service nobody touched.

Monkey puppet: Grafana says pod OOMKilled at 488 MiB; me, who wrote memory: 512M

In the lab the gap was easy to see. My kind node had 7.65 GiB. A pod with memory: 4G started fine. Its twin with memory: 4Gi, applied next to it:

Warning  FailedScheduling  default-scheduler
  0/1 nodes are available: 1 Insufficient memory.
Enter fullscreen mode Exit fullscreen mode

Same number, same node, one letter, one pod Running and one Pending. That's the 6.9% gap made visible, and it works the same way at 3 a.m. on a full production node.

Down in the kernel the units disappear. The memory.max that containerd wrote for each pod:

spec          memory.max (bytes)
500M          499998720      (rounded down to a 4 KiB page)
500Mi         524288000
512M          512000000
512Mi         536870912
4G            3999997952
Enter fullscreen mode Exit fullscreen mode

kubectl describe node reports capacity in Ki, the cloud sells RAM in GiB, and half the dashboards show GB. Pick binary units in manifests and stay with them.

 

 

Lowercase m means milli. Everywhere.

On CPU, m is the normal unit: 500m is half a core. On memory it still means milli, so 512m is half a byte. Here is what my 1.37 cluster said when I applied it:

Warning: spec.containers[0].resources.limits[memory]: fractional byte value "512m" is invalid, must be an integer
Warning: spec.containers[0].resources.requests[memory]: fractional byte value "512m" is invalid, must be an integer
pod/mem-512m created
Enter fullscreen mode Exit fullscreen mode

"Is invalid", and then "created". It's a warning, not a rejection, and --dry-run=client doesn't show it at all. In a CI log or a Helm upgrade it scrolls past with the rest. The object is stored exactly as written: memory: 512m in get -o yaml, in describe, everywhere.

One more edge: memory: 2000m gets no warning at all, because it's a whole number of bytes. The API server quietly stores it as 2. Two bytes.

 

 

As a request: invisible

A pod with requests.memory: 512m and no limit started in seconds. The scheduler counts it as zero, the cgroup has no limit (memory.max = max), and the pod can land on any node. The only symptom is that your capacity planning is wrong, which nobody notices until the node is full.

There's a funnier side effect. Once a single 512m pod is on a node, kubectl describe node switches the whole memory row to milli-bytes, because it has to add your half-byte to everything else:

  Resource           Requests                Limits
  memory             304087041536m (3%)      356515841024m (4%)
Enter fullscreen mode Exit fullscreen mode

That's 304 billion milli-bytes, also known as 290 MiB. Later in the run, with a dozen pods on the node, it read 7249661186048m (88%). If you ever see an m in your node's memory column, someone in the cluster wrote a lowercase m.

 

 

As a limit: the error message lies

I expected an OOMKill. There wasn't one, because there's no process to kill. The pod's cgroup got memory.max = 0 (0.512 bytes, rounded down), and the pod sat in ContainerCreating forever with this:

Warning  FailedMount  kubelet
  MountVolume.SetUp failed for volume "kube-api-access-4xxpn" :
  write .../kube-api-access-4xxpn/...: no space left on device
Enter fullscreen mode Exit fullscreen mode

"No space left on device", about a 2 KB certificate file. The service-account token volume is a tmpfs, tmpfs writes are charged to the pod's memory cgroup, and the pod's memory budget is zero bytes. Nothing in the message says "memory". Nothing says "limit".

Panik Kalm Panik: pod stuck in ContainerCreating; it says no space left on device, just a disk thing; the disk is fine, the memory limit is 512m

With automountServiceAccountToken: false the kubelet gets one step further and the container runtime fails instead:

FailedCreatePodSandBox ... runc create failed: unable to start container process:
unable to apply cgroup configuration: unable to start unit "cri-containerd-….scope" ... got 'failed'
Enter fullscreen mode Exit fullscreen mode

and the node's journal adds Failed to add PIDs to scope's control group: Cannot allocate memory. Two different error messages, neither of them mentions your YAML.

Just did a bad thing

 

 

CPU: 0.1 and 100m are the same, and tiny values round up

cpu: 0.1, cpu: "100m" and cpu: 100m all mean a tenth of a vCPU. The API server stores 0.1 as 100m and 1000m as 1, and the cgroup gets the same cpu.max (10000/100000 for 100m, 100000/100000 for 1). One CPU is one vCPU: a hyperthread on most cloud instances, not a physical core.

I used to write that Kubernetes doesn't accept precision finer than 1m. That's wrong: it accepts it and rounds up, silently:

you write       stored as
cpu: 0.0001     1m
cpu: 0.0005     1m
cpu: 0.0015     2m
cpu: 100.5m     101m
cpu: 1n         1m
Enter fullscreen mode Exit fullscreen mode

No error, no warning. Harmless for CPU, but it's the same parser that keeps 512m of memory as written. Kubernetes rounds what's harmless and keeps what breaks.

Canonicalisation has its own surprises: 0.5Gi is stored as 512Mi, but 0.5G becomes 500M, and 512.5Mi comes back as 524800Ki. If you diff live objects against Git, that's where the noise comes from.

 

 

How to catch it before the cluster does

  1. Lint for [0-9]m under memory:. A one-line grep in CI catches every memory case in this post. A Kyverno or Gatekeeper policy can do the same at admission time.
  2. Treat API warnings as errors in CI. kubectl apply --warnings-as-errors exists, and it would have stopped the 512m pod above.
  3. Binary suffixes only for memory (Mi, Gi). Decide once, write it in the contributing guide, and let the linter enforce it.
  4. When a pod is stuck in ContainerCreating with a weird storage error, check its memory limit before you check the disk.

 

 

Check before you apply

I put a small converter on NodeSize. Type a quantity the way you write it in YAML and it shows millicores and cores, Mi, Gi, M, G and bytes, and warns about the decimal and milli traps: Kubernetes units converter. To see how much of a node those requests can actually use after kubelet's reservations, the node size calculator on the same site does that.

Your turn

What's the strangest error message you've chased that turned out to be a typo in resources? Mine is now "no space left on device" for a half-byte memory limit.

Top comments (0)