DEV Community

Kazu
Kazu

Posted on

K8s upgrade drops a path; fixed scripts read other values

#linux #devops #kubernetes #monitoring

After a Kubernetes version upgrade replaces the node image, procedures and scripts that read container CPU and memory values from under /sys/fs/cgroup stop working.

$ kubectl exec <pod> -- cat /sys/fs/cgroup/cpu/cpu.stat
cat: can't open '/sys/fs/cgroup/cpu/cpu.stat': No such file or directory
command terminated with exit code 1
Enter fullscreen mode Exit fullscreen mode

Running ls inside the container shows no cpu/ and no memory/. cpu.stat and memory.stat, which used to live in separate directories, are listed directly. The node's /sys/fs/cgroup looks the same.

$ kubectl exec <pod> -- ls /sys/fs/cgroup
cgroup.controllers
cgroup.events
...
cpu.stat
...
memory.current
memory.events
...
memory.stat
...
Enter fullscreen mode Exit fullscreen mode

The layout with one directory per controller is cgroup v1, and this single flat layout is cgroup v2. The switch from v1 to v2 happened when the node image changed.

Removing cpu/ from the script's path opens a file with the same name.

$ kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat
usage_usec 768869
user_usec 684571
system_usec 84297
nice_usec 0
core_sched.force_idle_usec 0
nr_periods 38
nr_throttled 37
throttled_usec 1592698
nr_bursts 0
burst_usec 0
Enter fullscreen mode Exit fullscreen mode

The throttled_time line from the old cpu.stat is missing, and there is a throttled_usec line in its place. A script that collected throttled_time finds no matching line and returns an empty result without raising an error.

Even once the files open, some values read the v1 way come out different. This article covers three of them: CPU throttle time, memory usage, and OOM count.

It is written for people who read the files under /sys/fs/cgroup with their own scripts or runbooks. If you see these values through a monitoring agent such as cAdvisor, the agent handles v2 itself, and that case is outside the scope here. An explanation of v2 itself and the migration steps are on the Kubernetes blog.

Three reads that change

Throttle time

throttled_time, missing from the cpu.stat above, became throttled_usec. It is the total time the cgroup was stopped for hitting its CPU limit, and its unit changed from nanoseconds to microseconds.1 If you change only the key name and keep the /1e9 that converts nanoseconds to seconds, you get a value 1/1000 of the real one.

$ kubectl exec <pod> -- awk '$1=="throttled_usec"{printf "%.6f\n", $2/1e9}' /sys/fs/cgroup/cpu.stat
0.001593
Enter fullscreen mode Exit fullscreen mode

With /1e6 instead, it comes out to 1.592698 seconds.

nr_throttled and nr_periods, which give the fraction of periods that hit the limit, keep the same names and meanings as in v1, so the division works unchanged.

Memory usage

In v1, memory usage was calculated with this formula.

memory.usage_in_bytes − total_inactive_file (a line in memory.stat)
Enter fullscreen mode Exit fullscreen mode

When a process reads or writes a file, the kernel keeps the contents in memory as page cache, and the cgroup's usage includes it. The part of the page cache that hasn't been used recently is inactive_file, which the kernel reclaims as soon as memory runs short. Usage minus inactive_file is the part that can't be released right away: the working set. The kubelet exposes this value as container_memory_working_set_bytes, and the memory column of kubectl top pod shows the same value.2

In v2, the formula becomes:3

memory.current − inactive_file (a line in memory.stat)
Enter fullscreen mode Exit fullscreen mode

v2's memory.stat has no lines starting with total_. The aggregate lines from v1 appear in v2 under their names with total_ removed.

$ kubectl exec <pod> -- cat /sys/fs/cgroup/memory.current
108158976
$ kubectl exec <pod> -- grep ^inactive_file /sys/fs/cgroup/memory.stat
inactive_file 104906752
Enter fullscreen mode Exit fullscreen mode

The difference, about 3 MiB, is the working set.

The corrected value matches container_memory_working_set_bytes from the kubelet.

$ kubectl get --raw "/api/v1/nodes/<node>/proxy/metrics/resource" | grep ^container_memory_working_set_bytes
Enter fullscreen mode Exit fullscreen mode

OOM count

In v1, a container's OOM count came from oom_kill in memory.oom_control.4 In v2, the same oom_kill is in memory.events, a file that counts the memory events in that cgroup by type. However, oom_kill in the container's memory.events reads 0 even after the container has restarted due to OOM. The count is kept in the memory.events of the Pod's slice, one level above the container.

The rest of this section covers why it reads 0, how to find the Pod's slice, and how to read the slice's memory.events, in that order.

Why the container's oom_kill reads 0

In a Pod that restarted due to OOM, the container's memory.events reads like this.

$ kubectl exec <pod> -- grep oom /sys/fs/cgroup/memory.events
oom 0
oom_kill 0
oom_group_kill 0
Enter fullscreen mode Exit fullscreen mode

On v2 nodes, the kubelet sets the container's memory.oom.group to 1.5 When OOM occurs in this cgroup, the kernel kills every process in the container, the container is recreated along with its cgroup, and the counters start again from 0.

To check whether your nodes behave this way, see whether the container's /sys/fs/cgroup/memory.oom.group is 1. If the kubelet's singleProcessOOMKill is true, it stays 0.

Finding the Pod's slice

The Pod's slice is the cgroup that groups all of the Pod's containers, and its memory.events adds up the counts of the containers under it. The slice survives container restarts, so the count survives too.

/sys/fs/cgroup/
└── kubepods.slice/
    └── kubepods-burstable.slice/
        └── kubepods-burstable-pod<uid>.slice/         ← Pod's slice
            └── cri-containerd-<container-id>.scope/   ← container's scope
Enter fullscreen mode Exit fullscreen mode

To locate the slice, search for the scope by container ID. The container ID is the part of the Pod's containerID after containerd://. When you enter the node with kubectl debug node, the node's files appear under /host.

$ kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].containerID}'
containerd://<container-id>
$ kubectl debug node/<node> -it --image=busybox
# find /host/sys/fs/cgroup -type d -name 'cri-containerd-<container-id>.scope'
Enter fullscreen mode Exit fullscreen mode

The parent of the directory it finds is the Pod's slice. cri-containerd- is the prefix used when the runtime is containerd. If you can log in to the node as root, drop /host from the path.

Reading the slice's memory.events

In the slice's memory.events, oom_group_kill is the OOM count, and it matches restartCount. oom_group_kill is the number of times all processes in the cgroup were killed together because of memory.oom.group.6 For a Pod with a restartCount of 3, it looks like this.

# grep oom <slice>/memory.events
oom 3
oom_kill 9
oom_group_kill 3
Enter fullscreen mode Exit fullscreen mode

oom_kill goes up each time the kernel kills a process, so a single OOM raises it by more than one. In this Pod, three OOMs brought it to 9.

  1. The cpu.stat entry in the kernel's cgroup v2 documentation. ↩

  2. The memory entry in the Kubernetes Resource metrics pipeline documentation. ↩

  3. cAdvisor's handler.go subtracts total_inactive_file for v1 and inactive_file for v2. The memory-available-cgroupv2.sh example in the Kubernetes documentation writes the v2 formula the same way. ↩

  4. The memory.oom_control entry in the kernel's cgroup v1 memory documentation. oom_kill is the number of processes in that cgroup killed by OOM. ↩

  5. In the kubelet's kuberuntime_container_linux.go, memory.oom.group is set to 1 on v2 nodes unless singleProcessOOMKill is true. This was added in kubernetes/kubernetes#117793. ↩

  6. The memory.events entry in the kernel's cgroup v2 documentation. ↩

Top comments (0)