Four Kubernetes node bugs I found by reading CI failures to the end

Kubernetes' node tests fail a lot. Most of the time someone re-runs them, they go green, and everyone moves on.

Over the past week I picked four failures in SIG Node territory and refused to move on — I kept reading the logs until the timeline explained the failure. Three were CI flakes. The fourth was a second, separate bug hiding inside a reported kubelet 1.37 memory regression. In every case, the bug wasn't where the error message pointed.

Status, to be upfront: as of October 10, 2026, all four of my fixes are open pull requests, triaged or awaiting review, not merged. This post is about the investigations — which is the part I'd want to read anyway.

How I read a failing CI run

Every Kubernetes CI run on Prow keeps its artifacts: the build log, the JUnit results, the kubelet and containerd logs from the node, and for node tests, the VM's serial console. TestGrid shows the history, so you can tell a one-off from a pattern.

My loop for each flake:

1. Find the pattern in TestGrid. Which job, how often, and — most useful — which similar jobs don't fail.
2. Build a timeline across logs. Test log, kubelet log and serial console on one clock, to the second.
3. Explain the difference. If only one job fails, something about that job's environment is the clue.
4. Reproduce before fixing. Either in a unit test that fails on master, or on a machine set up like the CI node.

Bug 1: the image record that kept disappearing

PR: kubernetes/kubernetes#142801 · fixes #140942

The symptom

Tests for KubeletEnsureSecretPulledImages — the feature that makes the kubelet re-check credentials before reusing a private image — kept failing in one job, ci-kubernetes-node-swap-ubuntu-serial: 3 of the last 10 runs, 6 of 20 over a longer window. A test helper waits up to five seconds for the kubelet to write an ImagePulledRecord for an image, and the record never appeared.

This had already been through two issues. One was closed by a PR that only added debug logging. The working theory in the open issue was that the kubelet doesn't write a fresh record when an image is already cached from an earlier test.

The timeline

From the failing run (prow job 2080994746846679040):

The record was written. Then it was deleted one second later. The cached-image theory was wrong.

Why only this job

That job's image filesystem sat at about 85% for the whole run — exactly the kubelet's default high threshold for image garbage collection. Every kubelet restart in that job triggered a GC pass. Other jobs never got there.

The root cause

After a GC pass, the kubelet calls post-GC hooks with the list of images that "remain". The ImagePullManager's hook, PruneUnknownRecords, deletes every pulled-record for an image that isn't in that list.

The list was built inside freeSpace(), from the images it evaluated. But images in use by a running container, and pinned images, are filtered out before freeSpace() ever sees them. And freeSpace() stops early once it has freed enough bytes, so images after that point were never added either:

// imagesInEvictionOrder, simplified
for image, record := range im.imageRecords {
    if isImageUsed(image, imagesInUse) {
        continue          // in-use images never reach freeSpace()...
    }
    if record.pinned {
        continue          // ...and neither do pinned ones
    }
    images = append(images, evictionInfo{id: image, imageRecord: *record})
}

So the list of "remaining" images was really "the eviction candidates GC looked at and kept". The image the test pod was running wasn't on it, and its record got pruned.

It fails safe — a later pod just has to re-verify credentials — but it's wrong, and it's invisible: the deletions weren't logged.

The fix

Pass the hooks every image the GC manager still tracks, not just the candidates it evaluated:

// remainingImages returns the IDs of all images still known after a GC pass.
// This must include in-use and pinned images, not just the eviction candidates
// that survived: post-GC hooks treat anything missing from the list as gone.
func (im *realImageGCManager) remainingImages() []string {
    im.imageRecordsLock.Lock()
    defer im.imageRecordsLock.Unlock()

    // The same image can be tracked once per runtime handler.
    imageIDs := sets.New[string]()
    for imageKey := range im.imageRecords {
        imageIDs.Insert(getImageIDFromTuple(imageKey))
    }
    return sets.List(imageIDs)
}

Both GarbageCollect and DeleteUnusedImages now call im.runPostGCHooks(ctx, im.remainingImages(), freeTime), and pruned records are logged at V(4). The new TestPostGCHooksReceiveAllRemainingImages fails on master — the hook receives an empty list — and passes with the change.

Status (Oct 10, 2026): open, awaiting triage and review from SIG Node.

Bug 2: the swap test that hung nodes for hours

PR: kubernetes/kubernetes#142850 · related to #138226

The symptom

A swap e2e test, "should be able to use more than the node memory capacity", sometimes passed, sometimes failed, and sometimes produced a run that ended in exit status 255 with no JUnit results at all — the node had stopped responding.

The numbers didn't add up

The test allocates memory equal to the node's full capacity — 8 GiB on the CI node, which also has 1 GiB of swap. The system needs some of that RAM too, so part of the pod's allocation can't fit in memory and has to go to swap. In my reproduction below, when the pod was allowed to finish, about 761 MiB of it ended up in swap.

Under LimitedSwap, a container's swap limit is proportional to its memory request:

swap limit = (memory request ÷ node memory) × node swap

Request	Swap limit	Against ~761 MiB of overflow
30% (before)	~307 MiB	too small
90% (after)	~921 MiB	fits

So in every run, the pod hit its swap limit with memory still left to place. What happened next varied — which is why the test flaked instead of failing every time:

• The OOM killer stepped in quickly. The run recovered, and sometimes the test passed.
• It didn't. The kernel tried to reclaim memory elsewhere. On this image system.slice and user.slice can't swap, so it thrashed the page cache instead — and the node hung for hours. Those are the exit status 255 runs with no JUnit results.

The serial console of one failed run showed exactly this: the pod pinned at its swap limit, with roughly 700 MB of swap still free on the node.

Reproducing it

On a VM set up like the CI node (8 GiB RAM, 1 GiB swap, Ubuntu 24.04, the same slice swap settings): at 30% the pod hit its swap limit and was OOM-killed. At 90% it finished allocating with about 761 MiB in swap, with no OOM and no stalls.

The fix

Three changes, all test-only:

1. Request 90% of memory, so the pod's swap limit can hold the overflow.
2. Lower the memory.available hard-eviction threshold to 100 Mi for these tests. Under node-level pressure, swap only starts a few seconds before the default 250 Mi threshold would evict the pod — before the test can see any swap usage.
3. Delete the pod with DeleteSync, so it's gone before the kubelet restarts in AfterEach. Nothing kills the pod while the kubelet is down.

Status (Oct 10, 2026): open, triaged as important-longterm, with the swap tests' original author asked to review.

Bug 3: the 3.9 GB image that never finished pulling

PR: kubernetes/kubernetes#142802 · related to #141200

The symptom

The GPU sanity test (nvidia-smi plus a CUDA demo) timed out after 30 minutes with the pod stuck in ErrImagePull. Even passing runs spent up to 24 minutes pulling the image.

The containerd log

cancel pulling image docker.io/nvidia/cuda:12.5.0-devel-ubuntu22.04 because of no progress in 5m0s
stop pulling image ...: active requests=1, bytes read=3918316892
failed to extract layer (... sha256:496b16dd...) to overlayfs as "extract-...": context canceled

The image is about 3.9 GB, and its largest layer is about 2.5 GB. The log shows 3.9 GB had been read before the pull was cancelled during layer extraction. So the expensive part wasn't the network — it was unpacking the big layer, which reported no "progress" to containerd's five-minute watchdog. The cancelled pull didn't reuse what it had already fetched, so each kubelet retry started the download again. That's containerd/containerd#13909, with a proposed fix upstream.

The fix

Use a smaller image. On amd64, the test only needs nvidia-smi (which comes from the driver) and the cuda-demo-suite-12-5 apt package, whose binaries link the CUDA runtime statically. The ~90 MB nvidia/cuda:12.5.0-base-ubuntu22.04 image has the same CUDA apt repository, keyring and environment. Other architectures build samples with nvcc, so they keep the devel image, and the pod is pinned to the GPU node's architecture so the image always matches the script's uname -m branch.

I left arm64 alone because I couldn't test it on arm64 GPU nodes. It can still hit the containerd behavior.

Status (Oct 10, 2026): open, triaged as important-longterm, awaiting /ok-to-test and review.

Bug 4: a second bug inside the 1.37 kubelet memory regression

PR: google/cadvisor#3946 · issue kubernetes/kubernetes#142874

This one isn't a flake. A SIG Node contributor reported 20–30% higher kubelet memory in Kubernetes 1.37, traced to cAdvisor — the library the kubelet uses to collect container stats — and wrote the main fix. My part is a separate, smaller bug I found while reproducing it. To keep the credit straight:

	What it fixes	Author
google/cadvisor#3944	The reported cause: changing counters that stop housekeeping from ever backing off	the reporter
google/cadvisor#3946	A back-off reset bug that makes the interval oscillate even when stats are unchanged	me

The reported cause

cAdvisor polls each container's stats on a "housekeeping" interval. When stats haven't changed, it backs off — doubling the interval up to a maximum. When they change, it drops back to the baseline.

MemoryQoS, default-on in 1.37, sets memory.high on containers. The kernel then reclaims memory continuously, which keeps advancing counters in memory.stat. cAdvisor compared whole stats structs, so they never looked unchanged, and housekeeping never backed off. The reporter's fix, google/cadvisor#3944, excludes those counters from the comparison.

The second bug underneath

I reproduced the regression and confirmed #3944 fixes the counters. But while simulating it, I found the back-off had a bug of its own:

if stats[0].StatsEq(stats[1]) && (cd.housekeepingInterval < cd.maxHousekeepingInterval) {
    cd.housekeepingInterval *= 2
    // ... cap at max ...
} else if cd.housekeepingInterval != *HousekeepingInterval {
    // Lower interval back to the baseline.
    cd.housekeepingInterval = *HousekeepingInterval
}

Once the interval reaches the maximum, interval < max is false — so even with unchanged stats, execution falls into the else branch and resets to the baseline. Instead of staying backed off, the interval cycles: with cAdvisor's defaults, 1s → 2s → 4s → … → 1m → 1s; with the kubelet's settings, 10s → 15s → 10s → 15s.

The fix separates the two conditions, so only changed stats reset the interval:

if stats[0].StatsEq(stats[1]) {
    if cd.housekeepingInterval < cd.maxHousekeepingInterval {
        cd.housekeepingInterval *= 2
        // ... cap at max ...
    }
} else if cd.housekeepingInterval != *HousekeepingInterval {
    cd.housekeepingInterval = *HousekeepingInterval
}

The new TestNextHousekeepingIntervalDynamic steps through unchanged and changed samples. Run against the original code, it fails exactly where the sawtooth starts:

In my simulation of a 10-minute run with the kubelet's settings and MemoryQoS-style advancing counters, a container got 60 housekeeping ticks on the current code, 48 with #3944, and 41 with both fixes. Most of the improvement comes from #3944; mine removes the oscillation that remains. Neither patch alone gets the kubelet to back off as intended, and the simulation measures housekeeping ticks, not the production memory figure in the report.

Status (Oct 10, 2026): #3946 is open; the issue is assigned to me, and the reporter called the patch a good one.

What the four had in common

Each bug lived at an edge that normal runs never reach:

• A list that silently meant "some of everything". "Remaining images" excluded in-use and pinned images, and nothing downstream knew.
• A number that couldn't work. A swap limit smaller than the overflow it had to hold.
• A timeout that measured the wrong thing. A "no progress" watchdog applied to an unpack that reports no progress — and the fetched data not reused.
• An else branch that fired at the boundary. Correct below the maximum, wrong exactly at it.

And each one was explained by a difference: the one job at 85% disk, the one test whose request didn't match its allocation, the one image that was 40 times bigger than it needed to be, the one comparison that never settled.

Flakes aren't noise. They're bugs that only show up under one environment's conditions. The job that fails is telling you which conditions.

What the evidence does and doesn't establish

Evidence	Establishes	Does not establish
Prow timeline for bug 1	The record was written, then pruned by GC one second later, in that run	That every historical failure of these tests had this cause
TestPostGCHooksReceiveAllRemainingImages	The hook now receives in-use and pinned images	That the fix has been reviewed or accepted
VM reproduction for bug 2	30% fails and 90% holds on a CI-like machine	Behavior on every node image the swap tests run on
containerd log for bug 3	3.9 GB had been read before the pull was cancelled during layer extraction	That every layer had fully downloaded, or that the smaller image never hits the same timeout
cAdvisor test, failing on original and passing with the fix	The back-off reset bug exists and the change removes it	The production memory savings
60 / 48 / 41 tick simulation	The relative effect of #3944 and #3946 in my model	The 20–30% memory figure, which comes from the reporter's environment

Check each one yourself

Bug 1 — kubelet image GC. Check out the PR branch and run the new unit test. Without the fix, the hook receives an empty list:

go test ./pkg/kubelet/images/ -run TestPostGCHooksReceiveAllRemainingImages -v

Bug 2 — swap test. This is a node e2e test, so it needs a Linux node with swap enabled. The PR describes the VM I used (8 GiB RAM, 1 GiB swap, Ubuntu 24.04). From a Kubernetes checkout:

make test-e2e-node FOCUS="should be able to use more than the node memory capacity"

Bug 3 — GPU image. The full test needs a GPU node; in CI it's the pull-kubernetes-e2e-gce-device-plugin-gpu job. The size difference behind the fix is easy to see anywhere crictl runs:

crictl pull nvidia/cuda:12.5.0-devel-ubuntu22.04   # ~3.9 GB, ~2.5 GB largest layer
crictl pull nvidia/cuda:12.5.0-base-ubuntu22.04    # ~90 MB

The cancellation itself is containerd's image_pull_progress_timeout (five minutes by default) firing during extraction; containerd/containerd#13909 has the details.

Bug 4 — cAdvisor back-off.

git clone https://github.com/google/cadvisor && cd cadvisor/lib
# add TestNextHousekeepingIntervalDynamic from the PR, then:
go test ./manager/ -run TestNextHousekeepingIntervalDynamic -v

On the original code it fails at step 3; with the fix applied, it passes.

---

I'm Sachin, an SRE / platform engineer working on Kubernetes, observability and distributed systems — currently open to new roles. LinkedIn · GitHub