DEV Community: Nahum Litvin

My fix for this hang was already running. The sandbox hung anyway.

Nahum Litvin — Thu, 22 Oct 2026 12:23:21 +0000

gVisor shifts its stub memory by a random gap, but since 2023 one queue was not moved with it. When the gap is small, that queue can land on a thread's stack and the sandbox can hang for good. The fix was one line.

Same 16-thread workload. Same arm64 VM. Same forced memory layout. Stock gVisor hung 2 runs out of 2, with a stub process stuck at 100% of a core. Add one line, and it ran clean 20 times out of 20.

The only variable was where one queue sat in memory.

Getting to that line took me through my own safety net first. In August I had shipped a fail-stop to gVisor: if an app thread stays stuck for 30 seconds and gVisor's interrupt doesn't free it, gVisor kills the one stub process running it instead of letting it hang the sandbox. A month later a handful of sandboxes hung with that fail-stop running. The fix is google/gvisor#15573.

gVisor, the short version

Deeper intro: gVisor, a kernel in Go.

gVisor runs a second kernel, the Sentry, written in Go, between the app and Linux. On its default platform, systrap, the app's code runs in host processes called stubs. The Sentry puts work for the stubs on a queue in shared memory, and every system call traps back to it.

context: one app thread's saved state, waiting to run or waiting for the Sentry.
interrupt: a signal the Sentry sends to the stub thread running a context, to force it back.
spinning queue: idle stub threads busy-waiting for work, private to the stub. Each entry is stamped with when it started spinning.

What did I think it was?

The bug from last time (gVisor: one lost signal): one lost interrupt left an app thread stuck. Same symptoms: pods stuck in Terminating, every Sentry thread asleep, hundreds of kills from Kubernetes queued behind one app thread that never paused.

One thing didn't match. Last time the stuck sandbox burned no CPU. This time a stub was burning a full core. And my fail-stop was supposed to catch exactly this kind of hang.

Why didn't my fail-stop fire?

A core dump of the Sentry showed one app thread waiting on its context, with AckedTime=0 and an owner of 0xfefefefe, the "no thread" marker. No stub had picked it up since it was queued.

The fail-stop only fires once the Sentry has sent an interrupt, and the Sentry only interrupts a stub thread that took the context. Nobody took this one. No interrupt, no fail-stop.

So my safety net had a hole, and it was mine. It covered a stub that took the work and then stopped responding. It didn't cover work no stub ever took.

What was the stub doing at 100% CPU?

Every sample of the busy stub landed in one tight loop in spinning_queue_remove_first. It waits while the oldest entry's timestamp reads 0, since a new entry writes its timestamp last. Here it read 0 every time, and that check sits before the timeout check, so the loop never gives up.

Its other threads never took the context either. From the code, the Sentry only wakes another stub thread when none looks available (subprocess.go). The stuck spinner still counted as one.

Where was the queue, really?

In the stub's /proc/<pid>/maps, the spinning queue's address wasn't in its own private page. It sat inside a per-thread stack mapping. Its start field read 0x7ff9, which is where the high half of a 0x00007ff9... pointer lands when a stack stores one there. The queue was reading a thread's stack as queue data.

How does a queue end up on a stack?

gVisor puts a random gap into the stub's memory layout, so attackers can't guess where things are. Every region after it has to move by the gap. Here is stub_unsafe.go before the fix, with the missing line marked by me:

gap := uintptr(rand.Uint64()) * hostarch.PageSize %
    (maximumUserAddress - stubStart - mapLen)
stubSysmsgStack += uintptr(gap)
stubContextQueueRegion += uintptr(gap)
stubContextRegion += uintptr(gap)
// missing: stubSpinningThreadQueueAddr += uintptr(gap)

Three regions move. The spinning queue, between them in the layout, stays put. With a big gap it sits alone in empty space. With a small one, the shifted stacks slide over its old address. It's been that way since the commit that introduced the spinning queue, in March 2023.

It's an availability bug, not an isolation one: the queue lands on memory the stub could already write, which the Sentry treats as untrusted anyway. The damage stays inside the sandbox.

Why did it hide for three years?

The gap is random below the room M left above the stub block, and M is itself random up to R, about 1 TiB on x86-64. The queue gets hit when the gap is smaller than the stack area T, about 612 KiB for a 16-CPU sandbox:

P = (T / R) * (1 + ln(R / T)) ~ 1 / 114,000

About 1 in 100,000 starts at 16 CPUs, and about 1 in 35,000 at 64. Waiting for it was useless.

Can I prove it with one variable?

So I forced it: a debug-only patch picks a gap that puts the old queue address inside a live thread stack.

Build, same forced gap	Queue lands on a stack	Result
Stock gVisor	yes	hung 2 of 2, stubs at 100% CPU, no fail-stop log line
Stock + `stubSpinningThreadQueueAddr += uintptr(gap)`	no, it moves with the gap	clean, 20 of 20

The code is shared by both architectures; arm64 was just faster to boot. The fix adds a layout test, and the repro tool is here.

Are you affected?

Systrap sandboxes on release-20230417.0 through release-20261005.0 can hit it; google/gvisor#15572 has details by version. The tell: a stub pinned at 100% forever, a pod stuck in Terminating, runsc kill hanging.

1. Upgrade to any runsc build that contains c7f458aa3, the first release after release-20261005.0.

2. Until then, --systrap-disable-fast-path (on release-20260622.0 or later) stops stub threads from spinning, so the spinning queue is never used. It's a partial workaround: it costs syscall latency, and a much rarer tiny gap can still hit the context queue. Only the upgrade fixes that.

3. To clear a hung sandbox, kill its Sentry on the host: the runsc-sandbox process ending with the pod sandbox ID from crictl pods. Its stubs die with it.

What did the AI do?

AI agents read the dump, mapped a program counter to a source line, built the VM repro and drafted the upstream PR. I made the calls, starting with not stopping at the hole in my own fail-stop.

A few things I'm keeping:

Run the cheap check before the theory. One busy core killed my first guess.
Your own safety net has holes, so look there first. Mine only covered work a stub had taken.
Your debugging tool can kill the patient. A gcore of the Sentry faults in the sandbox's shared memory and can OOM it; dump the stub first.
When a bug needs bad luck, make the bad luck happen on demand. A forced gap turned a 1 in 100,000 hang into 2 of 2.

The line is merged. That queue can't land on a stack anymore.

Thanks to konstantin-s-bogom for reviewing and approving the fix, and to avagin, who wrote the spinning queue and fixed the same symptom from a different cause in 2024 (google/gvisor#10000). Investigation assisted by Claude Code, with Codex as a second reviewer.

STPA-inspired questions tied a plain checklist in my coding agent. Their one win: Claude asking about timing.

Nahum Litvin — Fri, 16 Oct 2026 07:24:41 +0000

STPA is a safety method from MIT that asks when an automatic action becomes unsafe. I put its questions into my AI coding agent's rules, then replayed three real past changes with no rule, with a plain "check what this could break" rule, and with STPA. On average STPA did no better than the plain rule. It won clearly in one case, where the risks were about timing. Only with STPA did Claude ask whether a timeout was too long (3 of 6 grades, against 0) or when to stop waiting (4 of 6, against 0).

The test

My first version of this post said STPA helped. One question, whether a change breaks apps outside its target, had jumped from 1 in 6 to 5 in 6 in a single replay of case A. Two reviewers asked the obvious thing: would any rule that says "check what you might break" do the same? So I ran it properly. The two rules, word for word.

The STPA rule (62 words):

When a plan or design adds or changes something that acts automatically (scaler, quarantine, drain, hook, mutating script), list each control action and ask: unsafe if missing, if wrong context, if early/late, if it stops too soon or runs too long. Name the feedback it acts on and what happens when that feedback is stale or empty.

The plain checklist (35 words):

Before shipping any change, check what it could affect beyond its intended target, how you would notice if it goes wrong in production, and how you would roll it back quickly.

The bugs are the ones each change's real review or production run found. The judges agreed on 90% of calls.

The three changes, in generic form. A: a controller taking over a value another controller used to write. B: automatic recovery for a worker that stops responding. C: a dispatcher that picks workers from a cache that can be stale.

First, the headline from my first version. In case A, regraded with stricter blinding: no rule 4 of 6, plain list 6, STPA 6. The gap shrank, and the plain list closed it too.

Orange: STPA. Both models pooled. Across all three, the plain list and STPA tie at 6.1.

The plain list helped a little everywhere. STPA beat it only in B, and fell below no rule in C.

STPA's "too late" and "applied too long". Out of 6 judge grades, 3 plans each graded twice. Without STPA, zero.

In B, the one thing the method added for Claude was questions about time. Codex asked about the timeout even with no rule, and more with either rule. But neither model asked when waiting stops helping, except Claude with STPA. One honest limit: the STPA rule names timing ("early/late", "runs too long") and the checklist doesn't, so this test can't tell STPA apart from a rule that just mentions time.

In C, STPA cost Codex the basics, even though Claude gained a little.

With STPA, Codex dropped all three. The plain list kept them.

Where the rule came from

My agent learns from a git repo of rules, and until recently almost every rule came from something that had already gone wrong. One says "never conclude from a zero". During an incident, a metrics query with a wrong label returned nothing, with no error. The graph looked calm while the thing it measured was on fire.

On 13 October I was at SREcon in Dublin, listening to John Thomas of MIT, co-author of the STPA Handbook. His talk: What Your SLOs Can't See: Applying STPA to a Distributed Software Service. My laptop was open.

The idea comes from Nancy Leveson at MIT. Her accident model, STAMP (System-Theoretic Accident Model and Processes, 2004), later expanded in her book Engineering a Safer World (2011), treats safety as a control problem.

STAMP's loop. Accidents come from inadequate control: a wrong picture, flawed logic, or missing feedback.

STPA (System-Theoretic Process Analysis) is the hazard analysis built on it. For every control action you ask when it becomes an unsafe control action (UCA): not provided, provided when it causes a hazard, too early, too late or out of order, stopped too soon or applied too long.

I asked my agent if any of it applied to me. It said: "You're doing half of it, and only after the fact." So I put the questions where the agent already decides:

The post-mortem step borrows from CAST, Leveson's accident analysis, also built on STAMP.

That is mostly STPA's third step. No losses, hazards or control structure written down. A real STPA starts there. Mine doesn't, yet.

A second model and a review bot then found my rules breaking their own method: a post-mortem line that only blamed bad feedback, a missing UCA type, and a writer check placed after the writers act. A check that runs too late, written during a talk on what monitoring misses. I fixed it and merged before the lecture ended.

What I kept

A plain checklist does most of the work. Across the three cases it matched STPA, 6.1 to 6.1, against 5.7 with no rule. What I keep is the timing questions, whether the credit goes to STPA or just to naming time: my checklist now adds "too early, too late, too long". For Claude nothing else raised them, 0 of 6 without and 3 and 4 with. And I now watch what a method pushes out. In C, Codex's three operational checks went to 0 of 6 with STPA.

Three changes is not a study, but it was enough to retract my first headline.

Credits and links

The talk: John Thomas, What Your SLOs Can't See: Applying STPA to a Distributed Software Service, SREcon26 EMEA, Dublin, 13 October 2026.
STPA Handbook, Nancy Leveson and John Thomas (2018). CAST Handbook, Nancy Leveson (2019).

Originally published at catchkill9.dev, where the diagrams follow your light or dark theme.

I built it for my terminal. A week later, strangers were fixing it for each other

Nahum Litvin — Fri, 09 Oct 2026 00:00:00 +0000

Friday 2 Oct: built for me

Every day I watch Claude Code print tables as pipes and dashes. prismantis is my fix: a Claude Code mod that redraws replies with colored tables, highlighted code and mermaid charts. I launched it on 2 Oct and shipped 8 releases that afternoon, most of the code written by Claude with me saying "almost" at screenshots.

I use it all day. That is the whole reason it exists. Every reply in my terminal now draws the way I wanted it to, and a chart showing up where a wall of text used to be makes me smile every time.

It was tested on exactly one setup: my Mac, my terminal, my window size.

Sunday 4 Oct: other people

16 stars in one day. The first forks showed up. I had no idea who these people were or what terminals they ran, and that was the first time it felt like it wasn't only mine.

Monday 5 Oct: a bug I could never see

@willowite opened #6. On Windows, the ⏺ reply marker falls back to the Segoe UI Emoji font, so Windows Terminal draws a blue emoji and ignores the theme color. The report named the line, the codepoint and the fix: use ●, which prismantis already used for tool rows.

I don't have a Windows box. Claude doesn't either. It was fixed and released the same afternoon, and I couldn't have found it in a year.

Tuesday 6 Oct: the best report, and the first PR

@cesar32az opened #28, the best bug report I've had on anything: versions, OS build, two terminals, settings, steps, --debug output. Copy buttons drew fine and did nothing. Two releases came out of it, including keyboard copy commands, because the report also showed my README was wrong. Then they found the cause on their own side, a CLAUDE_CODE_DISABLE_MOUSE=1 left in ~/.zshrc for months, and apologized for the noise. No apology needed. The keyboard commands are how I copy now.

That evening @Malorn44 sent #38, about 50 lines that let another mod hand prismantis some markdown and get it drawn in the user's theme. I built a renderer for my own replies. Now other mods can draw with it. Merged and shipped the same evening.

Wednesday 7 Oct: the Forks page

In the morning I opened the repo's Forks page, mostly out of curiosity. 6 forks. 3 had real work nobody had sent me: @vrevolverrr had LaTeX math and a fix for table borders, @rooterkyberian had tables that copy into docs as real tables, @AlexeyHRDesign had made the whole thing work in the Claude desktop app.

That is normal. You hit a pain, fix it for yourself, move on. Opening a PR on a stranger's repo costs something, and nobody pays it without a reason.

GitHub has no DMs and their forks had Issues turned off, so I opened one issue per fork on my repo and @mentioned the author. This is the one to @vrevolverrr, #43:

@vrevolverrr love the LaTeX math in your fork (d49e0c9). Typeset formulas via RaTeX in kitty/Ghostty, with a text fallback everywhere else, is exactly right.

Would you open a PR here? Happy to review it fast. If you'd rather not, I can port hooks/latex.ts and the $$ parsing myself and credit you as co-author on the commit.

Specific praise, one ask, a promise I could keep, and an easy out.

Within 10 minutes @AlexeyHRDesign took the out: they're building their own desktop mod, port it with co-author credit please. Then they left a list of what didn't hold up in daily use, including a stray \x08 in a regex where \b was meant. I valued that list as much as the code.

In under 20 minutes @rooterkyberian's PR #47 was up, and @vrevolverrr answered that the border fix was coming first. In the same reply they warned me their LaTeX work runs an external program, which breaks a rule in my own repo. They flagged it before I noticed.

Wednesday, an hour later: two strangers and one bug

Without any invite, @b33eep had opened #46 for the same border bug. Wrapped table rows lost their borders, and in a narrow terminal, as they put it, "that is almost every table". I work in a wide window. I'd never have seen it.

So two people I'd never met had fixed the same bug two different ways, within half an hour of each other. I pointed @vrevolverrr at #46 so they wouldn't do it twice. Then I mostly watched.

@b33eep ran both branches side by side in a terminal without clickable links and found a row where @vrevolverrr's version drew an extra line. @vrevolverrr reproduced it and proposed something better than either fix. @b33eep agreed and asked them to fold their own border idea into #48. It was folded in, re-tested, and @b33eep closed their own PR in favor of the combined one:

thanks for picking this up so quickly and for the co-author credit. Dropping the wrapping for link rows was a better idea than mine, and it was good working through this with you.

About an hour, start to finish. The best review of the week happened between two strangers. I just merged it.

Then @b33eep sent #49, the column-width fix I'd asked for in #41, rebased on top of the merged #48. @vrevolverrr sent #51 for a Serena tool name drawn with "serena" in bold, a 9-line fix for a tool I don't run. I merged it within the hour.

The rest of the week: leading it

My job changed that week. I wrote less code and made more calls.

@rooterkyberian's clipboard PR was solid but touched macOS and Linux at once, so I asked to split it. They had #54 and #55 up the same evening. Both are merged.

@vrevolverrr's LaTeX PR, #50, brought back running an external program, the one thing I'd removed earlier on purpose. I asked for six changes: one switch, off by default, nothing runs unless you turn it on. They sent all six, one commit per point. Merged on 9 Oct. Math now draws as real formulas in kitty and Ghostty, for anyone who opts in.

Most of it was saying yes fast, sometimes saying no kindly, and holding the few rules that keep a mod safe to install. I didn't expect to enjoy that part this much.

What I took from it

Agents made the code cheap. 19 releases in 5 days says more about what code costs now than about me.

The pain is the part I can't generate. Windows fonts, narrow terminals, Serena tool names, the desktop app, pasting tables into docs, math. None of it shows up on my Mac. No prompt would have written issue #6.

So ask the forks. 3 of 6 had fixes nobody told me about. One friendly issue each got me PRs, a list of traps, and a front-row seat to two strangers making each other's code better.

Thanks to @willowite, @cesar32az, @Malorn44, @b33eep, @vrevolverrr, @rooterkyberian and @AlexeyHRDesign. prismantis is better than anything I'd have built alone, and it's the first thing I see every time Claude answers me.

Nahum Litvin is a Technical Lead - DevOps Engineering at Wix.

Claude Code mod: themes engine, charts, graphs, colors. All you ever wanted.

Nahum Litvin — Fri, 02 Oct 2026 14:43:06 +0000

Claude Code mod: themes engine, charts, graphs, colors. All you ever wanted.

claude plugin marketplace add NahumLitvin/prismantis
claude plugin install prismantis@prismantis

https://github.com/NahumLitvin/prismantis

One lost signal, five days stuck, 45,000 frozen threads: fixing a gVisor hang upstream

Nahum Litvin — Tue, 29 Sep 2026 00:00:00 +0000

It looked like a deadlock in gVisor. It was a single missed wake-up, hiding behind a symptom teams have been reporting since 2021, and two strangers and a maintainer fixed it in about a week.

The hang in one picture: one signal, no retry, and a deadline check whose only action is a log line.

On a Tuesday morning our alerting claimed that hundreds of pods were stuck Terminating in production. The real number was one. The rule counts kubelet failed-kill events rather than pods, and a sandbox that never dies gets retried forever, so a single wedged pod read as a fleet on fire. That pod turned out to be the interesting part.

At Wix, our Velo grid runs users' backend JavaScript as untrusted code inside gVisor sandboxes on Amazon EKS. gVisor puts a userspace kernel between that code and the host kernel, so untrusted code never talks to the host kernel directly. Thousands of sandboxes per node group, short lifecycles, heavy churn. Deleting a pod should take seconds. This one had been Terminating since 07:11. kubelet's kill requests got DeadlineExceeded every 2 minutes and would have continued forever. We intervened by hand after about 90 minutes. A week earlier, several pods on a few nodes had sat like that for days.

Why nobody's timeout saved us

Before the forensics, the shape of the problem. Deleting a pod is a relay: each layer asks the next one, politely, to make something die. Every layer should hold two more things beyond the polite ask: a bound on how long it waits, and an escalation that works without the cooperation of the thing it is waiting on. In our case the relay runs from kubelet (the Kubernetes node agent) to containerd (the container runtime) to the gVisor shim (the per-sandbox process containerd talks to) to the sandbox itself. Here is that stack, graded on all three:

Three arrows per hop: blue attempts, amber detects, violet recovers. Blue failed once, at the bottom. Amber either did not exist, fired into a log line, or fired and could only re-send blue. Violet existed nowhere, so one lost message at the bottom of the stack became the whole stack's hang.

Look at the violet column. Every escalation asked the very component being escalated against to cooperate. containerd's dead-shim cleanup only fires when the shim disconnects. Ours kept answering Connect and State, which never touched the lock, while Kill and Stats blocked behind it. The connection stayed up, so cleanup never fired. Alive enough to prevent its own cleanup.

What a frozen sandbox looks like

That userspace kernel is called the sentry. On systrap, the platform we run, the app's threads run in stripped-down stub processes and the sentry coordinates them. Our wedged pod had a sentry that was alive but answering nothing. runsc kill, gVisor's own command for stopping a sandbox, hung. runsc debug --stacks, the tool that is supposed to tell you why something hangs, also hung. The panic log was 0 bytes.

The node told the rest of the story. Load average climbing about 3 per hour while actual CPU sat near 30%. Hundreds of sentry threads parked, waiting on shared memory. Stub processes accumulating. On the worst nodes, load kept climbing for days with the machine mostly idle. The only remediation that worked was SIGKILL to the whole sandbox process tree, by hand, on the node.

Stub processes pile up behind the frozen sentry: load average keeps climbing while the CPU does almost nothing. Threads stuck waiting, not working: the rising top panel against the flat bottom one is the bug.

So we filed gvisor#14408 with what we had: the outside view, two affected releases, and an honest admission that we could not get stacks because the debug tooling was wedged along with the sandbox. Our proposed fix, PR #14201, had been open since the week before.

Then a stranger showed up with the other half

The same day, another engineer filed gvisor#14405. Same bug at a different company, and they had the one thing we could not get: a full dump of the sentry's goroutines, Go's lightweight threads, from inside a frozen sentry. Their dump showed one of the sentry's internal worker threads sitting in the same wait loop for over five days, waiting for a stub thread that had missed a single wake-up signal.

In their shim, about 45,000 waiting threads were queued behind one lock held by a kill that would not finish, and their containerd and kubelet memory grew until nodes ran out.

The bug itself is painfully simple. The sentry sends the stub one interrupt signal. One. If that signal is lost, and nobody has yet pinned down why it sometimes is, the sentry keeps waiting for the answer with no retry and no escape. There is even a 30 second deadline in the code that notices the wait is too long. Here is the whole thing it did about it:

if time.Now().After(deadline) { log.Warningf("Systrap task goroutine has been waiting on " + "ThreadContext.State futex too long. ...") // no retry, no escalation }

Where did 45,000 waiting threads come from? cAdvisor kept polling every container's Stats. Each request added a goroutine that blocked behind the hung kill's lock. Its caller timed out, but a mutex wait cannot observe cancellation, so the goroutine stayed. Over five days, 45,000 accumulated: roughly one every 10 seconds, the scrape interval fossilized in a thread dump. The metrics system asking "how are you?" every 10 seconds had pushed the shim to about 600MB.

A timeout that only logs a warning is not a timeout. It is a diary.

And this exact wait sits on the teardown path. Killing a sandbox starts with a freeze-everything step that waits for every worker thread to park, so one stuck worker means the kill command can never answer, the shim call hangs, kubelet gets DeadlineExceeded, and each retry parks another goroutine behind the first one. The pod stays Terminating for days.

So, how did we fix it?

The merged fix escalates in stages, gentlest first:

1. Tap again. The interrupt is resent on each 5 second checkup wakeup instead of once. When the resend lands, a lost signal costs the workload a stall of a few seconds, then it carries on.

2. Photograph the scene. If the stub is still unresponsive after the 30 second deadline, the sentry dumps every internal stack trace to the log. A frozen sandbox can not be debugged from outside unless it was started with --panic-signal; ours were not, theirs were, which is where their dump came from. So this failure now documents itself. The next team to hit anything like this gets for free what took two companies a week to assemble.

3. Remove only the broken part. Then the sentry kills just the one stuck subprocess, through the same path it already uses when a stub dies naturally. The blocked task unwinds, teardown can finish, and the healthy subprocesses in the sandbox are untouched.

My first version killed the entire sentry. gVisor maintainer Konstantin Bogomolov pushed for killing only the stuck subprocess, sparing healthy neighbors. He was right. The other reporter had raised the same concern earlier that day. Konstantin also caught a dropped continue, which led me to a race: a context that recovered during the wait could still get killed. Three review rounds in under a week took the PR from nobody having looked at it to merged on master.

The merged fix, staged from least to most disruptive. My first version jumped straight to killing the whole sandbox; review scoped it down.

The part that bothers me

After the merge, we found reports since 2021 from a Cloudflare engineer, a GKE user, Knative, Talos Linux, Arista and others on the containerd tracker. Roughly eight teams over five years. Causes differed, and some cases got fixes. But the pattern kept returning: pods stuck Terminating, debugging tools hanging, someone killing processes by hand, and more often than not the thread ends without a stack trace from inside the sandbox.

Two teams filing in the same week broke that pattern: our production evidence and proposed fix met the other team's internal stack dump.

Five years of the same symptom, with different root causes underneath. Most threads ended with the manual workaround, until two half-reports completed each other.

The relay diagram near the top of this post still isn't fully green. The merged fix repairs the layer where the hang started. We filed gvisor#14548 to bound shim kill waits, then SIGKILL the sandbox tree, and containerd#14081 to SIGKILL the shim after repeated kill RPC timeouts. In review of the shim PR, gvisor#14549, the same maintainer asked to narrow it to bounded waits on Kill, Stats and Status, because killing the sandbox from the shim would take out healthy containers too, the same over-reaction he flagged in the first fix. That revision is in progress, and escalation stays open. Either escalation, as originally filed, would have capped our incident at minutes.

Lessons learned

A watchdog that only barks is not a watchdog. If code detects a should-never-happen state, it must act: retry, self-report, fail. Grep your own codebase for deadline checks whose only body is a log line. This one surfaced because a sandbox sat stuck for five days behind it.
File the issue even with half the evidence. Our report had no stacks and said so plainly. Within a day a stranger's independent report supplied them, and their dump changed the fix. The half-report you are embarrassed to file is someone else's missing half.
Remediate the smallest thing that unblocks you. Kill the subprocess, not the sandbox. The instinct under pressure is the big hammer. Review cut the kill from the whole sentry to one subprocess. The same restraint belongs in your incident runbooks.
Audit your escalation paths for circular dependencies. Walk your stack with that diagram's two questions: bounded wait, and escalation that works without the target's cooperation. If the answer to the second bottoms out at "a human with SSH", write that down, because that is your actual design.

The fix shipped in gVisor release-20260831.0. This one has an ending, and the next report of this hang will arrive with its stacks attached.

This post was written by Nahum Litvin.

A zero timeout in Go is not no timeout

Nahum Litvin — Tue, 29 Sep 2026 00:00:00 +0000

While working on snapshotter timeouts in containerd, I assumed a timeout of 0 meant "no limit". It doesn't.

ctx, cancel := context.WithTimeout(context.Background(), 0)
defer cancel()
fmt.Println(ctx.Err()) // context deadline exceeded

The deadline is now, so the context is expired before the first call. containerd's timeout.WithContext passes the configured value straight to context.WithTimeout, so a key set to 0 fails every call on the spot.

That's why the new default_timeout for proxy snapshotters is off when empty, and only wraps calls whose context has no deadline yet: https://github.com/containerd/containerd/pull/14187

A 12 second go run settled it. Cheaper than arguing about it.

containerd part 3: the fix got reverted

Nahum Litvin — Fri, 25 Sep 2026 00:00:00 +0000

People share their victories. A month ago I shared mine: my containerd fix got merged after 40 days and 30 review rounds.

13 days later it got reverted. It never made it into a release.

My fix put a timeout on the one call that hung on my nodes. It worked for my incident, and it felt like a win.

So I opened a second PR to cover more of the calls that could hang the same way.

That's where it came apart. Working through it with the maintainers, we saw the real problem was one layer out. The thing hanging was an external plugin, and it can hang on any of its calls. Patching them one at a time would never end.

The right fix is a single timeout where containerd talks to the plugin, covering every call at once. That made my first fix redundant, so with a release about to be cut, Michael Brown reverted it. I agreed.

In Part 2 I said the review is the distance between a patch that's correct for your incident and one that's correct for everyone. Turns out it doesn't stop at merge.

A new PR is on its way. Round 31. Not giving up.

containerd part 2: 17 lines, 40 days, 30 reviews

Nahum Litvin — Tue, 25 Aug 2026 00:00:00 +0000

The fix was 17 lines. Getting it into containerd took 40 days and 30 review rounds, and taught me more than writing it did.

Part 2 of the bricked-node story: the resilience patch got merged upstream this week. My biggest open source contribution to date.

Writing the code was the easy part. The review was where the real engineering happened.

The timeout key got renamed twice, because naming in a project with hundreds of config keys is an API commitment, not a variable name.

The default went from 30 minutes to 5 and back to 30. I pushed hard for 5, a maintainer pushed back with an argument I hadn't considered: the old behavior was unbounded, so on upgrade the least surprising default is the big one, and anyone running remote snapshotters can tune it down. He was thinking about thousands of clusters upgrading. I was thinking about mine.

My test got rewritten around Go's synctest, so it runs in milliseconds on a fake clock instead of mutating global state.

And when CI wedged in the merge queue, Michael Brown restarted the buckets, herded the reviewers, and queued it again until it landed. Maintainers do this invisible work daily, for free, for strangers.

A patch that's correct for your incident and a patch that's correct for everyone running containerd are two different things. The review is the distance between the two.

What's the hardest review you've been through, and what did it fix that the code didn't?

A node that reported Ready and started nothing: four parts, one containerd fix

Nahum Litvin — Tue, 11 Aug 2026 00:00:00 +0000

What happened

In July one of our Kubernetes nodes stopped starting pods for hours. It reported Ready the entire time. The scheduler kept placing pods on it, every new container hung in creation, and nothing on the node's health path looked wrong.

We run untrusted user code at Wix, so pods come and go all day. A node that silently stops starting them is worse than a node that dies: the scheduler keeps feeding it.

The easy answer was a file descriptor limit. The snapshotter logs were full of "too many open files". Raise the limit, restart, move on. I almost shipped that.

This is the whole story in one place: the outage, the two bugs, the fix that got reverted, and the one that stays.

Why it happens

Two bugs lined up.

The first was in containerd. During snapshot garbage collection, containerd calls the snapshotter's Remove while holding that snapshotter's write lock in containerd's metadata layer, with no deadline. It even drops the caller's cancellation. If the snapshotter never answers, the lock is held forever, and every snapshot operation on the node queues behind it, including the one that starts a new pod sandbox. Nothing on the readiness path takes that lock, which is why the node kept saying Ready.

Readiness is a separate path. Kubelet asks the runtime for its status, the runtime answers from memory, and nobody touches the snapshot metadata. So the node was healthy by every check we had, and useless by the only one that mattered: can it start a container.

The second was the trigger, in AWS's soci-snapshotter, the plugin that lazily loads container images. Every file verified on first access opened a reader from the span cache and never closed it. The sibling function about 45 lines up closes the exact same reader. One leaked file descriptor per file, on every node running lazy-loaded images. Under a burst of pod starts the snapshotter hit its limit of 65,535 and stopped answering. containerd was waiting on it, holding the lock.

A proxy snapshotter like soci runs in its own process, and containerd talks to it over gRPC. That boundary is where the real problem lived, though it took me two more PRs to see it.

The goroutine dumps told the story once we knew where to look. Hundreds of goroutines parked on that snapshotter lock, and one GC goroutine holding it, waiting for a gRPC header from the snapshotter that was never coming. The node was not broken. It was waiting, politely and forever.

What we got wrong first

I got the diagnosis wrong twice before I got it right. After the storm the file descriptor count looked low, 151 out of 65,535, so I concluded the limit theory was wrong. Then I flip-flopped back. The leaked descriptors had been reclaimed once Go's finalizers ran, so the steady state hid the peak.

I did most of that investigation with Claude, reading goroutine dumps from the wedged node and two unfamiliar codebases side by side. It didn't write the fix. It made the investigation cheap enough to do properly, including catching my own wrong conclusion.

The bigger mistake came later. My first containerd fix put a timeout on the one call that hung on my nodes, the GC's Remove. It was correct for my incident. It got merged after 40 days and 30 review rounds, and I wrote about it as a win.

Those 30 rounds were the real engineering. The timeout's config key got renamed twice, because in a project with hundreds of keys a name is an API commitment. The default went from 30 minutes to 5 and back to 30. I pushed hard for 5. A maintainer pushed back: the old behavior was unbounded, so on upgrade the least surprising default is the big one, and anyone running remote snapshotters can tune it down. He was thinking about thousands of clusters upgrading. I was thinking about mine.

My test got rewritten around Go's synctest, so it runs in milliseconds on a fake clock instead of sleeping and mutating global state. And when CI wedged in the merge queue, Michael Brown restarted the buckets, herded reviewers and queued it again until it landed.

Then I opened a second PR to cover another call that could hang the same way. Working through it with the maintainers, we saw that I was patching symptoms one call at a time. The thing that hangs is an external plugin, and it can hang on any of its calls. With a release about to be cut, Michael Brown reverted my first fix 13 days after it merged, so it never shipped. I agreed with the call.

The suggestion that changed the plan came in that second PR's thread: stop bounding components one by one, and put the bound where containerd talks to the plugin. Michael agreed. Reverting my first fix was the price of doing it that way, because two mechanisms bounding the same call would have been worse than one.

The fix

soci-snapshotter: close the reader in file Verify. One line, defer r.Close(), merged upstream in July.
containerd, first attempt: bound the GC path's Remove with a configurable timeout. Merged, then reverted before release, because it fixed one call out of many.
containerd, the fix that stays: a default_timeout setting on proxy plugins. When it is set, every call containerd makes to a proxy snapshotter gets that deadline if the incoming context has none. A caller that brings its own deadline keeps it. Leave it empty and nothing changes on upgrade.

The proxy snapshotter has 10 gRPC methods and none of them had a deadline unless the caller already set one. Putting the bound at the client covers all 10 at once, including the GC path my first fix targeted. It also covers paths I never hit in production, which is the point.

Streaming calls needed a decision. Walk gets one deadline for the whole stream, not one per message, so time spent in callbacks counts against it. The content store proxy has the same gap but a different shape: a writer outlives the call that created it, so a per-call deadline would cut an ingest that is still being written. That one needs its own setting and its own PR.

The review on the final PR was shorter, and it still taught me something. I had added an options argument to the exported constructor. Compatible, I thought. The maintainer pointed out it still breaks anyone who stores that function as a value, so the old constructor stays untouched and a new one takes the options.

The SOCI side was the quick part. The PR went in on July 15 and AWS merged it five days later. The containerd side took three PRs and two months. The difference is not the size of the change. A one-line fix to an obvious leak has one right answer. A timeout in a runtime that thousands of clusters run has many plausible answers, and the review is where you find out which one the project can live with.

It doesn't fix everything, and the PR says so. A GC pass over many slow snapshots can still add up while it holds the lock, because each call finishes just under the limit. That needs the lock itself reworked, and it is the next piece.

Four posts, three PRs, one revert. The fix that finally lands is smaller than the one that got reverted, and it covers more. That is usually how it goes when the review is doing its job.

Lessons

A patch that is correct for your incident and a patch that is correct for everyone running the project are different things. The review is the distance between them.
Review doesn't stop at merge. Mine got reverted, and the revert was the right engineering.
Put the bound at the boundary. One deadline where containerd talks to the plugin covers every call; one deadline per call never ends.
The steady state lies. Look at the peak, not the numbers after the storm.
Maintainers do invisible work daily, for free, for strangers. Say so when they do it for you.

Credits and links

Michael Brown and the containerd maintainers, for 30 review rounds, a revert I agreed with, and a better fix
containerd/containerd#13799
containerd/containerd#14187
awslabs/soci-snapshotter#2043

Krew is K8s Game Changer!

Nahum Litvin — Thu, 11 Jul 2024 00:00:00 +0000

Kubernetes can be quite complex, but it doesn't have to slow you down. Here's how I simplified my K8s management with some awesome Krew plugins.

installation guide for Krew

list of all Krew plugins

alternate list of all plugins

Discovering Krew Plugins

When I first started using Kubernetes, I found it overwhelming to manage all the resources efficiently. That's when I stumbled upon Krew , a package manager for kubectl plugins, which transformed my workflow. Here are the plugins that made a huge difference for me:

foreach This plugin is a game-changer. Imagine being able to execute commands across multiple resources with just one line. It's perfect for bulk operations, saving me a ton of time when managing multiple pods or services.
get-all Getting a comprehensive view of all resources in a namespace used to be a hassle. Not like using kubectl get all With get-all, I can actually retrieve every resource type in one go.
ktop Monitoring cluster resources is crucial, and ktop makes it so much easier. It's like having a real-time dashboard for your cluster's health and performance. I check it regularly to ensure everything is running smoothly.
netshoot This one really changed my life. it allows me to easily debug any networking issues, attaching debug container to any pod with ease.
node-shell Sometimes, you just need direct access to the nodes. Node-shell allows me to get a shell on any node in the cluster. It's invaluable for troubleshooting and performing maintenance tasks directly on the node. and doesn't require ssh,ssm,ec2-connect or any of those. it is a very privileged container and might cause your security team to call you to ask WTF?
stern Debugging applications can be tedious, especially when you need to tail logs from multiple pods. Stern makes this process seamless by streaming logs from all relevant pods in real-time. It's a lifesaver when tracking down issues across pods.

Running kubectl commands on multiple clusters

Nahum Litvin — Mon, 01 Jul 2024 00:00:00 +0000

Managing multiple Kubernetes clusters can be a real headache. I know firsthand how frustrating it can be to juggle multiple contexts, keep deployments consistent, and maintain stability across different environments.

so I found this awesome repo: https://github.com/ahmetb/kubectl-foreach

Run a kubectl command in one or more contexts (clusters) in parallel (similar to GNU parallel/xargs).

Usage

kubectl foreach [OPTIONS] [PATTERN]... -- [KUBECTL_ARGS...]

and created a small helper function to use it. It's a simple way to run kubectl commands across multiple clusters in one go, making life a bit easier.

Here's the function you can add to your bashrc or zshrc:

function kfe() {
  eval "kubectl foreach -q dc1 dc2 dc3 -- $*"
}

Just use kfe followed by your kubectl command, for example:

kfe get pods

and it will execute across dc1, dc2, and dc3 without any hassle. No more repetitive tasks or constant context switching, just a smoother workflow.

I hope this helps lighten the load for anyone dealing with the complexities of Kubernetes. We're all in this together, and sharing little tools like this can make a big difference.

Fixing EKS overprivileged cluster settings

Nahum Litvin — Sun, 28 Jan 2024 00:00:00 +0000

Are you aware that your Amazon EKS cluster could be at risk due to a default permission setting? By default, the IAM role that creates an EKS cluster is granted the system:master RBAC role, providing broad administrative access. This could be a significant security concern.

In the past, rectifying this issue was a cumbersome process, often requiring the recreation of the entire cluster. But now, AWS has streamlined this with new cluster access management APIs. With a simple command:

aws eks delete-access-entry --cluster-name <CLUSTER_NAME> --principal-arn <IAM_PRINCIPAL_ARN>

you can adjust these permissions efficiently without the need to recreate the cluster.

This development is a game-changer in EKS cluster security management, aligning with the principle of least privilege access. It's crucial for EKS users to be aware of these changes and implement them to ensure their clusters are secure.

For more detailed steps and insights on this topic, make sure to read the full AWS blog post here.