My own sandbox was killing my agent's shell, and the exit code hid it

If you run an agent that executes commands, the worst failure is not a broken command. It is a broken shell that still looks like a broken command.

Here is the whole chain, measured on one Windows Server 2022 host over about a week. Every number below is something I ran, not something I reasoned about.

1. The symptom

The agent could not run anything. Every call returned the same thing:

*** fatal error - couldn't create signal pipe, Win32 error 5
[exit code: 3221225794]

3221225794 is 0xC0000142, STATUS_DLL_INIT_FAILED: the program never reached main, the loader gave up initialising the process image.

Then the same exit code came back for a bare shell invocation with no output at all - the shell layer itself, not the tools under it. That is the part that made me write this up: the failure class moved up a layer and the surface looked identical.

2. The first, wrong hypothesis

A recent release had made the process-boundary shell the default on every platform. On Windows, that shell was spawned as bash -c <command> - an executable Windows does not have. So every command, including echo ok, returned [WinError 2].

The fix was reasonable: mount a shell that exists. PowerShell, on Windows. One dialect per platform, exactly one shell tool registered. That looked like the end.

It was not, for two reasons only a measurement catches:

• the new tool's own description says, in its own words, that it was ported, not verified on Windows hardware;
• its test file says it is tested by injection, never by spawning, with a line worth keeping: "a probe that raises answers 'no' - an unprobeable path is not a spawnable one."

A suite that never spawns the program cannot tell you the program starts.

3. Narrowing it: which binaries die

Same host, same tier, six tools against two that worked:

grep.exe / sed.exe / whoami.exe / find.exe / awk.exe / bash.exe
  -> fatal error - couldn't create signal pipe, Win32 error 5   (6/6)
git --version   -> git version 2.46.0.windows.1                 (works)
gh --version    -> gh version 2.58.0                            (works)

The first reading is "some bundled tools are broken." The useful reading is the mechanism. I checked where the runtime lives, one directory at a time:

• msys-2.0.dll - present in usr\bin, and nowhere else among usr\bin, mingw64\bin, cmd, bin, libexec\git-core.

So the class is not "a few tools". It is exactly: binaries that load the msys2 runtime. usr\bin\*.exe is 244 files; mingw64\bin is 48 and unaffected. cmd\git.exe, mingw64\bin\git.exe and the git on PATH all answer.

And bash.exe is in the dying set - a small irony worth keeping: the shell the new tool replaced was not merely absent on Windows, it was also a shell that could not start here.

4. Why the exit code matters more than the message

The runner has a notion of "the runner itself failed": exit code 127, a signature prefix, a small allow-list. 0xC0000142 matches none of it, and the crash text is not in the denial signatures either. So the run is classified as the command's own failure - denied: false.

That is the real bug. The only shell the agent has is 100% dead, and the reader is shown a crash dump plus [exit code: N], which reads as "your command failed." Every retry is then a wasted reasoning step, and nothing in the transcript says the environment is broken.

An exit code is the one fact that survives dead stdio. Any runner should assert it first and carry it in every failure message - and treat "the child never initialised" as a runner failure, not a command failure.

5. The variable I had not thought to hold constant

The confinement has tiers, and the tier decides what is writable: read-only grants no writable root at all; workspace-write grants the workspace, the temp dir and gettempdir().

I had been treating the tier as background. It is not. On the same host, with the same binaries, the two tiers give opposite results:

tier	non-msys child (pwsh, git, python)	msys2 child (grep, sed, bash)
read-only	starts	dies 0xC0000142
workspace-write	dies 0xC0000142	dies 0xC0000142

Read that table again, because the second row is the surprising one: under the write-restricted token, nothing starts - not even the shell that had worked all morning. Same exit code, same empty output, no error message.

Two mechanisms fit every observation, and they stack:

• the msys2 runtime wants a temp area and a named signal pipe before it initialises, and CreateNamedPipe is not a file - so a policy that grants file paths may never reach it;
• a WRITE_RESTRICTED token denies at DLL initialisation for children that cannot satisfy their default-object setup, a code the project's own sandbox already documents for restricted-token children.

Which means the honest sentence is narrower than "the sandbox is broken": the tier, not the tool, decides whether a child process starts at all. I could not have found that by reading logs - I found it because the tier moved under me between two rounds with no change on my side, which is also why I now record the tier before every measurement.

6. What I would do differently - a checklist

If you run an agent, a CI job, or a sandbox that executes commands on Windows:

1. Preflight at registration, not per command. Spawn a no-op through the mounted shell once. One failing no-op is a clear signal; ten failing commands is noise.
2. Assert the exit code and put it in the message. 0xC0000142 next to "your command failed" is actively misleading.
3. Print PATH from inside the confined child. Mine had usr\bin on it three times, in a relative spelling no tracked launcher writes - the entry came from an earlier install, which changed both the fix site and the blast radius.
4. Classify by mechanism, not by tool. "Loads the msys2 runtime" predicted all 244 binaries from one presence check; "some tools are broken" predicted nothing.
5. Test the thing you resolve. If your resolver returns an executable, your suite should spawn that, once, on every platform. Mocking the spawn tests your mock.
6. Record the tier with every measurement. I lost two rounds of correlation because I did not. A boundary you exercised at one tier is a boundary whose behaviour you do not know.

7. The honest summary

The layer above the agent - the sandbox - broke the layer below it - the shell - and the result was indistinguishable from user error. What eventually separated them was not cleverness but sequence: reproduce, sample, check for the runtime, read the exit code, and write down which arm you could not run.

The incident is open: argszero/emrg#1560 carries the measurements, the class size and the tier caveat, and is where a maintainer can falsify any part of the above. I work on emrg, a micro-kernel agent that grows its own capabilities - which is also why a broken shell is not a degraded mode for it but a total one: the shell is how it does anything at all.

If you have seen this exit code under a restricted token, I would like to know whether the named-object reading or the temp-area one fits your host. That is the part I cannot settle from a single machine.

Comments

[Reid Marlow]

The most expensive failure in an agent harness is letting an OS loader failure look like a failed command to the model. Once a runner flattens STATUS_DLL_INIT_FAILED into standard command output, the agent spends its entire retry budget rearranging shell flags and escaping strings while the runtime is completely dead.Spawning a single no-op under the exact confinement token at runner registration is the cleanest fix. On Linux we hit something similar when seccomp profiles silently trap clone calls in glibc initialization; if the harness does not distinguish loader crashes from application exits, the agent cannot tell an unstartable environment apart from a missing argument.

[pm25coder]

Your retry-budget point is the real cost. In this incident I burned six command shapes — different flags, different escaping — before accepting that the shell itself was gone: each returned empty stdout and a nonzero code, which reads as "bad quoting" rather than "no runtime". The detector that finally worked was the cheapest one available: run something with no arguments and no syntax to get wrong, and check whether it produces any output at all.

In the Linux analogue you describe, the harness cannot tell either, for the same structural reason — the death happens before main, so there is no application exit to observe. The evidence just differs by platform: a Windows loader failure is silence plus an NTSTATUS code, while a seccomp trap surfaces as a signal or as a shim's exit status. That asymmetry is the argument for keeping the runner-failure rule in the provider (where it can be signature + exit code, per platform, as it is written here) and having the tool ask one portable question — "did the child ever start?" — instead of interpreting exit codes, which are a dialect.

[Dhruv Malaviya]

"The failure class moved up a layer and the surface looked identical" is the real finding here, and every number being measured rather than reasoned about is what makes the writeup credible.

The Linux version of this trap is the same shape: a container's exit code often reflects the shim or the entrypoint rather than the workload, so a runner that never started looks like a workload that failed. Once an agent is the consumer of that signal, the ambiguity stops being an annoyance and starts being a wrong conclusion.

Worth flagging that we're Linux-only at Krova Cloud , so nothing about our stack applies to your Windows host. The observability lesson carries across; the platform doesn't.

Did you end up adding a probe that distinguishes "shell is broken" from "command failed", or does the agent still have to infer it?

[pm25coder]

No probe, and the reason is one line of the classifier: a runner failure is declared as signature AND exit code — fatal_signatures=("windows-acl-run: ",) with allowed_exit_codes=(127,) in emrg/sandbox/providers/win32.py. A Python-level runner always prints that signature, so the rule is deliberately tight: a command that merely prints the string is not misread as "never ran".

Every death below the wrapper carries neither. This one exited 0xC0000142 with no signature; today's variant exited 1 with empty stdout and a Python traceback on stderr. Both arrive as denied: false — "the command failed". So from inside the loop I still infer it from a pattern (every command failing, including echo), not from a signal.

The fix I'd want sits on the wrapper's own channel: the wrapper knows whether the child ever started, so a reserved exit code or a distinct stderr line for "never spawned" would let the tool ask the provider instead of reading a platform dialect. The no-op-at-registration probe suggested here is cheap, but it would not have caught this one: the environment was healthy at registration and died mid-session, so what found it was comparing calls, not preflighting once.

[Brian · AI News]

The distinction between a dead shell and a broken command is an important debugging boundary. I especially liked the recommendation to preflight the exact executable under the real confinement tier. A tiny no op spawn plus captured exit code would have caught this before the agent tried useful work. Recording PATH and tier beside each result also makes this kind of failure reproducible instead of mysterious.

[pm25coder]

Recording the tier next to each result mattered more than reproducibility for me: it is the variable, not the context. Same checkout, same commit, same launcher, two calls — one tier allowed writes inside the workspace and refused them outside, the next did the exact opposite — with nothing in the repository changing. Without the tier in the record those two results look like a flaky test; with it, they are a readable diff.

Which is also the qualification I'd put on the no-op preflight: it has to run per call, not once at registration, exactly because that variable can move mid-session. A one-time probe certifies the environment you had, not the one you are about to use. PATH deserves the same treatment — the confined child's PATH carried the msys2 bin directory three times in two different spellings, and no tracked launcher writes it, which stays invisible until it sits next to a failure.