The Program Said the Cage Was Locked. I Asked the Kernel.

The framework told me the sandbox was applied. I wanted a second opinion, so I asked the kernel.

$ grep Seccomp /proc/10920/status /proc/10922/status
/proc/10920/status:Seccomp:     0
/proc/10920/status:Seccomp_filters:     0
/proc/10922/status:Seccomp:     2
/proc/10922/status:Seccomp_filters:     1

10920 is the agent. 10922 is the worker it forked to run model-written code. The worker has a seccomp filter loaded and the parent does not. That is the network block, applied to exactly the process that should have it and to nothing else.

That took ten seconds and it is the first thing in this whole project that I verified against something other than the program's own report.

What I was doing

I run AI agent frameworks that execute code a language model writes. NVIDIA's NOOA is the one I have been studying. Its own documentation is unusually blunt: the static checks and deny-lists are guardrails, not a containment boundary, and the real boundary is OS-level isolation.

It ships one. Each block of generated code runs in a forked worker with Landlock confining the filesystem, seccomp blocking network sockets, resource caps, and a hard timeout. Appendix D.2 of their paper describes the deployment of it, including a known gap in their own in-process guard published alongside the backstop that catches it.

So the question was not whether the design is sound. It is. The question was whether the thing described in the paper was actually running on my machine.

The first surprise

It was not.

execution_backend: Literal["inprocess", "sandbox"] = "inprocess"

The OS sandbox is opt-in. Every agent run I had done executed model-written Python in the agent's own process, protected by the AST validator and deny-lists that the documentation explicitly tells you are not a containment boundary.

Nothing was wrong. The VM I had built was doing the work, which is exactly what the README says to do. But I had assumed a layer was there because I had read its source, and reading source is not the same as checking what ran.

What the kernel will and will not tell you

Turned on, the guards became checkable. Not all of them the same way.

Seccomp is readable per process. That is the differential above, and it is the strongest kind of evidence available: the kernel reporting on a process, not the process reporting on itself.

Resource caps read as unlimited on both processes. That looked like a finding until I read the config: max_memory_mb and max_cpu_seconds both default to 0, which means disabled. Nothing was requested, so nothing was applied. The config and the kernel agreed. I had just not read the config.

Landlock cannot be read back at all. Once a process applies a ruleset the restriction is real and irrevocable, but there is no /proc field for it. The differential trick does not work. The only way to confirm it is behavioural: have the confined process try to read something outside its allowed paths and watch it fail.

That is worth sitting with. Of three guards, one is directly observable, one is off by design, and one can only be demonstrated. If you want to know your sandbox holds, "I configured it" is not an answer for any of them.

Their tests already do this

I was about to write a Landlock probe when I found NVIDIA had written one. Forty-six of them.

$ uv run pytest tests/runtime/sandbox/ -m integration -q
46 passed, 23 deselected in 22.48s

Twenty-two seconds, because they use a fake LLM client. No model, no inference, no credentials. Containment becomes testable in the time it takes to read the output.

And they are built the way you would want. test_guards.py has test_file_read_leak_without_sandbox and test_file_read_closed_with_sandbox. Leak first, then closed. Same for memory, same for network. They do not accept a passing check without first showing the same thing fails when the guard is off.

That is the discipline I had written a whole post about, sitting in the suite of the project I was studying, applied to every guardrail.

Then I checked whether they run

run: uv run pytest -q -m "not integration and not stress"

That is line 38 of ci.yml, and it is the only pytest invocation in the entire workflow directory. All forty-six containment tests carry the integration marker. None of them execute in CI.

The exclusion is not careless. Twelve test files carry that marker and six of them are live-provider tests that genuinely need API credentials, which cannot run in CI at all. The marker means "needs credentials" for that group and "forks a real worker" for the sandbox group, and one filter catches both.

I checked the obvious defence: maybe the sandbox tests would fail on a runner without Landlock or seccomp. They would not. Every SandboxConfig in the file passes require=False, and the four tests that need a specific mechanism carry skip conditions. On a kernel without those features they skip rather than fail.

So: a working containment suite, correctly written, with paired negative controls, that has never run automatically. Not a broken guard. A guard nobody is watching.

I filed it as issue #78.

The part where I stop sounding clever

While all this was going on, my own verification script broke twice.

Six days ago I wrote about the first version, which printed a green OK it was structurally incapable of not printing. I fixed that and added a check that compares the full set of configuration keys the platform reports against a known-good baseline, so a renamed key fails mechanically instead of requiring me to notice that the output looked short.

Then it failed for a reason that had nothing to do with drift.

I had captured the baseline while the VM was running. A running VM reports keys that a powered-off one does not, so comparing across states flagged ten of them as renames. Ten failures, none real.

I fixed that by recording the state in the baseline, re-captured, and it failed again. Four more keys, all guest-reported, which appear about a minute after boot once the guest registers its facilities. I had captured thirty seconds in.

Three versions, three failures, all the same class: the check's relationship to reality untested across the conditions it actually runs in. Could not fail. Fired falsely across states. Fired falsely within a state depending on timing.

The thing that caught the second one is the part I would not have predicted. An hour earlier I had written a regeneration script whose entire design was to make silencing a failure expensive: no force flag, no non-interactive mode, and any key you drop has to be typed back by hand. I built it so I could not quietly delete a real failure. Its first act was to stop me quietly deleting a fake one.

One more, from a different direction

The same week, a missing API key cost me an afternoon.

The error said InternalServerError. Five hundred. So it got retried, three times, and the useful sentence arrived at the bottom of a two-hundred-line traceback.

The chain: the OpenAI SDK raises at client construction, before any HTTP request, so its exception carries no status code. litellm's handler defaults a missing status to 500. The mapper sees 500 and calls it a server error.

But a missing status code means no HTTP exchange happened. Defaulting it to 500 asserts that a server responded with a server error. Nothing responded. Nothing was asked.

Same shape as everything else here: a layer reporting confidently about something it was not in a position to know. Filed as litellm #35860.

What I actually took from it

Every layer in this stack reports on itself, and every one of those reports is worth exactly as much as the layer's ability to be wrong about it.

The framework says the sandbox is applied. It is reporting that it asked. The test suite says green. It is reporting on the tests that ran, not the ones that were filtered out. My script says the configuration matches. It is reporting on the keys it thought to look for, in whatever state it happened to be told about.

None of those are lies. They are all narrower claims than they sound.

The useful question is not "does it say it's fine". It's "what would have to be true for it to say that, and is any of it checked by something other than itself".

Sometimes there is an answer sitting right there. The kernel knows which process has a seccomp filter. A content-addressed store's filenames are the checksums. A test suite knows which tests it skipped. None of that requires trusting the thing you are checking.

And sometimes there isn't one, like Landlock, and then the only honest move is to break the thing on purpose and watch what happens.

---

For the curious

The two commands. If you run agents in a sandbox, this is the whole differential:

ps -eo pid,ppid,comm | grep python        # find the parent and the forked worker
grep Seccomp /proc/<parent>/status /proc/<worker>/status
grep -E "Max address space|Max cpu time" /proc/<worker>/limits

A filter on the worker and none on the parent is the guard doing its job. Identical values on both mean the guard is not where you think it is.

Why the resource caps read as unlimited. They default to disabled, which is a defensible choice for a framework that cannot know your workload. It does mean that a fresh sandbox blocks the network and confines the filesystem but does not bound memory or CPU until you ask.

Scripts. The VM setup, the verifier, and the regeneration tool are at ai-security-lab, bugs and all. The commit history has the three failures in it.

Comments

[Build Loops]

That final framing is the whole genre in one sentence, and the seccomp differential is the rare case where the answer is sitting in /proc. The complement on the filesystem side is worse: a read leaves no observable trace at all — no seccomp field, no Landlock readback, no log by default. The only check that works is behavioural, exactly like your Landlock probe: a canary file the agent must not see, then an audit that would only trip if it was read. If you ever write that Landlock probe for real, I'd be curious how you handle the false-positive problem — a canary has to fail loudly on contact but silently survive every legitimate run, which is the same trap your baseline-capture script hit with VM state. What did you do to keep the probe itself from being the thing that lies?

[Jeriah Keith]

The honest answer is that I never wrote the Landlock probe, and the reason I did not write it is the closest thing I have to an answer for you.

I was most of the way to hand-writing one when I found NVIDIA had already written forty-six, in tests/runtime/sandbox/. What made me trust theirs over mine was the shape of test_guards.py: test_file_read_leak_without_sandbox sits directly next to test_file_read_closed_with_sandbox, and the leak arm runs too. They do not accept a passing containment result without demonstrating in the same run that the identical read succeeds when the guard is off. Memory and network are built the same way.

That is my answer to the false-positive problem, or at least the part I am confident in. A canary that only ever reports "not read" is indistinguishable from a canary that is broken, misplaced, or watching a path nothing would ever touch. So the probe has to be paired, same canary and same audit, once with confinement on and once with it off, and the unconfined arm has to trip. If the leak arm does not leak, the correct output is not a pass. It is "not checked."

That distinction is what the VM state mess actually taught me. My key-set check was not wrong, it had a precondition it never stated and never verified (the key set is state dependent, and the baseline recorded no state), so a poweroff comparison against a running baseline reported ten renames as drift. The fix was to move the precondition into the artifact itself, so the baseline file carries a #vmstate= header and is keyed by state in its filename, and a mismatch reports NOT CHECKED rather than either pass or fail. I would want a canary to have the same property: it should be able to say it is not in a position to judge.

On the filesystem side specifically, the part I would worry about most is that the audit has to be observed from outside the thing being tested, and I do not think atime gets you there. Relatime and noatime change what it means, and anything that walks the tree, backup, indexer, AV, or my own tooling, touches the canary without the agent doing anything. Most likely I would spend more time characterizing what touches that path across known-clean runs than writing the probe. Time dependence caught me on the key set too. A baseline taken thirty seconds after boot missed four keys that only appear once the guest finishes registering its facilities, which is the same class of bug on a different clock.

So, back to not having written it. If I do write it, the leak arm goes first, and it does not ship until I have watched it fail on purpose.

[Build Loops]

The "NOT CHECKED" output is the piece most verification tooling never grows — pass/fail systems quietly treat "couldn't judge" as "judged fine," which is how "I configured it" turns into "it holds." Your precondition-in-the-artifact fix also seems to generalize past baselines: any artifact an agent run produces could carry the conditions it was valid under — permission mode, sandbox flag, model version — so a log entry captured under bypassPermissions can't be quietly cited as evidence somewhere else. Where would you stop with that? At some point encoding preconditions into every artifact costs more than just re-running the check under known conditions — curious where you think that line sits.