My script printed a green [OK]. The setting it was checking was correct. Both of those things were true, and neither of them mattered, because the check that printed [OK] was incapable of printing anything else.
I want to walk through how that happened, because I wrote it, and because I think it is the most common way security tools fail.
What I was building
I am a cybersecurity student, and I was setting up a lab to study an AI agent framework that NVIDIA released this month. The framework can execute code written by a language model. Their own documentation is blunt about what that means: generated code might delete files, send private data somewhere it shouldn't go, or modify the environment it runs in. They tell you to run it inside a virtual machine, isolated from your real files.
So I built one. A virtual machine with no shared folders, no clipboard sharing between the VM and my computer, no drag and drop. Every channel between the sandbox and my actual hard drive, switched off.
Then I did the part I was proud of. Instead of trusting that I had ticked all the right boxes, I wrote a script that read the machine's settings back and confirmed each one. Set the control, then verify the control took effect. Those are two different things, and skipping the second is how people end up protected on paper only.
That was the right instinct. My implementation of it was broken.
The bug
The check asked whether the clipboard was disabled. It used the wrong name for the setting, so it got no answer back. And it treated no answer as a yes.
That is the whole thing. Nothing more sophisticated than that.
The tool I was querying reports the clipboard setting under one name. The command I used to configure it in the first place used a different name for the same thing. I assumed they matched. They didn't. So my check searched for a setting that, as far as it could tell, did not exist, found nothing, and concluded everything was fine.
If I had left the clipboard wide open, the check would have printed exactly the same green [OK].
How I found it
Not by testing the security check. I want to be honest about that, because the real answer is more instructive than a clean story would be.
The same script printed a summary of the machine's configuration at the end, purely for me to read. I noticed the summary was short. A few settings I expected to see weren't listed. It looked like a formatting glitch, the kind of thing you would normally shrug at.
I chased it anyway, and it turned out those settings were missing from the summary for the same reason the security check was broken: I had the names wrong. The cosmetic bug and the security bug had one shared cause. The harmless one was visible. The dangerous one was invisible by design, because a check that cannot fail looks exactly like a check that passed.
I got lucky. I would rather say I found it through rigor, but I found it because something unrelated looked slightly off and I didn't ignore it.
The fix, and the part that mattered more
Fixing the names was easy. Fixing the shape of the logic was the real work. Now, if the script can't find the setting it's looking for, it says so and fails. "I could not verify this" and "this is fine" are no longer the same outcome.
Then I did something I had not done the first time. I deliberately broke the isolation. I connected a shared folder pointing straight at the drive with all my work on it, and I turned the clipboard back on. Then I ran the check.
It failed. It named both problems and refused to give the all clear.
That took about ninety seconds and it is the only reason I believe the check works. Watching a security control report success proves nothing about the control. It only proves the control can produce that output. You learn whether it works by breaking the thing it's supposed to be watching and confirming that it notices.
The part I did not expect
A few hours later I was reading the source code of the framework I had built all this to study. Deep in the sandbox module, I found a specific error they raise when the computer can't enforce one of their security guarantees.
The comment next to it says failing closed there is deliberate, because the alternative is running untrusted code with a guard silently missing.
That is the same principle. Same reasoning, opposite direction. My check could not verify something and resolved that to success. Their sandbox cannot enforce something and resolves that to refusing to start at all.
I had written my fix hours before I read their code. Finding a team at NVIDIA arriving at the same conclusion, and treating it as important enough to explain in a comment, was the moment it stopped feeling like a personal lesson and started feeling like a rule.
What I actually took from it
A control that cannot fail is not a control. It is a message that says what you want to hear, and it will keep saying it long after the thing it was watching has stopped working.
Systems are full of these. A monitoring rule watching a field that got renamed. A scanner pointed at a folder that moved. A test that stopped running months ago and still shows green. None of them announce themselves. They all look precisely like everything is fine, which is the point, and which is why the ordinary failure mode of a broken safety check is silence.
"I don't know" and "you're safe" are different states.
Any system that collapses them into one output will eventually tell you that you're safe when you are not.
I know that now because I built one that did.
The scripts are on GitHub: ai-security-lab
Top comments (4)
The remaining gap is set coverage. Your fix makes the checker honest about settings it asks for, but the list of things worth asking about was authored from the same mental model as the setup that closed them. The name mismatch was findable because a second surface disagreed with that model loudly enough to create a weirdly short summary. A channel that never enters the list creates no disagreement anywhere.
Negative testing has the same boundary. Deliberately reopening clipboard or shared folders proves the checker can detect those enumerated controls. It certifies nothing about channels you did not know to break. The vendor warning you quoted includes generated code sending private data somewhere it shouldn't, while the controls listed here are host-filesystem channels: clipboard, shared folders, drag-and-drop. Those are paths to the hard drive; egress sits elsewhere. Unless you detached it or set host-only, which the article doesn't mention, most desktop hypervisors ship a NAT adapter attached and outbound-open. Guest code can make an HTTPS request. A fail-closed, negative-tested check can still go green while that line item never existed.
The artifact that saved you should probably become the check. The end summary was doing more work than the assertions, only you were reading it by eye.
Instead of only asking "is X disabled?", dump the platform's full property list and assert expected keys against the key set the platform actually returns. A rename then fails mechanically: expected key absent from a present, populated list. No need to notice that the output "looked short." Also diff the full key set against the last known-good run, including keys you didn't set. That catches renames, removed keys, added capabilities after upgrades, and previously unenumerated channels in a way individual assertions can't.
You're right, and it's a sharper version of my own point than I made.
My fix makes the checker honest about what it asks. It says nothing about what it never thought to ask. Negative testing inherits that boundary exactly as you describe: reopening the clipboard proves I detect the clipboard. A channel that was never a line item produces no disagreement anywhere, so there's nothing for me to notice.
One thing I left out of the post that I should have included. The verifier does report network posture, printing
[WARN] Network is LIVE (nic1=nat, cable=on)with a note to cut the cable before running untrusted code. But it's explicitly informational and doesn't affect the exit code, which means it isn't a control. It can go green with egress wide open, exactly as you say. Omitting it from the article made the tool look narrower than it is and the gap smaller than it is.Your closing line is the part I keep coming back to. The summary was doing more work than the assertions and I was reading it by eye. Turning that into the mechanism is obviously the right next version: dump the full property list, assert expected keys against the set the platform actually returns, and diff the whole set against a known-good baseline. A rename then fails mechanically instead of requiring me to notice the output looked short, and it catches keys that appear after an upgrade, which I couldn't have enumerated in advance.
For egress specifically I think the answer is a mode. Default warns, since you need network to install anything. A detonation mode asserts the adapter is disconnected or internal and fails otherwise, so the thing I've been doing by hand becomes something the tool enforces.
Thanks for this. Going to build it.
Built it. Both pieces.
Key-set diff. The script now dumps the full property key set and compares it against a committed baseline. A key in the baseline that the platform no longer reports is a failure, since that is exactly the case where a value assertion is searching for something that is gone. A key the platform reports that is not in the baseline is a warning. Negative-tested by editing the baseline to expect
clipboard-modewhile the platform reportsclipboard: FAIL on the missing key, WARN on the new one, exit 1. That is the original bug caught mechanically, with nobody needing to notice the output looked short.Egress. Now an assertion behind a
-Detonateswitch instead of a warning. Passes if the adapter isnone/intnet/hostonly, or if the cable is disconnected. Fails otherwise. Off by default because you need network to install packages, and a check that always fails is one people learn to ignore. Negative-tested by reconnecting the cable: FAIL, exit 1.A bug fell out of building it. The script was printing "before untrusted code, run
VBoxManage modifyvm ... --cable-connected1 off" two sections above where it printedVMState: running.modifyvmdoes not work on a running VM. It observed the state and then gave advice that fails when you follow it. State-aware now.Where it still doesn't reach your point. Existence coverage is 158 of 158 keys. Value coverage is 9. A new key surfaces as a warning, so an added capability does show up, but it gates nothing and I still have to decide what value is safe for it. And regenerating the baseline silences all of it. The README says not to do that to quiet a failure you have not read, which is discipline, not a mechanism.
Code: github.com/Yeriahz/ai-security-lab
Thanks for this. v1 would have kept passing.
Nice work, and the detail that stands out is the state-aware advice fix. A verifier's advice is also a claim about the world, specifically that following it will work. Catching the running-state modifyvm path means you held the output channel to the same standard as the checks.
The baseline-regeneration residual has a mechanical answer too. Make regeneration require a reference to the failure it is silencing: if the last run failed, the tool refuses to write a new baseline unless the command includes that run's hash or ID. Record the acknowledgment beside the new baseline, with the old one and the failing run it overrode, committed together.
That changes the failure mode. Quieting a failure you have not read becomes structurally difficult, because the silencing action itself creates a reviewable artifact. Regeneration turns into an audited state transition instead of an escape hatch hidden behind README discipline.
On value coverage, I would not chase 158 hand-written assertions. You already built the useful primitive, which is baseline comparison. Extend it to the values as well as the key names. The tool does not need to know whether a value is safe, only that it changed since the known-good capture, so an upgrade flipping a default from off to on becomes visible before anyone has enumerated that key as dangerous. Explicit value assertions stay reserved for the few keys that gate danger-bearing channels, where the policy is known and worth pinning.
The pattern across the whole arc: each fix moved a judgment from "someone notices" to "the artifact disagrees."