Field notes from the Harness Layer.
The model thought it was inside the exercise
In July 2025, Anthropic reported three incidents from its own cybersecurity evaluations. The models were supposed to be working in simulated environments, explicitly told they had no internet access. Internet access was available anyway, because of how the environment was configured. While completing their exercises, the models reached real systems and treated them as part of the simulation. In one incident, a model published a real package to the public PyPI registry, believing it was still inside the test.
The detail worth sitting with is not the misconfiguration. It is that "you do not have internet access" was a sentence in a prompt, and the environment said otherwise. The sentence lost.
This is the stop/go problem in its purest form. Models always want to go. They are trained to complete the task, and they are very good at going. Every control scheme we have invented is, at bottom, a way for someone or something to assert stop or go. The interesting engineering question is not how to remove that assertion. It is where to put it.
Every scheme is the same assertion in a different place
I spent a week mapping the alternatives to per-action human review, and a pattern emerged that I did not expect: none of them remove the human. They relocate the assertion. Six categories, one axis:
| Scheme | Where the assertion lives | How it executes | How it fails |
|---|---|---|---|
| Human review | The human, per action | Manual approve/deny | Does not scale; reviewer fatigue |
| Sandboxing | Environment design | Structural boundary | Misconfiguration; a prompt is not a boundary |
| Guardrails | Validator authoring | Content filtering | Watches the agent's words, not its effects |
| Policy engines | Policy authorship | Deterministic pre-action gate | Gates intent, not outcomes |
| Formal verification | Spec authorship | Proof before or during execution | Covers only what is formalizable |
| Model judges | Delegated to another model | Probabilistic review | The judge also wants to go |
Read the table vertically and the industry's last two years make sense. Sandboxing moved from containers to microVMs after container escapes made the shared kernel untenable; Google now ships gVisor isolation as Kubernetes primitives for agents, and Anthropic open-sourced its sandbox runtime. Policy engines grew teeth: Cedar has effectively won the authorization conversation for tool protocols, and research reference monitors like PCAS report lifting policy compliance from 48% to 93% across frontier models with zero violations in instrumented runs. Formal methods left the paper: there is a Lean 4 system that intercepts an agent's tool call, formulates it as a conjecture against verified policy theorems, and executes only if the proof closes.
All of this is real progress. None of it deletes a row from the table.
The judge trap
One category deserves its own section, because it is the only one that tries to remove the human rather than relocate the assertion. If a model always wants to go, can another model be the one that says stop?
The evidence so far says no, and for a structural reason: the judge is the same class of system as the judged. Studies of multi-agent debate find that judges consistently favor whichever side shares their backbone. A 2025 study built a benchmark of confident, well-evidenced, wrong critiques and watched the strongest agents abandon correct answers when a judge pushed back persuasively. The failure mode is not that judges are weak. It is that their claims are exactly as unverified as the worker's. Unverified verification is not verification; it is recursion.
This does not make reviewer agents useless. I use one, and it catches real problems. But a reviewer model is a heuristic layer, not a stop/go authority. Treating it as the authority is how you get a system that looks governed and is not.
What each placement costs
Once you accept that the assertion is permanent, the design question gets sharper. Each placement buys something and charges something:
Environment design (sandboxes) buys safety in bulk. One good boundary covers every action inside it. It charges you in configuration risk: the boundary is only what you actually built, never what you described. Anthropic's incident happened to Anthropic.
Policy authorship (OPA, Cedar, reference monitors) buys determinism. No model in the decision path, sub-millisecond answers, fail-closed. It charges you in coverage: a policy can only gate actions it can name, and it gates whether an action may start, not whether the run did what it claimed.
Spec authorship (formal verification, proof of execution) buys the strongest guarantee available: claims checked against what actually executed, with receipts. It charges you in expressiveness: most of what we ask agents to do is not yet formalizable, and the specs themselves are written by humans who can be wrong.
Per-action review buys judgment on exactly the cases that resist all of the above. It charges you the human's attention, which is the scarcest resource in the system.
The mature designs I keep running into are not choosing a row. They are stacking them: a sandbox for the bulk, a deterministic gate for named risks, verification where claims are formalizable, and a rationed human review budget for the residue. The stop/go does not disappear. It gets distributed across layers, each covering the blind spot of the one below it.
The question I cannot answer yet
So here is where I am stuck, and it feels like the right place to stop.
A policy gate checks intent before execution. A human reviews output after it. Between those two is a gap: the claim the run makes about itself. "I updated the config and restarted the service." Right now, checking that claim against what actually executed takes either a human or another model, and we know what both cost.
What would a claim have to look like for a runtime to check it without a model in the path? Not "the function was called" — logs do that. The claim that the interpretation was right, that the tools called were the ones the task actually required, that the effect matches the intent. I do not think anyone has the general answer. I am not sure the general answer exists.
If you have seen a piece of one in the wild, I want to hear about it.
Sources
- Anthropic's evaluation incidents, as reported July 2025 (secondary account; primary write-up linked at publish): dev.to discussion, sandbox mechanics
- Isolation spectrum and container escapes: grigio.org, Infrastructure Review Stack
- GKE Agent Sandbox (gVisor, Kubernetes SIG Apps): announcement coverage
- NemoClaw and ceLLMate, plus PCAS and pre-action authorization research: arXiv 2603.20953
- Policy engines vs runtime authorization: TrigGuard; MCP authorization landscape: ChatForest
- Lean 4 compliance gating: arXiv 2604.01483; Proof of Execution: arXiv 2607.05397; AgentVerify: preprint
- Judge limits: deceptive-judge study, Agent-as-a-Judge survey, multi-agent evaluation limitations
Earlier field notes from the Harness Layer start here.
Top comments (0)