A sandbox can contain a wrong answer.
That sounds obvious, but a lot of agent infrastructure still treats containment as the finish line. Put the agent in a disposable container. Limit its credentials. Tear the environment down afterward. Now the run is "safe."
Safer, yes. Correct, no.
Isolation can tell you that the agent did not overwrite your laptop or leak one task's filesystem into another. It cannot tell you that the checkout was reproducible, the tests measured the intended behavior, the page rendered correctly, or the final diff stayed inside the requested scope.
For production work, a coding agent needs a proof environment: an isolated runtime that starts from known state, checks the behavior you care about, and returns evidence another person can inspect.
Without that last part, you have a contained claim of success.
What a sandbox actually solves
Sandboxes are useful because agent runs are messy.
A realistic task may install dependencies, launch services, mutate a database, open a browser, create files, occupy ports, and leave background processes behind. Two agents sharing that state can interfere with each other in ways that are hard to reproduce. A failed experiment can poison the next attempt.
Per-task or per-user environments reduce that mess. Projects such as Agent-Sandbox expose isolated code, shell, browser, and computer sessions with explicit lifecycle management. Discussions around forkable coding environments push the same idea further: duplicate the full state before trying destructive tests or competing implementations.
That is a real improvement over handing every agent the same long-lived development machine.
But isolation answers a narrow question:
Where did this work run, and what was it allowed to affect?
Correctness is a different question:
What behavior changed, and what evidence supports the claim that the change is right?
A container boundary cannot answer that for you. Neither can an exit code.
Start with reproducible state
Before an agent can prove anything, the environment needs an identity.
"It passed in the sandbox" is weak if nobody knows what the sandbox contained. Record enough state to recreate the run:
- repository commit and dirty-tree status
- dependency lockfile and base image
- installed toolchain versions
- fixtures, seed data, and service versions
- relevant environment inputs, with secrets redacted
- viewport, device profile, and browser state
- the exact task prompt or acceptance criteria
Browser and application state deserve special attention. A git worktree can reproduce source files, but it does not capture a logged-in session, a populated database, an in-memory queue, or a half-finished multi-step flow. That is why environment-level forks are interesting: two approaches can begin from the same application state instead of merely the same commit.
If the starting points differ, a comparison between agent runs is mostly theater. You do not know whether the patch, the fixture, or yesterday's leftover process caused the result.
The environment identity should travel with the output. A reviewer should not have to reverse-engineer it from CI logs.
Green commands are not a behavioral oracle
Agents are very good at satisfying visible checks. That becomes a problem when the visible checks are incomplete.
Suppose the task is "keep the form open and show an inline error when payment fails." The agent changes the component, updates a mock, rewrites a test, and reports:
typecheck: passed
unit tests: passed
build: passed
The page may still redirect on failure. The test may now assert the new, wrong behavior. The mock may never exercise the failure branch. The agent did not lie; it optimized against a weak contract.
A proof environment needs a behavioral oracle: a task-specific definition of what should happen.
For an API change, that might be golden inputs and outputs plus checks around side effects. For a migration, it might be the before-and-after schema, representative data, and a rollback run. For a frontend task, it might include explicit interaction steps, visible text, accessibility state, responsive behavior, console errors, and relevant network requests.
The oracle does not need to be fancy. It does need to be independent of the implementation the agent just wrote.
This is the uncomfortable part: an agent should not be allowed to redefine success while implementing the task. If it changes the acceptance test, that change needs separate scrutiny. Otherwise "all tests pass" can mean "the agent moved the goalposts cleanly."
Golden tests have their own risk. They can preserve behavior that was already wrong. Treat them as a lock on known behavior, not a substitute for deciding what the task should accomplish.
A screenshot is evidence, not proof
Frontend work exposes the gap quickly.
A build can pass while the page is clipped on mobile. A component test can pass while focus disappears after a dialog opens. A screenshot can look fine while the console fills with errors or an API request returns the wrong payload.
Browser tooling now makes it practical to collect much richer artifacts. Agent-browser, for example, documents accessibility snapshots, screenshots, snapshot and pixel diffs, traces, console messages, page errors, network inspection, device emulation, and saved session state.
Those primitives are useful because they let the environment return structured evidence instead of one confident paragraph from the agent.
Still, collecting artifacts is not the same as judging them.
A screenshot proves that some pixels appeared. It does not prove that they match the expected design. A trace proves that events occurred. It does not prove that the right user journey completed. A clean console says little about a silent backend data error.
Evidence becomes meaningful when it is paired with an expected outcome:
Given: a signed-in user with an expired card
When: the user submits the checkout form
Then:
- the page remains on /checkout
- the inline error is visible
- focus moves to the error summary
- no order is created
- the payment failure is logged once
Now the screenshot, accessibility snapshot, network log, and backend record each answer a specific question. The proof is the relationship between the contract and the artifacts, not the artifact pile itself.
Return an evidence bundle, not a victory message
An agent's final message is a summary. It should never be the only record of what happened.
A useful evidence bundle includes:
- the environment identity and starting revision
- exact commands, exit codes, and failed checks
- the final source diff
- test results tied to named acceptance criteria
- screenshots or visual diffs for relevant UI states
- browser trace, console output, and page errors
- network or backend logs when the behavior crosses that boundary
- any changed tests, mocks, fixtures, or baselines
- the cleanup or retention decision for the environment
Keep failures in the bundle. A flaky end-to-end check that passed on retry is part of the result. So is a mobile viewport that was skipped because the service failed to start. Deleting inconvenient evidence turns the bundle back into marketing.
The bundle also needs to be reviewable without asking the same agent to interpret it. Store the diff, logs, traces, and images as artifacts. Link checks to the acceptance criteria. Make the environment revision visible.
This does not remove human review. It makes human review less dependent on trust.
The four-layer contract
The whole setup can be reduced to four layers.
1. Reproducible state
Identify the source, dependencies, data, services, browser session, and task contract. A rerun should begin close enough to the original state that its result means something.
2. Isolation
Give the task its own processes, filesystem, ports, and browser state. Scope credentials and external access separately; a sandbox does not automatically make outbound side effects safe.
3. Behavioral oracle
Define expected outcomes outside the implementation. Use golden inputs and outputs, explicit UI assertions, invariants, or domain checks that match the task.
4. Reviewable evidence
Return the commands, diff, checks, failures, screenshots, traces, and logs needed to evaluate the claim. Preserve enough context for somebody else to disagree.
Skip any layer and the result gets weaker.
Reproducible but unisolated runs contaminate each other. Isolated runs without an oracle execute safely but prove little. Checks without artifacts force reviewers to trust a summary. Artifacts without expected outcomes become a folder of screenshots nobody can interpret.
A checklist for your agent harness
Before accepting an agent-written change, ask:
- Can we identify and recreate the environment's starting state?
- Did the task run in an isolated filesystem, process, service, and browser context?
- Were credentials and external side effects scoped explicitly?
- Did the checks measure the requested behavior rather than generic repository health?
- Did the agent change any test, mock, fixture, snapshot, or baseline that defines success?
- Does the evidence include failed checks and skipped coverage?
- Can a reviewer inspect the diff and runtime artifacts without the agent's narration?
- Does the environment have an explicit cleanup or retention outcome?
You do not need a giant platform to start. Add an environment manifest to the run. Keep acceptance criteria outside the generated patch. Save browser traces for UI changes. Treat changed tests as first-class review items. Attach the evidence bundle to the pull request.
Then ask a better question.
Do not ask whether the agent ran safely.
Ask whether another person can verify what it proved.
Source notes
Top comments (0)