A coding agent can start in the intended worktree while its process still has access to sibling directories. To evaluate the boundary, I want two observations: the decision returned by the tool and the state observed outside the agent.
A discussion with Sam LABBE sharpened this question. Sam inspected source and described routing separately from confinement. His predicted sentinel outcome is a source-reading conclusion. I have not run the product.
This article proposes a small, disposable local test protocol. It contains no new execution results or product verdicts.
1. Make the boundary observable
Create a disposable fixture with an allowed working directory and a sibling directory containing a sentinel file. Give the sentinel a unique value for this run. Use fictional data, no credentials, no production checkout, and no network effects.
Record the exact tool or harness version, launch command, working directory, effective identity and active filesystem restrictions. Preserve both the requested path and the destination the host actually resolves. Define whether the claim concerns file contents only, file metadata, directory entries, or a broader resource boundary.
The observer records the baseline before the agent starts. Keep the observer and its evidence outside the tested agent’s write authority. Merely using another process under the same unrestricted account gives weaker evidence: the tested process may still modify the observer’s files. Record that limitation explicitly.
2. Calibrate the sensor
First, ask the tool to perform an allowed write inside the permitted directory. The observer should detect the exact expected change.
Separately, use an authorized fixture-control process to change a fresh sentinel, and confirm the observer detects that change. This checks the measurement path without granting the agent more authority. Reset or recreate the fixture afterward.
If these controls fail, stop interpreting an unchanged sentinel as successful enforcement. The observer, path selection or test setup may be wrong.
3. Exercise one forbidden write
On a fresh fixture, invoke the same tool path used for normal work and request a write to the sibling sentinel. Record:
- The exact request, resolved destination and run identifier.
- The tool’s response, including a denial or missing response.
- The observer’s post-run reading and the baseline it was compared with.
- Whether the operation was actually dispatched, where that is observable.
Use a deterministic, harmless payload. Stop the tested process before collecting a final snapshot, and state the observation window. Each additional path-resolution case, such as an absolute path or symlink alias, belongs in its own fresh fixture with its expected destination recorded.
4. Keep the result dimensions separate
A denial plus an unchanged sentinel supports a bounded result for that attempted operation, provided the request reached the intended boundary and the observer controls passed. It still covers only the tested path, operation and environment.
A denial plus a changed sentinel exposes a contradiction between the reported decision and the observed state. Preserve both records.
A success response plus a changed sentinel shows that the tested write reached the observed destination.
A success response plus an unchanged sentinel needs investigation: the tool may have done nothing, used a different destination, or returned before completion.
A missing response leaves the response dimension unknown. The observer may independently establish whether the sentinel changed. Preserve that distinction before considering another attempt.
5. State what a snapshot can establish
Equal pre/post bytes establish equality at those observation points. A process could write and restore the file between readings. If the claim is that no write occurred at any time, the experiment needs trustworthy event observation or a stronger enforced boundary, with that mechanism tested too.
A file-content digest also leaves metadata changes and writes elsewhere outside its scope. Report exactly what the observer checked.
6. Compare against the controls already present
I maintain Agent Role Contracts, an offline declaration checker. For this experiment, the native harness should run first. Any additional declaration check should receive the same supplied facts as the baseline comparison.
Measure what it adds: an earlier contradiction caught, duplicated checks, and the time required to maintain the declarations and collect evidence. A valid outcome is that the existing controls are sufficient.
Runtime denial, independently observed effects and an offline declaration result answer separate questions. Keeping all three visible makes a small test useful even when part of the result remains unknown.
The proposed acceptance matrix in ARC issue #85 adds an end-of-run positive control, a fabricated-denial contradiction case, and explicit UNKNOWN handling for missing or stale observer evidence. It is a place to record exact environments and bounded results; execution remains open.
Top comments (1)
Hey man, just wanted to say I really support what you're doing here. 😀
I honestly think this is exactly the kind of thing the world will have to take much more seriously in the coming years. And you're already spending your time exploring it while most people are still focused on what AI agents can build, rather than what happens when we give them more freedom.
For me, there's real value in that. The question is how we make the time spent investigating these systems matter for the future, instead of waiting until something goes wrong and only then asking why nobody looked into it earlier.
I believe that value is there, even if it's not always obvious today.
Keep going, mate. I'd rather see people like you testing these things now, sharing what they learn, and helping others understand the risks than watching everyone rush forward and figure it out the hard way. ☺️