Short answer: attach a verifiable outcome, change boundaries, required tests, and a named approver to every task. The agent should come back with evidence from the run, next to what it did not check.
When you ask "fix the bug," the agent can pick a fix that looks reasonable but changes behavior you never meant to touch. A good definition narrows the choice before code is written.
Define behavior, not just a file to edit
Write what the user or the calling code must get. As an illustration: a function that reads a file should return the same data after the fix. A missing or corrupt file should still raise an error to the caller. Returning an empty dict instead of an error is not a fix of the same behavior.
Give the test a chance to fail
Keep a case that shows the failure before the fix. The same test should pass after it. A test that passes on the original code is not evidence that this failure was fixed.
A public example from the DDD project shows the difference. A Python function passed tomllib.load a file opened in text mode. The experiment record documented a TypeError before the change. The fix was opening the file in binary mode, "rb", as the Python 3.11 documentation requires. After the fix, two tests passed, and an extra check confirmed a corrupt TOML file still raises TOMLDecodeError. That is the project author's experiment, not mine, and not a time-saving benchmark.
Separate success from permissions
Write what may change and what stays out of scope: public interface, dependencies, secrets, other files, deployment. You can allow a fix and test runs in an experiment directory without allowing a global install or a release.
An instruction file can remind the agent of the boundary. It is no substitute for access permissions, environment separation, or the approval of whoever owns the system.
Ask for a verifiable finish
A template, for illustration:
"Fix this failure without changing the public interface. Keep a test that fails on the original code and passes after the fix. Check the error behavior too. Edit only the allowed files, no new dependency, no deployment. When done, show the change, the commands you ran, the actual results, and what remains untested."
In the completion report I want to tell four things apart: code written, test run, result observed, and approval to use. Those are different steps. The local fix can be complete while the work still waits for review or another environment check.
Decide what happens when a test fails
Decide upfront whether the agent may attempt another fix, must stop, or must roll back to a stable version. Define when to escalate to a person: missing information, stepping outside the allowed files, or a change that touches real data.
That way the same task can move to a different agent without losing its intended outcome and boundaries.
From my experience
In my earlier post about defining success for agents I separated "do a refactor" from a task with a preserved interface, clear boundaries, and tests. The principle stays useful even for a small fix: the result must be defined before the agent declares it is done.
Top comments (0)