$ python -m autocoder ./my-project
working on a copy at /tmp/my-project.autocoder
outcome: no_progress
attempts 2, edits 2 applied / 0 refused, 3 test runs
baseline fail 0 passed, 1 failed
final fail 0 passed, 1 failed
That is a successful run. The agent could not fix the bug, noticed it was making no progress after two attempts, and stopped — rather than burning fifty model calls to reach the same place, or quietly editing the test until it passed.
The loop everyone writes is four lines:
while not green:
edit = propose(failure)
apply(edit)
green = run_tests()
It works in a demo. It has three failure modes a demo cannot show you, because a demo is one happy path.
Repo: https://github.com/dev48v/autocoder — PUBLIC, MIT, 24 tests, standard library only.
The whole loop runs in your browser: https://dev48.infy.uk/agentlab/vol1-05-autocoder.html
1. It cheats
The shortest path to a green suite is editing the test. Not because the agent is devious — because it is correct. You asked for a green suite; rewriting the assertion is the smallest edit that produces one. The objective was underspecified and the agent found the hole.
A system prompt saying "do not modify tests" is a request. The workspace refusing to open the file is an enforcement point:
DEFAULT_PROTECTED = ("tests/**", "test_*.py", "*_test.py", "conftest.py",
".git/**", "pyproject.toml", "setup.cfg")
Only one of those survives a model that has decided otherwise, and only one is testable.
And every guard ships with a control. The same cheating agent with the guard removed succeeds, and leaves assert True behind:
def test_without_the_guard_the_cheat_would_have_worked(project):
ws = Workspace(project, protected=()) # guard removed on purpose
ws.write("tests/test_target.py", "def test_target():\n assert True\n")
assert "assert True" in ws.read("tests/test_target.py")
Without that test, "the cheat was blocked" might mean the cheat never worked. A guard that prevents nothing passes its own test forever.
Refused edits are still recorded. "It never tried" and "it tried and was stopped" are very different things to find in a log.
2. It spins
Propose an edit, tests fail, propose the same edit, forever. If the thing that stops it is max_attempts, you have not built a stopping condition — you have built a timer, and you pay for every tick.
The fix needs an identity for a failure that survives noise:
def signature(self) -> str:
if self.status == "pass":
return "green"
return "|".join(sorted(self.failing)) # test names, sorted
Sorted, so ordering cannot change it. Names rather than raw output — output carries timings and temp paths, so two runs of an identical failure look different and the detector never fires.
report = run_agent(ws, runner, repeating_proposer(useless), max_attempts=50, patience=2)
assert report.outcome == "no_progress"
assert report.test_runs <= 4 # not 51
Repeated attempts at a protected file count toward patience too. Being refused is not progress; an agent that keeps reaching for the same door is stuck, not persistent.
3. It declares victory it did not earn
A collection error, a syntax error, a timeout — none of those are a test result. They are the absence of one.
@property
def usable(self) -> bool:
return self.status in ("pass", "fail")
An agent that conflates "could not run" with "zero passed" starts fixing tests that never executed. If the baseline run is unusable, this one stops immediately with unrunnable and touches nothing — there is no signal to work against, so any edit is a guess, and a guess that modifies a repo is worse than no action at all.
Five outcomes, and it always names one
| outcome | meaning |
|---|---|
fixed |
green, reached by editing source |
no_progress |
same failure twice, or repeated attempts at protected files |
budget |
attempts exhausted while still making progress |
gave_up |
the proposer had nothing left to try |
unrunnable |
the suite does not run, so nothing can be evaluated |
budget should be rare, and it is worth watching: a loop that routinely ends there has no stopping condition.
Rollback beats reasoning about wreckage
if not result.usable: # the edit broke the suite
ws.revert_last()
regressed = result.passed < report.final.passed
if regressed and revert_on_regression:
ws.revert_last()
Both decrement edits_applied, so the report counts what survived rather than what was attempted. A run that applied four edits and reverted three did not make four changes.
The workspace is the security boundary
target = (self.root / rel).resolve()
if target != self.root and self.root not in target.parents:
raise OutsideWorkspace(...)
The check runs after resolve(). Checking the string first catches ../ and misses absolute paths and symlinks; resolving first catches all three with one comparison. And by default the agent works on a copy — --in-place exists and is not the default.
Testing an agent without a model
Every test but one uses a deterministic stand-in that reads the actual files — so an agent cannot satisfy it by doing nothing, which a hard-coded sequence of verdicts would allow. One test shells out to real pytest end to end, so the stand-in is a convenience rather than the only thing ever exercised.
Swapping in a real model is one callable
def model_proposer(ws, result, attempt) -> Proposal | None:
reply = call_your_model(build_prompt(ws.files(), result.output))
return parse_edit(reply) # or None to give up
The guards, the rollback, the signature comparison and the five outcomes never learn what a model is. The part that keeps the agent safe should not depend on the part that makes it clever.
That is Agent Lab Vol 1 finished — five tools, five repos: a diagram agent, a cited researcher, a personal RAG assistant, a research crew that reports its own bill, and this. The thread through all five is the same one: the interesting engineering is almost never the capability. It is what the thing does when it cannot do the job.
Top comments (0)