The first time my local agent loop "finished" a task, nothing had changed on disk. The drafter reported success, the critic signed off, the task moved to done - and the working directory was exactly as I'd left it. It took me a week of staring at audit logs to understand why, and the answer was one dead branch of code that I had written myself and never exercised.
This is what happens when you build a drafter/critic loop that runs entirely on your own hardware. I'll walk through the loop, the five things that actually broke it, and the one non-obvious lesson that came out of all of it.
The loop, in one paragraph
The pattern is deliberately simple. A drafter model - a CLI coding agent with real file access - is dispatched into a task's working directory and told to produce a change. A separate critic model then judges that attempt against the task's acceptance criteria. If the critic fails it, the drafter retries with the critic's named errors folded back into the prompt. After a configurable number of consecutive failures - the iteration cap, default three - the task escalates to a human as blocked; a comment resets the cap and resumes the loop with your guidance.
Two independent models judging each other, and a hard cap so it can't spin forever. On paper it's a reasonable loop. In practice, on a single 24 GB GPU, it taught me a lot.
War story 1: the critic was never the critic you thought
The design said the critic would be a "direct structured-output call at temperature 0" - a pure judgment, no tools, schema-enforced. That is the whole point of the role: a judge that returns a verdict, not an agent that does things.
The code did not do that. The critic's model tag was read off the critic's agent record, and that record was always populated. So the flag that decides "dispatch as agent" versus "dispatch as a structured call" was always true. The structured-output branch - the one that would have enforced the verdict schema - was dead code. It never ran.
The consequence: the critic ran inside the same CLI harness as the drafter, with tools. And a small model handed a tool harness doesn't behave like a judge. It behaves like an agent. Instead of returning a verdict, it tried to read the prompt file - it emitted a tool call for read, narrated that it "can use the Read tool," and then failed to parse. The loop retried, failed again, and gave up.
The fix was a new setting that makes the structured path reachable again - but the lesson was that a role you describe in a doc is not a role your code enforces. The critic was "conceptually" read-only; nothing in the dispatch made it so.
War story 2: the judge wrote the files it was judging
This is the one I'd tell anyone building an autonomous pipeline. During a run, the working directory suddenly contained a complete Angular project scaffold - 23 files, angular.json, package.json, the whole lot. The task explicitly forbade a framework and a package manager.
It wasn't the drafter. The audit log's action diffs - computed from real before/after directory snapshots, so a model can't fake them - attributed those 23 files to the critic model. In the same dispatch, it returned a clean {"verdict":"pass"} and scaffolded an application into the very directory it was supposed to be impartially judging.
Why? Because of war story 1. The critic dispatching through the CLI agent got the exact same invocation as the drafter: a working directory, full write access, and a flag that skips permission prompts. A model that decides to "build the thing to check it" simply does.
The fix that finally made this structurally impossible: route the critic through a text-only judgment call with no subprocess and no working directory - a component with no capability to write anything at all. A read-only judge that can't touch the artifact is not a nicety; it's the difference between a verdict you can trust and a verdict rendered against a directory the judge itself changed.
War story 3: model choice silently decides whether any file gets written
After fixing the runtime, the drafter ran twice, reported success, and wrote nothing. The audit log showed why: one of my "coder" models emits a fabricated <tool> block as plain prose instead of performing a real tool call. The harness sees text, takes zero actions, and nothing touches disk. The model described writing the file; it never did.
I had assigned that model based on its file size and reputation. The audit log already contained the evidence to predict the failure - a count of which models actually produce real file actions through the harness. Two models made hundreds of real actions; two others made zero. I hadn't consulted the log before reassigning.
The systemic gap: the agent dropdown offered a model purely because the local server had it pulled. There was no capability probe for tool use. A drafter on a non-tool-calling model will burn its entire iteration cap producing confident, empty attempts, and report success the whole time.
War story 4: the load tax
A drafter/critic loop alternates roles constantly - drafter, critic, drafter, critic - inside a single run. Each model switch on a local runtime can mean a cold load. In my first run, 81% of the wall-clock time went to model loading, not to actual generation. The runtime was configured to keep only one model resident at a time and pin it forever. The fix was two environment variables; the insight was that the bottleneck was the runtime config, not the models and not the GPU.
What worked (so it's not all war stories)
Mostly, it held. Auto-generated acceptance criteria were genuinely strong - nineteen items, each traceable to the task description, locked with no prompting. The state machine never produced an illegal transition, even through repeated dispatch failures. And the observability is what made all of this debuggable at all: a JSONL audit log of every model call, with durations, token counts, and - crucially - file-action diffs from real directory snapshots. Nearly every number in this post came straight out of that log.
When I did re-run the pipeline after the fixes, it delivered a seven-file site with zero human intervention - and when I checked it outside the system, with a script that never consults the loop, it held on 64 of 66 points.
The non-obvious lesson
A claim is not a verification. The drafter claimed it wrote the file. The critic claimed it judged it. The task claimed it was done. None of those claims were backed by what actually happened on disk - until I reconstructed the run from the recorded state, directory snapshots and all. The single strongest signal in the whole system, status: done, turned out to be a copy of one step's verdict, not a whole-task check.
If you're building a local agentic loop, three things I'd do before trusting it:
- Enforce the judge's role in the dispatch, not just in the doc. A read-only critic that cannot write is the difference between a trustworthy verdict and a self-contradiction.
- Probe models for the capability you're actually using. Tool-calling and structured output are different capabilities. "It's a coding model" is not evidence it can drive your harness.
- Make the runtime config a first-class suspect. When 80% of your time is loading, the problem is almost never the model.
Top comments (0)