Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own Prompts · The Edit That Fixed 4 Tasks and Broke 1 · I Let an LLM Rewrite Its Own Prompt. The Real Win Was the Gate That Rejected It. · I Tried 4 Models to Save My Self-Improving Agent. All 4 Failed.
AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners.
Repo: github.com/deghosal-2026/agent-self-edit/tree/v0.3.0
Release notes: docs/release/v0.3.0/release-notes.md
Field test report: docs/field-test/v0.3.0/FIELD_TEST_REPORT.md
I thought this comparison was going to be boring.
Small local model bad. Bigger cloud model better. Split the roles across models and things get even better.
That story would have been clean. It would also have told me less.
What v0.3.0 actually showed was messier and more useful.
The Three Setups
I ran three versions of the same self-edit loop:
- local
Qwen3-4B-Instruct-2507-4bit - cloud
mistralai/mistral-small-3.2-24b-instruct - a separated-role setup with a different executor and analyzer
Same basic flow every time: collect failed traces, let the analyzer propose an edit, A/B test candidate vs current prompt, then run the gate.
For the main classification field test, the promotion corpus had 40 tasks, the held-out set had 25 tasks, the gate required p < 0.05, and minimum effect size was >= 5%.
So this was not a vibes comparison. The models were all walking into the same evidence bar.
The Local 4B Was Flatter Than I Expected
The local model was Qwen3-4B-Instruct-2507-4bit.
Top-line result:
- baseline:
60.0% - final:
60.0% - iterations:
5 - promotions:
0
That already isn't great. But the per-task breakdown was what made it click for me.
Most iterations did nothing.
- iteration 1:
0improved,0regressed - iteration 2:
0improved,0regressed - iteration 3:
1improved,1regressed - iteration 4:
0improved,0regressed - iteration 5:
1improved,1regressed
That is not a "statistics hid the real improvement" story. It is mostly a null-edit story.
The local model was still useful. It was cheap, stable, and good enough for executor or baseline work. But as an analyzer it kept circling the same little patch of ground.
That was surprise number one.
The Bigger Cloud Model Helped. Just Not Enough.
Then I switched to mistralai/mistral-small-3.2-24b-instruct.
This was the first run that felt like it might actually teach me something new.
Top-line result:
- baseline:
64.0% - reported final held-out artifact:
68.0% - iterations:
8 - promotions:
0
That 64 -> 68 number needs a warning label. No prompt was actually promoted, so that is not a deployed improvement claim. The field test report is careful about that, and it should be.
What mattered more was the best iteration in the A/B data:
-
2improved -
0regressed mean_delta = 0.05effect_size = 0.0625p = 0.79
That is the first place the project produced small, clean positive movement instead of a tie or a cancel-out.
So yes, the stronger model helped.
But it helped less than the usual "just use a bigger model" story would suggest. It made the optimizer better. It did not make it good enough.
That was surprise number two.
The Role-Separated Run Was the Weirdest Result of All
v0.3.0 shipped a separated-role runner so executor, analyzer, and judge could be different models.
That sounded like the obvious next move. Use a cheaper executor, a stronger analyzer, maybe a stronger judge when needed. Mix and match. Get the best of each role.
The first run used:
- executor:
qwen/qwen3-30b-a3b-instruct-2507 - analyzer:
mistralai/mistral-small-3.2-24b-instruct - iterations:
3 - held-out sample:
5 - promotion sample:
10
And then it did something I did not expect at all.
It produced zero proposals.
Not bad proposals. Not weak proposals. None.
Baseline was sane at 60.0%, the run directories existed, analysis.json files existed, prompt-a.md existed, and there were no error.txt files. But prompt-b.md and ab-comparison.json never appeared because the run never got that far.
That was surprise number three, and easily the biggest.
The obvious theory was: stronger analyzer equals better proposals.
The separated-role run says that theory is incomplete.
The analyzer does not work on abstract task labels. It works on failure traces. Change the executor and you change the failures. Change the failures and you may change whether the analyzer sees anything stable enough to turn into an edit.
That is a much more interesting systems result than a simple leaderboard.
What Stayed the Same Across All Three
This was the part that convinced me the bottleneck is not just model size.
Even when the model changed, the failure pattern stayed familiar. The analyzers kept drifting toward the same edit family:
- urgency-boundary rewrites
- local wording clarifications
- narrow rule tightening
What I didn't see much of:
- structural prompt changes
- broader task decomposition changes
- genuinely diverse proposal families
- any real escape from the first obvious interpretation of the failures
That points to a search problem, not just a horsepower problem.
The Honest Comparison
| Setup | What worked | What failed | Main learning |
|---|---|---|---|
| Local 4B | Cheap, stable, good for mechanics | Mostly null edits | Fine for executor/baseline work, weak for analyzer search |
| Single-model Mistral 24B | First real positive signal | Still no promotion | Bigger model helps proposal quality, but not enough |
| Separated-role Qwen 30B + Mistral | Runner worked, baseline sane | Zero proposals | Better analyzer alone is not enough; executor output shapes analyzer behavior |
That is not the result I expected to write about.
It is better than the result I expected to get.
What I Learned
First, bigger models can improve signal without solving the actual problem. That sounds obvious, but people flatten this distinction all the time. Better is not the same thing as good enough to trust.
Second, role separation changes the optimization landscape. I had assumed it would mostly be a routing and cost story. It turned out to be a behavior-shaping story. Different executor outputs gave the analyzer a different failure surface to work from.
Third, model comparison without artifact inspection is too shallow for systems like this. The useful truth was not in a single score. It was in how many proposals appeared, what kind they were, which tasks moved, and where the loop stopped.
And one more thing became obvious in v0.3.0: reported final accuracy in a no-promotion run needs a warning label. If the deployed prompt never changed, a better-looking final held-out number is not the same thing as a shipped improvement. That should probably become a standard reporting rule for agent experiments, because otherwise people read movement in the artifacts as movement in production behavior.
Why Developers Should Care
If you're building any multi-model agent system, I think this is the important takeaway: changing the model changes the failure surface, not just the score.
That means a stronger analyzer may still stay trapped in a narrow neighborhood. A different executor may change what the analyzer can even notice. And a more complicated architecture is not automatically a better one.
That is not anti-agent advice. It is just a reminder that coupling is real, even when the boxes in the diagram look clean.
I went into v0.3.0 expecting a neat story about better models.
What I got was a more useful story about where model quality helps, where it doesn't, and how quickly the architecture itself starts shaping what the system can learn.
If you were choosing the next experiment, what would you run: another stronger single-model analyzer, a deeper role-separated run, or a larger corpus with the same models? And when a bigger model helps only at the margin, how do you decide whether the problem is model quality, search strategy, or just the math of a conservative gate?
Top comments (0)