Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own Prompts · The Edit That Fixed 4 Tasks and Broke 1 · I Let an LLM Rewrite Its Own Prompt. The Real Win Was the Gate That Rejected It. · I Tried 4 Models to Save My Self-Improving Agent. All 4 Failed.
AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners.
Repo: github.com/deghosal-2026/agent-self-edit/tree/v0.3.0
Release notes: docs/release/v0.3.0/release-notes.md
Field test report: docs/field-test/v0.3.0/FIELD_TEST_REPORT.md
Role separation felt like the obvious upgrade.
Stop forcing one model to do everything. Let one model execute tasks, another analyze failures, another judge outputs when needed. Use the right model for the right job.
That sounds so reasonable it almost feels boring.
So when v0.3.0 shipped a separated-role runner, I expected the first result to be at least directionally better.
Instead, the first real run produced zero proposals.
That stung a little. It also taught me more than a clean win probably would have.
Why I Thought Role Separation Would Help
The single-model story already had a clear problem.
The local 4B model was cheap and mechanically useful, but weak as an analyzer. The stronger cloud Mistral model produced the first small positive signal, but still got stuck making narrow, local edits.
So the natural next move was: keep a decent executor, use a stronger analyzer, maybe keep a separate judge for tasks that need it.
That idea is attractive because it matches how people talk about agent systems in general. Planner here. Critic there. Specialist over there. Split the roles and improve the whole.
I bought that story too.
What We Shipped in v0.3.0
v0.3.0 added separated-role support so executor, analyzer, and judge can come from different provider configs with fallback.
That was a real feature, not a paper design.
The first separated-role classification run used:
- executor:
qwen/qwen3-30b-a3b-instruct-2507 - analyzer:
mistralai/mistral-small-3.2-24b-instruct - iterations:
3 - held-out sample:
5 - promotion sample:
10
Top-line result:
- baseline held-out:
60.0% - final held-out:
60.0% - promotions:
0 - proposals:
0in every iteration
The zero is the whole story.
Zero Proposals Is a Different Kind of Failure
The local 4B run failed one way: weak edits.
The single-model Mistral run failed another way: small positive movement, still not enough to promote.
The separated-role run failed earlier.
It did not even generate proposals.
That distinction matters.
The artifacts made the failure point pretty clear:
-
analysis.jsonexisted in every iteration -
prompt-a.mdexisted -
accuracy.jsonexisted -
prompt-b.mddid not exist -
ab-comparison.jsondid not exist -
prompt-after.mddid not exist - no
error.txtfiles were written
That means the run started normally, collected failures, entered the analyzer path, produced an analysis artifact, and then stopped because there were no proposals to continue with.
This was not a weak-promotion problem. It was a no-proposal problem.
The Part That Surprised Me Most
The separated-role run did not fail because the executor was obviously broken. Baseline was plausible at 60.0%.
It did not fail because the runner crashed. The directories and artifacts were there.
It failed because the analyzer produced nothing.
That changed how I thought about role separation.
I had treated it like a model-quality upgrade. Better model in the analyzer slot, problem improves.
The run pushed back on that assumption hard.
The analyzer doesn't reason over abstract task IDs. It reasons over the concrete failure traces coming from the executor. So changing the executor changes the analyzer's input surface. Different failures, different clustering, different patterns, different chance of seeing something edit-worthy.
That means role separation is not just a routing improvement. It is a learning-surface change.
Why a Stronger Analyzer Wasn't Enough
This is the part I think is easiest to miss.
The analyzer model in the separated-role run was not worse than the one that produced weak positive signal in the single-model cloud run. In fact it was the same stronger analyzer candidate: mistralai/mistral-small-3.2-24b-instruct.
So why did one setup give small positive edits and the other give none?
The best explanation from the evidence is that the analyzer is sensitive to the shape of the failures it sees.
Three possibilities seem plausible:
- executor-output shift: the 30B executor produced a different kind of mistake distribution, one the analyzer did not turn into proposals
- smoke-size underexposure: with only 5 held-out and 10 promotion samples, the analyzer may not have seen enough recurring structure
- staged-analyzer brittleness: the pipeline may still be too sensitive to exact failure phrasing coming from a different executor model
I don't think the lesson is "role separation is bad."
I think the lesson is that role separation is not a free win.
What This Taught Me About Agent Architecture
Agent diagrams make roles look cleaner than they are.
Planner, executor, critic, judge. Nice boxes. Nice arrows.
In real systems, those roles are coupled through the artifacts they produce for each other. Change one box and you change the texture of the evidence flowing into the next one.
That coupling is exactly what this run exposed.
The analyzer was not just "a better model waiting to help." It was a model reacting to a different failure landscape, and in this case that landscape was apparently much less fertile.
That is a more interesting architectural lesson than a simple benchmark gain.
What Worked Anyway
Even though the run produced zero proposals, I don't see it as a wasted experiment.
The separated-role runner itself worked.
The unique output directory logic worked.
The baseline wasn't nonsensical.
And, more importantly, the run ruled out an overly simple story I was half ready to believe: "just keep the executor cheap and swap in a stronger analyzer."
The project is better off not believing that story anymore.
What I Learned
First, role separation changes more than cost and quality. It changes what the system can learn from.
Second, stronger models are not enough if the surrounding evidence surface changes in ways the analyzer can't use well.
Third, zero-proposal runs deserve as much attention as weak-positive runs. They are different failure classes. If you lump them together under "no promotion," you throw away the most useful part of the diagnosis.
And there is another lesson here that I did not expect to matter this much: the artifact shape tells you almost as much as the numeric result. analysis.json without prompt-b.md or ab-comparison.json is a very different story from a run that reaches A/B and then fails confidence. Those should be treated as different architectural events, not just different rows in the same score table.
Why Developers Should Care
If you're building multi-model agent systems, I think this is the important warning label: specialization creates interaction effects.
Swapping a better model into one role does not just improve that role. It can change the whole system's learning behavior, because the roles are coupled through real outputs, not abstract interfaces.
That is the kind of thing you only really appreciate once a run fails in a new way.
I shipped role separation expecting an upgrade.
The first real run gave me a negative result instead.
I trust that result enough now to say this without hedging: that was still progress.
If you are splitting roles across models in your own system, how are you checking whether the roles are actually helping each other instead of just making the architecture look smarter? And when a stronger analyzer produces no proposals at all, do you treat that as a model problem, a dataset problem, or a sign that the executor changed the learning surface too much?
Top comments (2)
The part that matches my own pipeline runs is how the failure distribution shifts the moment you change the executor. When we split the optimizer from the worker, the analyzer stopped seeing neat deterministic syntax errors and started getting semantic misfires where the task context looked plausible but subtly drifted off-spec. If the analyzer prompt was tuned to extract diffs from structured trace blocks, an unstructured executor output basically gives it zero actionable handles.
Reid, that matches exactly what we saw in the separated-role run and it's helpful to hear you've hit the same pattern.
The shift from deterministic syntax errors to semantic misfires is the right way to describe it. In our case, the Qwen 30B executor produced outputs that looked reasonable — they just didn't fail in the structured ways the analyzer had learned to exploit from the local 4B run. The analyzer prompt had implicitly tuned itself to the local model's failure signature without us realizing it, and swapping the executor collapsed the analyzer's effective search space.
The trace format normalization question is one I haven't solved yet. The analyzer's input prompt uses a batch of raw trace objects with task_input, final_output, expected_output, and failure_reason fields — so the format is stable regardless of executor. But the content of those fields changes dramatically: the local 4B produces short, brittle outputs that fail on exact matches, while Qwen 30B produces verbose, plausible-sounding misclassifications. Same schema, completely different signal structure.
One direction I'm considering: a trace "normalizer" stage that pre-processes executor output into a standard failure representation before the analyzer sees it, so the analyzer can focus on edit strategy rather than deciphering a new executor's output style each time. Have you tried anything like that in your pipeline, or did you find the fix was to keep executor and analyzer tuned to each other as a pair rather than normalizing between them?