Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own Prompts · The Edit That Fixed 4 Tasks and Broke 1 · I Let an LLM Rewrite Its Own Prompt. The Real Win Was the Gate That Rejected It. · I Tried 4 Models to Save My Self-Improving Agent. All 4 Failed.
AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners.
Repo: github.com/deghosal-2026/agent-self-edit/tree/v0.3.0
Release notes: docs/release/v0.3.0/release-notes.md
Field test report: docs/field-test/v0.3.0/FIELD_TEST_REPORT.md
I wanted v0.3.0 to be the release where the system finally promoted a good edit.
That didn't happen.
What happened instead was stranger and, honestly, more important. For the first time, the system failed in a way I actually trusted.
That sounds like a consolation prize. It isn't.
Earlier versions could still fail for dumb reasons. A result might be flat because the analyzer was weak, or because the A/B path was wrong, or because the prompt never really changed, or because the field test runner wasn't measuring the thing I thought it was measuring. A bad result still had wiggle room. I could always tell myself, maybe the loop is better than it looks.
v0.3.0 took a lot of that comfort away. We shipped 807 hermetic tests, 16/16 Docker integration tests, 94.86% coverage at release time, Oracle Drift Guard, a real gold corpus, adversarial edit validation, and enough artifacts to inspect every serious run. After all that, the strongest analyzer still did not produce a promotable edit.
That is the first negative result from this project that I believe without adding an asterisk.
Before v0.3.0, a Failed Run Could Still Be a Framework Bug in Disguise
This project does one thing: it watches execution traces, proposes prompt edits, A/B tests them, and promotes only the ones that are measurably better.
That sounds neat. In practice, systems like this can lie to you in boring ways.
The candidate prompt might not actually be materialized. The A/B engine might compare the wrong thing. The gate might not run in the path you think it runs in. The analyzer might be learning from distorted failures. The report might summarize something cleanly while hiding the fact that the underlying pipeline is sloppy.
That was the real problem I wanted to kill in v0.3.0. Not "make the model smarter." First: make the failure legible.
What We Actually Shipped
v0.3.0 was the release where the project got a lot less magical and a lot more accountable.
The big things we shipped:
- 807 hermetic tests with zero paid LLM calls in CI
- 16/16 Docker tests across classification, extraction, generation, staged analyzer, mixed-domain, adversarial, A/B cache, and materialize guard paths
-
94.86% coverage during release validation, with
--cov-fail-under=91 - Oracle Drift Guard so the optimizer, scorer, and corpus can't quietly agree on the same wrong success definition
-
materialize_candidate_prompt()replacing rawstr.replace()so missingold_textfails loudly - real-trace gold corpus with 30 traces, 7 failure clusters, and 7 ideal interventions
- separated-role runner so executor, analyzer, and judge can be different models
- adversarial edit checks showing 8/8 bad edits blocked, 0 false negatives
None of that makes for sexy marketing copy. It does make for a system I can stop apologizing for.
The Best Analyzer Still Couldn't Get Over the Line
The strongest analyzer we tested in the main classification loop was mistralai/mistral-small-3.2-24b-instruct.
It was better than the local 4B model. That's real. It produced the first clean positive movement in the A/B data.
Best iteration:
-
2improved -
0regressed effect_size = 0.0625p = 0.79- gate decision: reject
The local Qwen3-4B-Instruct-2507-4bit analyzer was worse. Mostly null edits. A couple of 1 up, 1 down iterations. No real momentum.
So yes, if you only want the headline, here it is: the optimizer still didn't produce a promotable edit.
But this time that sentence means something specific. The prompts were materialized correctly. The A/B path executed. The gate ran. The artifacts were written. The Docker path matched. The tests backed the mechanics. The framework did its part.
The optimizer failed honestly.
That Was the Breakthrough
I know that sounds backwards, but this is the part I think developers will get.
There is a stage in building systems like this where a trustworthy "no" is much more valuable than a suspicious "yes."
In v0.3.0, a failed run usually narrows down to one of three real explanations:
- the analyzer proposed a weak edit
- the analyzer proposed a small but underpowered edit
- the analyzer got stuck proposing the same kind of local rewrite again
That is a huge upgrade from:
- maybe the prompt never changed
- maybe the gate wasn't actually in the path
- maybe the test runner measured the wrong thing
- maybe the whole result is just harness noise
I can't overstate how much saner the project feels once the failure modes get this narrow.
What Actually Worked
Three things clearly worked.
First, the gate stayed conservative. That is still the healthiest part of the system. We saw 0% false positives, 8/8 adversarial edits blocked, and no bad promotion slipping through local, cloud, or Docker-backed runs.
Second, the system now leaves behind enough evidence to debug behavior instead of arguing about it. analysis.json, accuracy.json, ab-comparison.json, prompt snapshots, traffic logs. If a run falls apart, I can usually tell where.
Third, Docker validation finally meant what it sounded like it meant. Earlier it was too easy to say "Docker passed" while skipping the hardest path. In v0.3.0, we exercised the actual integration surface that matters.
What Didn't Work
The main problem is not hidden anymore: the analyzer is weak.
Not fake. Not random. Weak.
It sees plausible failures and proposes plausible edits. But it keeps collapsing into a tiny search neighborhood. In classification that meant repeated urgency-boundary rewrites. In other corpora it showed up in different clothes, but the habit was the same: local wording tweaks, narrow corrections, not enough exploration.
The project's main question is no longer "does the loop run?" It is "can the analyzer find edits with enough breadth and magnitude to satisfy a conservative gate?"
That is a much better question. It is also a much harder one.
What I Learned
The big one is simple: framework correctness comes before optimizer cleverness. People skip that all the time in AI tooling. They jump straight to model quality while the harness is still blurry. Then every result turns into an argument.
I also learned that negative results are only valuable when they are inspectable. "No improvement" is meaningless if you can't trace how the system got there. "No improvement, and here are the artifacts, the gate reason, the A/B outputs, and the Docker evidence" is a real engineering result.
I learned something subtler too: not every rejection means the same thing. In v0.3.0, the gate is no longer just saying "no." It is separating null edits from net-zero edits and from weak-positive-but-underpowered edits. That matters, because those are different product signals. A candidate that moves nothing is not the same as a candidate that helps a little but cannot yet clear confidence.
And I learned, again, that safety is not the boring add-on at the end. The gate is the boundary that keeps the system from quietly wrecking its own baseline while you're still figuring out whether the optimizer deserves any trust at all.
Why Developers Should Care
If you're building anything that adapts from feedback, I think v0.3.0 points to the right order of operations.
Don't ask first whether the model can improve itself.
Ask whether the system can distinguish signal from noise, reject bad edits safely, explain why it rejected them, preserve artifacts, and survive the same path in CI and containerized execution.
That is less flashy than "self-improving agent." It is also the part that makes the flashy thing worth shipping.
v0.3.0 did not give me the success story I wanted.
It gave me something I needed more: a result I don't have to squint at anymore.
And now I think the more interesting question is not "why didn't it promote?" It is "what kind of negative result should make us change the analyzer, and what kind should make us change the gate?"
If you were building this system, what would you trust more at this point: a stronger analyzer, a bigger promotion corpus, or a looser significance bar? And at what point does a self-improving system deserve to call itself self-improving if the safety layer is working but promotion still almost never happens?
Top comments (0)