Quick disclosure up front, because it matters for how much weight to put on anything below: this was one evening of self-directed practice, no external user or stakeholder involved, six small repos, all built with Claude Code (every commit is paired with a session, per the commit trailers). The biggest eval set in any of the six repos is 8 tasks. This is not "I built a self-improving agent." It's closer to: I ran five fast tests of one idea, found they shared a scoring bug, and then built a fix for that bug that I could actually verify. In mock mode. No live API calls anywhere in this story.
If you're working on anything with an optimize-against-a-judge shape (prompt search, RAG tuning, agent config search, eval-driven pipelines), the bug is worth knowing about before you hit it yourself.
The setup
The question I wanted an actual answer to: does mutate → judge → keep-if-better → repeat generalize as a loop shape, or does every kind of artifact need its own bespoke approach?
Instead of arguing about it, I built five tiny versions, back to back, in about 7 minutes of commit timestamps total:
-
self-improving-prompt-agent: mutates a prompt from a fixed candidate list, heuristic judge (0.0 to 1.0) -
context-improving-agent: mutates a context block, judged by0.8*fact_coverage + 0.2*(1-bloat_penalty) -
graph-improving-agent: inserts one role node at a time into a workflow graph, judged on role-keyword presence -
harness-improving-agent: adds one guardrail statement at a time from a list of ten, judged on keyword coverage -
agent-improving-agent, one level up: mutates the priority orderharness-improving-agenttries its ten guardrails in, since you rarely get to test all ten before your eval budget runs out
Four of the five repos have exactly one commit. That's what five fast structural tests of one hypothesis actually looks like.
The loop shape held up. All five ran the same structure and all five climbed over 10 rounds:
| Repo | Result (10 rounds) |
|---|---|
| prompt | 0.10 → 0.80 |
| context | 0.20 → 0.28 |
| graph | 0.10 → 0.70 |
| harness | 0.00 → 0.90 |
| agent (meta) | 0.45 → 0.61 (stated max: 0.80) |
Five for five, on paper. That wasn't actually the interesting result.
The bug hiding in plain sight
Every one of these repos has a "Known Limitations" section, because writing down what's wrong with what I just built while I still remember it is a habit. Read individually, each limitation sounds like a minor, honest caveat:
- graph: "no check that the wiring is actually runnable/useful"
- harness: "not a real check that the agent obeys the stated guardrails at runtime"
Read together, though, these are the same bug five times, wearing five different filenames. Every judge in this set scored whether the right words or shapes showed up, not whether the artifact actually worked:
- A graph judge that checks for a "verifier" keyword scores an inert, do-nothing verifier node exactly the same as a functional one.
- A harness judge that checks for the word "rollback" scores that word being present the same whether or not any rollback mechanism exists.
-
context-improving-agent'sfact_coverageterm almost certainly measures whether facts are textually present, not whether the context is coherent once they're added.
None of the judges were malfunctioning. They did exactly what I told them to measure. The problem is I told five of them, separately, to measure the wrong thing, and I only caught it by reading all five READMEs side by side instead of shipping straight to a sixth prototype.
There's a second wrinkle worth calling out on harness-improving-agent specifically: its judge used loose keyword matching early on and stalled at 0.20 for several rounds, until I rewrote the judge's wording mid-run to match the mutation wording it was supposed to be scoring. The jump to 0.90 partly reflects that fix, not clean search progress, and the repo's own README says this directly.
The fix: sia
About 2.5 hours after the fifth prototype's commit, I built a sixth repo, sia (self-improving-agent), whose README opens by naming the first five as prior art and states plainly why it doesn't reuse their judges: porting a structure-presence judge into something meant to be real "would have meant the keep/rollback gate approved patches for saying the right words rather than answering questions correctly."
Four changes, each one closing a specific hole from the first five:
1. Frozen judge, held-out eval. sia optimizes a genome (versioned JSON: prompt/context/workflow/harness layers) against a fixed 8-task set, split 5 train / 3 holdout. The judge doesn't get edited mid-run, unlike what happened with harness-improving-agent.
2. A real gate, not a bare threshold:
keep iff train_child >= train_parent + 1.0
AND holdout_child >= holdout_parent
The +1.0 minimum lift on train, combined with non-regression on holdout, is there specifically to catch a patch that overfits the five training examples without generalizing. None of the first five repos had a holdout set at all: they scored and kept against the exact same signal they optimized against, every single round.
3. Integrity checks on the eval itself. sia/loop.py hashes the eval set and the judge file at the start of a run, and re-checks that hash every iteration. If either changes, the run aborts with a logged ABORT row. Watching five judges reward the appearance of a good answer taught me this directly: an optimizer that's allowed to edit what it's graded against will eventually find that editing the test is cheaper than improving the artifact.
4. Positive and negative controls, not just a demo run. Checked directly against sia's own ledger, not a printed summary I trusted at face value:
- Context patch: train 0.0→50.0, holdout 0.0→33.33 (KEPT)
- Harness patch: train 50→70, holdout 33.33→66.67 (KEPT)
- Workflow patch: train 70→90, holdout 66.67→100 (KEPT)
- A deliberately-constructed patch with zero train lift (ROLLBACK)
3 keeps and 1 correct rollback show the gate distinguishing real lift from no lift. That's a stronger claim than "the score went up." I also ran the test suite live: 21/21 unit tests passing.
One more scope decision worth mentioning: sia's README states "The AI is not allowed to call itself better." The improver component can write a hypothesis about why a patch might help, but that text is documentation only. It never feeds into the score.
What I can't claim
Everything above happened in mock mode, a simulated worker standing in for a real model call. There's an API key sitting in a .env file that was never used to make a real call: no logs, no genomes, no ledger rows anywhere attributable to a live run. sia's client hard-raises if it's ever invoked with mock=True, so there's no ambiguity about which mode produced these results. That also means I genuinely don't know whether the gate, the holdout split, or the hash-check survive contact with real, noisy model output. It might. I haven't tested it.
Also unresolved: the meta-loop from agent-improving-agent (the one that optimizes guardrail priority order) was designed to plug into exactly this kind of frozen judge, and I never wired it in. The README calls it future work, plainly.
The takeaway
If you're building any kind of optimization loop over LLM output, the thing worth taking from this isn't the five prototypes or sia's specific architecture. It's the order of operations: before investing in a better mutation strategy or a smarter search, check what your judge is actually measuring, and whether that's cheap to satisfy without being correct. A keyword-presence judge gets gamed eventually, not because the search is adversarial, but because "score well against this specific signal" is always an easier target than "be right," and any sufficiently capable optimizer finds the easier target first.
I built the same mistake five times before I noticed it, then spent the rest of the night on the unglamorous fix: a frozen test set, a holdout split, and a hash check that (so far) has never fired.
Top comments (0)