Debashish Ghosal

Posted on Sep 2 AI-assisted

I Let an LLM Rewrite Its Own Prompt. The Real Win Was the Gate That Rejected It.

#ai #healthydebate #llm #agents

Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own Prompts · The Edit That Fixed 4 Tasks and Broke 1

I built an agent that can rewrite its own system prompt. The impressive part was not the rewriting. It was the refusal.

That's the flashy part. It is also the wrong place to start.

The optimizer can be an LLM, a heuristic, or random guessing. The gate must be deterministic, verifiable, and conservative. If the gate is wrong, the system drifts. If the gate is right, the system is safe — even when the analyzer is proposing garbage.

After 4,150 LLM calls, 716,580 tokens, 4 domains, 4 models, 12/12 Docker tests, 489 passing tests, and 26 closed v0.2.0 issues, here's what actually worked — and the 2 releases it took to prove it.

What Worked: The Architecture

The critical design decision: the analyzer has zero authority. It only proposes. Every proposal goes through A/B test + gate. The LLM never decides whether its own edit is good. Code decides.

Agent executes task → trace stored (SQLite)
    ↓
Feedback Analyzer (LLM) reviews traces, proposes edits
    ↓
A/B Test Engine: candidate vs current prompt on 40-task set
    ↓
Promotion Gate (6 deterministic checks)
    ↓
Promote / Near-miss / Reject

The entire loop runs on a MacBook with a local Qwen 4B model via MLX. No cloud API keys. No per-token costs. 4,150 LLM calls in the full field test: $0.00. Total wall time: 37 minutes.

This matters because it changes the economics of debugging. When each iteration costs nothing and finishes in 42 seconds, you run, inspect, fix, and re-run without thinking about it. On cloud, each iteration takes 3+ minutes. You wait. You batch your debugging. You lose the rapid cycle where you find bugs fastest.

What Worked: The Gate

Six checks, fail-fast, deterministic, no LLM involved.

CHECK_ORDER = [
    sample_floor,      # 1. Enough data?
    effect_size,       # 2. Big enough improvement?
    confidence,        # 3. Statistically significant?
    frozen_sections,   # 4. Touched protected content?
    edit_distance,     # 5. Rewrote everything?
    drift,            # 6. Still recognizable?
]

Fail-fast means the first failure stops the chain. Fixing one bug reveals the next. This property is why v0.2.0 found bugs that v0.1.0 missed — when the confidence check was broken (p < 0.95 instead of p < 0.05), it masked the frozen sections bug. Fixing confidence revealed frozen sections. Fixing frozen sections revealed drift calibration. Each fix surfaced the next problem.

Sample floor — Minimum A/B trials completed. Default 5. Prevents promoting on 2-3 data points.

Effect size — Improvement must exceed 5%. The most common blocker in v0.2.0. The analyzer's edits typically moved 1-3 tasks out of 40. Real movement. Not enough lift.

Confidence — p < alpha where alpha = 0.05. This check caught the most insidious bug ever shipped: the original code checked p < 0.95. A p-value of 0.9 would pass. A coin flip would pass. This bug alone wasted weeks of iterative debugging on false results.

Frozen sections — User-annotated prompt regions the analyzer cannot touch. The gate reconstructs the candidate prompt and verifies every frozen block is still present.

Edit distance — Max 20 lines changed per cycle. No wholesale rewrites.

Drift — TF-IDF cosine distance from original prompt. Prevents evolution into unrecognizable prompts over hundreds of iterations.

Three outcomes

Outcome	Meaning
Promote	All 6 pass. Registry creates new version with full lineage.
Near-miss	Most pass. Logged for human review with proposal text.
Reject	Critical check failed. Full reasoning archived.

Near-misses are the most interesting outcome. In v0.2.0, every proposal was a near-miss or reject. The analyzer found directionally correct edits. It just couldn't find one strong enough to clear the bar.

What Worked: Docker Killed the Excuses

12/12 Docker tests passed across 4 domains. That matters because Docker was where the easy excuses should have shown up first.

Domain	Test	Result	LLM Calls
Classification	Full loop	PASS	13
Extraction	Full loop	PASS	13
Generation	Full loop (with judge role)	PASS	23
Staged analyzer	Full proposal pipeline	PASS	13

Docker proved:

Multi-domain scoring works. StructuredExtractionScorer, LLMJudgeScorer, ExactMatch — auto-selected at runtime.
Model role wiring is real. Generation required the judge role — 23 LLM calls vs 13 for classification.
Containerized execution is reproducible. Same results as bare metal.

After Docker passed, we ran out of excuses. The bottleneck was not packaging, missing dependencies, or misconfigured roles. It was something else entirely.

Where We Wasted Time

2 releases proving the plumbing instead of proving the optimization.

v0.1.0 was supposed to prove the loop worked. Instead, it proved we had 9 bugs that looked like a working system — fabricated traces, inverted confidence checks, comparing prompts against themselves. We fixed those in v0.1.0, but we still didn't know if the system could produce a promotable edit.

v0.2.0 was supposed to answer that. It did. But it took another full release — 9 milestones, 26 issues, 4 models — to prove that the pipeline is correct and the optimizer is the bottleneck.

That's 2 releases to learn what should have been learnable in 1. The wasted time came from:

Fixing bugs that masked the real problem. The confidence bug alone (p < 0.95) let us believe the system was working when it was making false promotions. We celebrated outcomes that were noise.
Chasing model size. v0.1.0 tested 3 models (4B, 9B, cloud). All made the same errors. The problem was never the model — it was the prompt and the search strategy.
Not running Docker sooner. Docker integration was M7 in v0.2.0. If we'd containerized from v0.1.0, we'd have known the system was mechanically sound 1 release earlier.

The frustration is real: we built a correct, safe, multi-domain framework — but we can't yet claim the system improves itself.

The Honest Result

Metric	Value
LLM calls	4,150
Gate false positive rate	0% (all runs, all domains)
Adversarial edits caught	5/5 (0 false negatives)
Multi-domain suites	4/4 with correct scoring
Docker tests	12/12 pass
Unit tests	489/489 pass

The gate works. Zero false positives across classification, extraction, generation, and adversarial stress tests. The system is safe. The optimizer is the only part still losing. That is exactly the failure mode you want.

Discussion Questions

How many safety checks are enough? Six felt right for prompt editing. What would you add for code modification or tool access?
Should the gate ever be LLM-judged? We made it pure code for testability. But an LLM-judged gate could handle fuzzy boundaries — at the cost of introducing its own failure modes.
At what point does a near-miss become a promotion? The ratio is configurable, but we never defined a clear transition rule. That's v0.3.0 work.
What's the right balance of framework vs optimization? We spent 2 releases building the framework. Was that the right call, or should we have prototyped the optimizer faster?

The Silver Lining

Two releases to prove the plumbing sounds like waste. It wasn't.

v0.1.0 left us with 9 bugs and no clarity. Was the pipeline broken? Was the data fake? Was the gate wrong? We couldn't tell. Every result was suspect.

v0.2.0 eliminated every suspect. The pipeline is validated across 4 domains in Docker. The gate is proven with 0% false positives. The A/B tests are real. The math is honest. And after all that, the bottleneck is one specific problem: the analyzer can't find edits strong enough to clear the gate.

That's not a vague concern. It's a concrete engineering target with a clear solution: generate multiple candidates, score them cheaply, test the best one. We know exactly what to build for v0.3.0.

The gate works. The framework works. The safety works. The only thing that doesn't work yet is the optimizer — and we know why.

DEV Community