We built a loop that edits an AI agent's tools, prompts, MCP surfaces, and skill packages. Then we let the agent drive the loop itself. Here's what happened.
⭐ Every GitHub star helps. Thank you! ❤️
The problem
There's a class of agent failure that prompt-tuning alone doesn't fix.
The agent understands the policy. It can quote the rule back at you. But it fails on the same task patterns anyway — because the failure isn't a knowledge gap, it's structural. A constraint enforced in tool code can't be forgotten mid-conversation. A validation guard at the tool boundary catches a booking error before the model can reason around it. A composite tool that hard-wires a multi-step workflow removes the opportunity to drop a step.
And these failures are only visible in the full trajectory: you have to read what the agent actually did across every task, cluster the failure modes, figure out which prior edits broke what was working, propose a fix that addresses every cluster without regressions — and repeat across dozens of iterations.
This shows up on real benchmarks:
- τ²-bench — tool-calling agents in customer service domains (airline booking, retail)
- SWE-bench — coding agents resolving real GitHub issues, graded by whether the test suite passes
- SkillsBench — agents running from procedural skill packages, tested across 84 real-world tasks with deterministic verifiers
The bottleneck in all three is the same: the iteration loop — evaluate, diagnose, edit, gate, repeat.
cap-evolve is that loop.
What it does
cap-evolve runs an outer optimization loop over an AI agent's capabilities — the things the agent reads and executes:
- System prompts
- Executable tool code
- MCP tool surfaces (docs, examples, exposed tools)
- Skill packages (
SKILL.mdbodies, references, scripts)
It works across domains: tool-calling agents (τ²-bench), coding agents that patch real GitHub issues (SWE-bench), and agents that run from procedural skill packages (SkillsBench). Adapter templates for all three ship in the repo.
You bring the agent and an eval you already have. Each iteration:
- Evaluate the current best candidate on a train/val split
- Diagnose — read the full trajectories (successes and failures), cluster failure modes, identify which prior edits broke what was working and fixed what wasn't
- Propose a large, multi-part edit addressing every failure cluster without regressing the wins
- Gate — accept the edit only if it beats the held-out val split by a significance margin (Δ > k·SE)
- Commit — every candidate is a git commit, win or loss
At the end, a sealed test split — which the optimizer never touched — is scored exactly once. That's the headline number.
Two orchestration modes
cap-evolve can run the loop in two ways, set by a single key in capevolve.yaml:
Deterministic mode (default): cap-evolve run sequences the phases itself — intake → baseline → algorithm → finalize — spawning a per-iteration optimizer subprocess (Claude Code, Codex, Gemini CLI, etc.) to propose each edit. The engine evaluates on val, applies the gate, and accepts/rejects. Fully hands-off and reproducible — set it running and come back to a dashboard.
Agent mode: cap-evolve run does check + baseline, hands back the run directory, and returns. The same conversational agent that ran intake then drives the optimization itself using the agent-optimize algorithm — reading the failing-task feedback, proposing a targeted edit, evaluating on full val, accepting only on the paired significance gate. No optimizer subprocess; the agent is the optimizer. The user can steer or halt at any round.
Both modes share the same honesty invariants — seeded splits, val-only gating, sealed test — because those live in core/ and can't be bypassed by any skill, prompt, or CLI flag.
What the optimizer actually changed
On the τ²-bench airline run (deterministic mode, Claude Code optimizer), the optimizer didn't just rewrite rules. It expanded tools.py from 593 to 832 lines — adding deterministic enforcement of fare class constraints, input validation that catches booking errors at the tool boundary, and composite helper tools that short-circuit common multi-step patterns.
The policy prompt grew too (166 → 233 lines), but most of the gain came from the tool edits. Five of ten iterations were gate-accepted; the rest were rejected as within-noise. You can read the five trajectory-verified before/after edits in OPTIMIZATION_EXAMPLES.md and walk through the full run in DEMO.md.
Here's what that looks like in the dashboard — the fitness stair only ever steps up, because the gate refused every change it couldn't distinguish from noise:
And here's the lineage tree — green accepted candidates, red rejected branches, and the paired-gate math (Δ > k·SE) behind each decision:
Results
Here's what the loop actually produced across different benchmarks and models. (Full results page with all caveats →)
τ²-bench airline — fit-metric run (committed, reproducible)
Run with train == val == test == all 50 tasks — a fit metric, not a generalization claim, but fully committed and reproducible:
| reward (50 tasks · 10 trials) | Δ vs baseline | |
|---|---|---|
| Baseline (seed policy + tools) | 0.536 (53.6%) | — |
Best candidate (cand_0007) — val |
0.712 (71.2%) | +32.8% relative |
cand_0007 — sealed test (fit metric) |
0.694 (69.4%) | — |
Artifact: examples/tau2_airline/run_full/. Reproducible via REPRODUCE_tau2.md.
The dashboard for this committed run is interactive — you can browse every iteration's git diff, cost breakdown, and gate decision offline:
c 20/50 run (reported; artifact pending)
Same benchmark and capability, run with a real holdout split (30 train / 20 test):
| split | baseline | optimized | Δ |
|---|---|---|---|
| val (30 tasks) | 56.7% | 70.0% | +13.3 pp / +23.5% relative |
| sealed test (20 tasks, scored once) | 30.0% | 47.5% | +17.5 pp / +58.3% relative |
Note: the
run_fullartifact for this run is not yet committed. The numbers are reported; the reproducible artifact-backed run is the fit-metric run above.
τ²-bench airline — agent orchestration mode (train=val=30, test=20)
First run driven entirely in agent mode with agent-optimize: the conversational agent ran the baseline, proposed airline-policy edits itself, gated every candidate on full val, and sealed the test once.
Single-trial (pipeline default):
| split | baseline | best (cand_5) |
Δ |
|---|---|---|---|
| val (30, fit) | 0.500 | 0.633 | +26.7% relative — gate-significant |
| sealed test (20, held-out, scored once) | 0.400 | 0.550 | +37.5% relative |
Honest reading: τ²-bench airline is high-variance at a single trial. On a stable 3-trial re-evaluation, the val fit gain is +18.4% (paired-significant, Δ/SE = 1.80) while the held-out test is flat within noise (−0.067, SE 0.105). Because
train == val, the val figure is a fit, not generalization. Agent mode produced the only gate-accepted improvement in a head-to-head against a bounded deterministic run (whose best stayed the seed; sealed test 0.35). A genuine stable held-out gain needs a stronger runner model or tool-level edits.
τ²-bench airline — Qwen 2.5 14B (self-hosted, held-out)
Same benchmark and split (30/20 held-out), replacing the cloud runner with Qwen 2.5 14B-Instruct via vLLM on OpenShift. This is what it looks like on a self-hosted open model with no API dependency.
Tools + policy optimized jointly:
| split | baseline | optimized | Δ |
|---|---|---|---|
| val (10 tasks) | 20.0% | 38.7% | +93.5% relative |
| sealed test (10 tasks) | 17.0% | 24.0% | +41.2% relative |
All three capabilities ([skill-package, system-prompt, tools]) optimized jointly:
| split | baseline | optimized | Δ |
|---|---|---|---|
| val (10 tasks) | 27.3% | 52.0% | +90.5% relative |
| sealed test (10 tasks) | 12.0% | 27.0% | +125.0% relative |
3 of 10 iterations accepted in both runs. On a self-hosted 14B model, jointly optimizing all three capability types outperformed the tools-only run (+125% vs +41.2% on test).
SkillsBench — skill-package optimization (held-out, committed)
SkillsBench tests agents across 84 real-world tasks using deterministic verifiers. Agents are evaluated under three conditions: no skills, curated skills, and self-generated skills. cap-evolve targets the curated skills condition — it optimizes the SKILL.md packages the agent reads at runtime.
Setup: four shared office-document skill packages (docx/pptx/xlsx/pdf). Agent under test: claude-sonnet-4-6 in a Docker sandbox; optimizer: claude-opus-4-8.
| reward | Δ | |
|---|---|---|
| Baseline — val | 0.333 | — |
Optimized (cand_0004) — val |
0.714 | +114% relative |
| Baseline — sealed test | 0.556 | — |
| Optimized — sealed test (held-out) | 0.667 | +20.0% relative |
Artifact: examples/skillsbench/run_full/.
The optimizer edited all four SKILL.md bodies and added executable scripts across 7 iterations (4 accepted). Notably it stopped itself on the last two tasks after diagnosing them as broken oracles — rather than overfitting a general skill to pass a broken verifier.
The "money shot" in the dashboard is the git diff panel — you can see exactly what prose and code the optimizer added to each skill, and the append-only JOURNAL.md where each iteration's reasoning meets the framework's objective RESULT line:
SWE-bench — coding-agent prompt (adapter ready, production results pending)
SWE-bench gives agents real GitHub issues across 12 popular Python repositories. The agent must produce a patch that passes fail-to-pass tests without breaking the regression suite. It is the primary standard for measuring coding agent capability — top agents now score 70–80%+ on SWE-bench Verified.
cap-evolve optimizes the coding-agent's prompt.md against a pinned subset of SWE-bench Lite instances. The adapter is fully wired (HuggingFace + Docker harness, binary reward per instance, holdout split). Production results are in progress — and we'd love help running them.
These are within-run improvements on our own splits, with our models and budgets — not leaderboard comparisons. Full methodology, caveats, and committed artifacts →
Why evaluation honesty matters
The cost panel makes the evaluation accounting transparent — every candidate scored, every dollar spent, and the sealed test row that appears exactly once:
When you optimize against a metric, it's easy to accidentally overfit to the data you're measuring. We've all seen papers that look great on val and fall apart on test. We made the following hard to violate by putting the rules in code, not docs:
The test split is sealed in code. RunDir.consume_test() flips a test_used flag and raises TestSealError on any second call. The held-out number is produced exactly once, at finalize.
Acceptance gates on val, with significance. The gate rejects any candidate that doesn't exceed k · SE over the current best on the held-out val split. Noise doesn't count as progress — and the threshold is user-configurable so you can be as conservative as you need.
These rules live in the core, not in editable docs. Every algorithm skill calls the same cap_evolve library. There's no way to configure your way around them.
This matters in practice: on our τ²-bench runs, 5 of 10 iterations were rejected. If we'd accepted every improvement on train, we'd have reported higher numbers that didn't generalize.
The optimizer is whatever agent you already use
cap-evolve is optimizer-agnostic. The "edit proposer" is any coding agent you can invoke from a shell:
# optimizers/registry.yaml
claude-code:
cmd: claude -p "{prompt}" --permission-mode acceptEdits
cwd: "{workdir}"
codex:
cmd: codex exec --sandbox workspace-write "{prompt}"
cwd: "{workdir}"
gemini-cli:
cmd: gemini -p "{prompt}" --approval-mode=yolo
cwd: "{workdir}"
14 backends ship out of the box (Claude Code, Codex, Gemini CLI, Cursor, opencode, IBM Bob, and more), plus a mock backend for CI that edits deterministically without calling any model.
Try it in two minutes — no API key
The toy_calc example uses a deterministic mock agent. No model is called, no credentials needed:
git clone https://github.com/skillberry-ai/cap-evolve.git
cd cap-evolve
python3 -m venv .venv && source .venv/bin/activate
pip install ./core # zero runtime deps — pure stdlib
bash examples/toy_calc/run.sh
The seed prompt scores 0.0 on val. The mock optimizer adds a [CALC] marker. The sealed test scores 1.0. You get a dashboard.html showing the full run — splits, gate decisions, per-iteration diffs, cost breakdown.
This is a toy, but it's not a demo — the same core code runs the τ²-bench and SkillsBench results above. The loop, the gate, the sealing, and the dashboard are all real.
Bring your own benchmark
Wiring your agent takes three methods:
def tasks(split: str) -> list[Task]:
# return your eval cases for 'train'|'val'|'test'|'all'
def run_target(task: Task, ctx: Context, *, seed: int) -> Rollout:
# run your agent with the current candidate as context
def score(task: Task, rollout: Rollout) -> Score:
# reward in [0,1] + feedback — never leak the gold answer
Everything else — splits, trials, pass^k, the gate, the sealed test, memory across iterations, the dashboard — is in the core. You can also copy a ready-made adapter template for JSONL, HuggingFace, tau2-bench, SWE-bench, or SkillsBench and only change the provider.
The fastest path: drop your coding agent into the repo root and point it at RUN.md. It runs intake, writes the adapter, passes cap-evolve check, and starts the loop.
What we learned
A few things genuinely surprised us building this:
Tool edits outperform prompt edits for structural failures. On τ²-bench, the biggest single-iteration gains came from adding deterministic enforcement in tool code, not from rewriting policy prose. A constraint the model can't skip beats a rule it might forget — this maps directly to the class of failures SWE-bench exposes too, where the patch has to make tests pass, not just satisfy a prose description.
Jointly optimizing all capability types outperformed tools-only. On Qwen 14B, optimizing [skill-package, system-prompt, tools] together gave +125% on the held-out test vs +41.2% for tools alone. The optimizer distributed edits across code-level guards, policy clarifications, and SKILL.md structured methodology.
The optimizer knows when to stop. On SkillsBench, after 4 accepted iterations, the optimizer diagnosed the two remaining failing tasks as broken oracles — the verifier expected a value that wasn't derivable from any data the agent had access to. It stopped rather than overfitting. A loop that can distinguish "this task is hard" from "this oracle is wrong" wastes less budget.
Rejections are informative. Rejected iterations aren't wasted — the optimizer's memory carries forward what was tried and why it didn't clear the gate, so later iterations don't repeat the same dead ends.
Agent mode found improvements that deterministic mode missed. On the head-to-head run (same split, same runner), the deterministic hill-climb run proposed candidates that reached val 0.567 but none cleared the gate — its best stayed the seed, sealed test 0.35. The agent-mode run found a gate-accepted improvement and the only positive held-out test. The difference: agent-optimize can adjust its search strategy mid-run rather than following a fixed schedule.
Small task sets make this hard. With 10 val tasks, the significance gate is conservative — some real improvements get rejected as within-noise. That's intentional, but it means you need a reasonably large eval to make the loop useful.
Get involved
cap-evolve is beta (0.x), Apache-2.0, zero runtime deps. Made at IBM / Red Hat.
We're actively looking for:
- ⭐ Stars — if the approach resonates, a GitHub star helps others find it
- 🔍 Code reviews — the core is small and pure-stdlib; honest critique of the gate logic, adapter contract, or algorithm skills is very welcome
- 💬 Comments here — questions about the honesty gate, the two orchestration modes, the tool-vs-prompt tradeoff, how trace diagnosis works, or where this falls over
- 🐛 Issues & PRs — bug reports, adapter templates for new benchmarks, new optimizer backends, or improvements to existing skills — all welcome via GitHub Issues
Links
- Repo: github.com/skillberry-ai/cap-evolve
- Site + docs: skillberry-ai.github.io/cap-evolve
- Results + artifacts: skillberry-ai.github.io/cap-evolve/results.html
- Agent mode: skillberry-ai.github.io/cap-evolve/agent-orchestration.html
- Getting started: skillberry-ai.github.io/cap-evolve/getting-started.html
Top comments (0)