We open-sourced the tool that failed its own kill rule

Before we ran the experiment, we wrote down the rule that would kill it.

The product under test was a pipeline that builds skill files for AI agents: those structured instruction documents, SKILL.md and friends, that tell an agent when to trigger, what to do, and what never to do. Our hypothesis was the obvious one. Skills built through a structured pipeline, with a requirements interview, controlled generation, and a mechanical audit, should beat skills the model improvises for itself in one shot. We pre-registered four predictions and one kill criterion: if the pipeline could not measurably beat one-shot generation, we would say so in public and change the product.

Then we ran it. One-shot generation beat the pipeline 98.3% to 91.7% on trigger recall, 118 of 120 against 110 of 120, with overlapping 95% Wilson intervals, [94.1, 99.5] against [85.3, 95.4]. By the rule we had committed to disk before seeing a single result, the hypothesis was dead.

Today we are open-sourcing the tool anyway: Skill Forge, part of the wallaby-agent-rules repo, under MIT + Commons Clause.

That sentence needs defending, and the defense is the interesting part. The kill rule tested one claim, that structured generation beats improvisation at the routing layer, and that claim died. But the experiment program around it produced a pile of measured findings about how skill files actually fail, and every one of those findings is now load-bearing in the tool. What survived the falsification is narrower than what we set out to build, and considerably more honest. This post is the full account: what died, what survived, and what the surviving thing does.

What died, and what the corpse taught us

The full experiment write-up is on our blog (the falsification report, with the charts and the pre-registration text). The short version, because the details matter for what we shipped:

Round P3 was a routing test on the weak model we use for volume work, DeepSeek V4.1-Flash: ten task needs, twelve trigger questions each, two trials, both arms, 480 routing judgments, on-screen order shuffled by seed. The forge arm ran the full pipeline; the one-shot arm just asked the model to write the skill file directly. The pipeline lost, and every one of its misses failed the same way: the model declined to trigger at all. It never picked the wrong skill. Something in our generated files was telling the model to stand down.

That something turned out to be the description field. Our generator had been writing boundary clauses into it, things like "does not trigger for X" and "only when Y", the kind of negative scoping that reads like good hygiene to a human. To a small model deciding relevance, those clauses were stop signs. We patched the generator to keep boundary sentences out of descriptions entirely, and misses from that failure mode went from 10 to zero on the re-run.

Round P4 moved to harder ground: eight needs from one real project domain, where every skill has near-neighbors and the trap questions sound like they belong to two skills at once. The kill rule fired a second time, but the gap narrowed from 6.6 points to 2.7, and the failure mode changed shape. The model now picked the wrong neighbor instead of refusing to fire. One of our internal ops skills pulled both arms in like a magnet; both fell for its trap question twice out of two. The hand-tuned champion version of that skill answered it perfectly, because its description carried a routing verdict naming exactly that confusion. No generator saw its neighbors, so no generator could write that sentence. Now the pipeline asks, at generation time, which sibling skills yours must not be confused with.

Two things survived both rounds. First, the mechanical audit, the boring structural health check, caught the one-shot arm inventing a pitfall that never happened and quoting a price that does not exist. The pipeline arm produced not a single red flag or invented detail in any generated file. Discipline was doing something; just not the something we had predicted.

Second, and this became the finding our harshest external reviewer called the most valuable insight in the document: every forge skill ended with a maintenance instruction, if something went wrong during the run, record it to the failure ledger, sitting at the bottom of the file as a footnote. Across twenty executions on two models, it did not fire once. The fix was not better wording; we tried standalone sections and single-line notes with no difference. The fix was moving the instruction into the numbered task steps, as the closing step of the run. Same sentence, new position: the weak model's feedback-logging score went from 0.00 to 1.80 out of 2.

A third pre-registration failed too, and it deserves naming. A later round tested the "ideal form" hypothesis: a task class where both models start mediocre and the forge lifts both. It missed three of its four predictions. The tool you can download today carries the scars of all three rounds, not just the wins.

So what is the tool, after the funeral?

We rewrote the product's claim to match the evidence. Skill Forge does not make models equally capable; it makes the quality requirements consistent across models: the same triggers, red lines, acceptance checks, and failure records on every executor. Its center of gravity moved from authoring to lifecycle, meaning the health check, the failure ledger, and the controlled revision.

In practice, Skill Forge is three things:

1. A derivation protocol. An interview skeleton that extracts requirements from what the user already said instead of interrogating them. Every answer lands in one of three buckets: user-confirmed, safely derived, or open-and-critical. Only the third bucket is allowed to become a question. A high-stakes checklist forces explicit user confirmation for anything touching production deploys, deletions, money, credentials, or public posting.
2. A generation spec with a birth certificate. The generated skill ships with trigger test pairs, sentences that should fire it and sentences that should not, as a regression baseline you can re-run after any edit.
3. A maintenance loop. A failure ledger the skill itself writes to, thanks to the embedded closing step; a seven-item health check; and an optimize pass with four verdicts per sentence: cut, sink to a reference file, demote to read-on-demand, or keep.

Every one of these exists because a measurement put it there. That is the deep part, so here are the five design decisions that cost us the most to learn.

Five decisions, five receipts

1. The description carries only positive trigger conditions

The P3 collapse is why the generator now treats the frontmatter description as restricted territory: positive trigger phrasings only, written the way a user would actually say them. Input-shape restrictions like "plain text only" or "only when X" are banned outright, because routers execute them as do-not-trigger rules and recall falls off a cliff.

Boundaries are not banned; they are relocated. Instead of abstract negative scoping, a description may carry neighbor verdict lines: concrete sentences that name the specific confusable scene, such as "when the user only wants X and Y is not involved, do not trigger this skill". The P4 magnet is why. An abstract boundary is invisible at routing time, but a named verdict matches the trap question word for word. In our re-runs, named verdicts prevented the misfires that abstract boundaries could not.

2. Failure logging is the last numbered step, never a footnote

The zero-for-twenty finding is now structural. In every skill the pipeline generates, the failure-recording instruction is the final numbered step of the task flow. Not a trailing note, not its own section, not a "remember to" at the bottom. A step, with a number, in the work.

This generalizes past our tool, and it is probably the single most portable insight in the release: an agent executes the task steps, not the footnotes. If your AGENTS.md or your system prompt ends with "remember to update the changelog", you have written a footnote, and footnotes score zero. Move it into the work.

One nuance from the strong model: it sometimes logged failures with no instruction at all, 1.33 out of 2 unprompted, and the embedded step did not reliably lift it further on our small sample. The step exists for the models that need it; it costs the models that don't almost nothing.

3. Red lines are concrete, checkable, and never blanket

Two wording rules, both measured on the hard-task set. Every red line must be concrete and checkable: "never guess amount fields", not "never make mistakes". And blanket shutdown clauses are banned. "If conditions are not fully met, stop entirely" reads like prudence to a human, but strong models execute it as righteous shutdown, refusing tasks they could have done and skipping parts they could have completed.

That second rule comes from the program's second strand (the three-regimes map): the same scaffolding that lifted a weaker model's execution quality from 0.88 to 2.0 across four graded dimensions pushed a strong model into overcautious shutdowns on hard, fuzzy-boundary tasks, four dimensions, all down. Structure helps the weak and can hobble the strong. So the pipeline routes on both axes: model strength and boundary fuzziness decide whether your skill ships as a full scaffold or a thin shell.

4. Size is a hard budget, measured in bytes

The coda of the falsification round became its own law (the 6KB line): a naturally bloated 9.2KB skill file drove the weak model to empty responses six times in a row. Isolating size as a variable produced a clean gradient: zero burn-throughs in eight runs under 6KB, roughly 25% empty responses at 6–9KB, roughly 50% at 9KB and up for naturally bloated files.

So the tool treats 6,144 bytes as the safe zone and enforces it byte-exact. When a skill's body projects past the line, the spec routes it to a layered form: the body keeps only high-frequency rules plus a scene routing table, and scene detail sinks into references/ files the agent reads on demand. When the executor is a strong model facing fuzzy boundaries, it routes to shell form instead: red lines, criteria, and pointer tables only, with no duplicated step detail. Ritual content, meaning pitfall placeholders, feedback hooks, and changelogs, gets a hard budget of three lines, because every line of ceremony is a line of instruction the model is not reading.

5. The audit is mechanical, and the ledger has an idle alarm

The health check is deliberately boring. Is the description phrased as triggers? Is the size in the safe zone? Any hard-coded versions, dates, or prices? Full paths in commands? Red lines present when the content touches shell, network, files, or money? Is the pitfalls section real? Is the ritual budget held? Boring is the point: this is the check that caught the one-shot arm inventing a pitfall and a price.

The failure ledger appends one JSON line per failure: date, scene, what broke, with redaction rules for secrets. And because the zero-for-twenty finding taught us that silence usually means the hook is not running, the ledger has an idle rule. History but nothing new in 14 days is a signal, not a comfort: either nothing failed, which is possible, or nobody is recording, which is more likely, and the optimize pass tells you which to check first.

Three forms, one protocol

We distribute the same protocol three ways, because the audience for skill governance ranges from people who run MCP servers to people who just want to paste something into a chat.

The MCP server lives in extras/skill-forge/mcp/: zero dependencies, Python 3 standard library only, stdio transport. Three tools: skill_forge for creation, skill_optimize for the health check plus rewrite, and skill_feedback for the ledger write, called by the agent when a skill fails. It automates everything mechanical: the stakes scan, ledger writes with redaction, idle warnings, byte-exact size metering, the seven-item report.

The prompt pack lives in extras/skill-forge/prompt-pack/: the identical protocol as Markdown files you paste into any agent conversation. The five automations become manual checklists. No install, no server, works everywhere.

The skill form lives in extras/skill-forge/skill-form/: the protocol packaged as a SKILL.md itself, for Hermes and any tool that reads the convention. It is generated from the prompt pack by a sync script at release time, so there is one source of truth and no drift between forms.

The MCP quickstart is a three-line entry in any stdio-speaking client:

{
  "mcpServers": {
    "skill-forge": {
      "command": "python3",
      "args": ["/absolute/path/to/mcp/wallaby_skill_forge_mcp.py"]
    }
  }
}

Then verify your copy before trusting it. The repo ships a 78-assertion smoke suite that boots a live server instance against a temporary HOME and checks the protocol face, both forge stages, the router, the audit, and the ledger:

python3 mcp/smoke_test.py

Honest limits

The same discipline that killed the original claim applies to what we ship:

• Sample sizes are small, ten to twenty per arm in most rounds. We publish Wilson intervals with the numbers and treat near-ties as noise. The P3 loss was outside the intervals; several secondary findings are inside them, and we say so where they are.
• One model family did most of the work, with a judge model related to the weak executor. We ran later behavior checks through other models, including a paid pass where Claude reviewed the forge spec and sent back a test harness, but the core measurements describe one family's behavior.
• Generation and rewrite semantics still depend on the executing model. A weak model may not honor a long spec, and no scaffolding removes that. The tool raises the consistency of the requirements; the ceiling stays where the model put it.
• The entire experiment program behind the three blog posts cost ¥10.6 in API fees, about $1.50. That is the honest scale of the evidence: real, measured, and small.

Why open, and why now

We build agent infrastructure in a specific order: hit a real problem running our own agents, build the fix, run it in production, publish what happened, then productize. The memory side of that loop shipped earlier as wallaby-agent-rules, the plain-file memory system with receipts on every fact. Skill Forge is the same governance idea one layer up. The memory system governs what your agent remembers; this governs what it does. Both are the rulebooks we run on, published as they are, failure logs included.

The license is MIT + Commons Clause: free for personal projects, research, and use inside your own organization; selling it as a hosted or bundled product needs a commercial license; COMMERCIAL.md in the repo has the terms.

If you build skills for agents, or you have been bitten by one that kept failing the same way, take it for a run. The prompt pack costs you nothing but a paste. File issues with your own failure modes; the whole tool is built from ours, and the ledger format is designed to swallow new ones.

No claims about what should work. Only what we measured.