I ran a controlled pilot to test whether one procedural agent skill improved an
exact, checkable task.
The baseline passed 15 out of 15 attempts.
The treatment passed 15 out of 15 attempts.
That is not evidence that the skill was useless. It is not evidence that the
two approaches are equivalent, either. It is evidence that I chose a benchmark
with no visible failure surface left for the treatment to improve.
The ceiling effect became the most useful result.
How an Essay Turned Into an Evaluation
This started as a much broader claim: reusable agent skills could replace
memory while improving quality and reducing time and cost.
Three isolated agent reviewers broke that argument. Two separately called
"skills replace memory" a category error. All three pointed to costs I had
omitted: creating, testing, loading, selecting, updating, and retiring the
skill.
They were critiques, not scientific evidence. So I narrowed the claim and built
a small comparison.
Here, a skill is a reusable package of instructions, constraints, checks,
and optional scripts that an agent can load for a recurring class of work. The
pilot asked a deliberately limited question:
Does making one frozen procedural skill available improve exact contract
conformance on unseen synthetic Markdown public-release tasks?
What I Tested
I froze the protocol, runner, skill, fixtures, order, and analysis rule before
the scored batch. A separate, context-isolated adversarial agent review had to
return GO before the one-shot runner could start.
The design was:
- three unseen synthetic fixtures;
- baseline and treatment arms;
- five attempts per fixture and arm, 30 attempts total;
- fresh container and fresh agent session for every attempt;
- no retries, replacement attempts, or early stopping; and
- one exact binary pass defined as the logical AND of eight checks.
Those checks covered the artifact contract, source integrity, visible and
hidden sensitive-value removal, secret and path removal, preservation of safe
content, byte-exact Markdown, and byte-exact findings JSON.
Both arms received the same generic instruction to identify and read applicable
skills before doing the task. Only the treatment had the frozen sanitizer skill
available. That means this tested encouraged skill availability, not whether
an agent would naturally discover a skill with no cue.
All inputs used reserved synthetic examples. This was not a test with real PII,
and the result should not be read as evidence of real-world PII safety.
The Result: A Perfect Tie
| Arm | Passes | Wilson 95% interval |
|---|---|---|
| Baseline | 15/15 | [0.796117, 1.000000] |
| Treatment | 15/15 | [0.796117, 1.000000] |
The treatment-minus-baseline difference was 0.000000. Its Newcombe 95%
interval was [-0.203883, 0.203883].
Those intervals summarize repeated runs on only three fixed fixtures, with five
runs per fixture and arm. They do not estimate how either approach would
generalize across the broader population of public-release tasks.
The preregistered decision was no clear difference. The interval is an
important antidote to the perfect-looking scores: with only 15 attempts per
arm, the data are still compatible with practically meaningful benefit or
harm. A 15/15 tie does not establish equivalence.
Every case showed the same vector:
| Case | Baseline | Treatment |
|---|---|---|
| E | [1, 1, 1, 1, 1] | [1, 1, 1, 1, 1] |
| F | [1, 1, 1, 1, 1] | [1, 1, 1, 1, 1] |
| G | [1, 1, 1, 1, 1] | [1, 1, 1, 1, 1] |
All 30 trials were healthy. There were no diagnostic failures,
infrastructure errors, retries, or replacement attempts.
The Score Hid a Process Difference
The treatment skill was catalogued and completely read in all 15 treatment
attempts. It was completely read before the first path-explicit task action in
14 of 15.
The one attempt without measured pre-action uptake stayed in the treatment arm,
as preregistered, and still passed.
That separates three questions I had been treating as one:
- Was the procedure available?
- Did the agent use it in the intended order?
- Did it improve the outcome?
The pilot measured the first two. The treatment was available, discovered, and
usually read before task work. It did not answer the third because the baseline
already passed every case.
In other words, the evaluation instrumentation worked while the benchmark
failed to discriminate.
The Procedure Also Had Observable Overhead
The treatment runs used more tokens and had a higher harness cost estimate.
These are descriptive observations from this pilot, not a causal lifecycle-cost
claim.
| Arm | Trial seconds min / median / mean / max | Input / cache / output tokens | Harness cost estimate |
|---|---|---|---|
| Baseline | 49.633 / 69.514 / 161.409 / 984.219 | 1,262,148 / 1,029,888 / 37,437 | $0.41990310 |
| Treatment | 61.637 / 82.117 / 181.689 / 1,066.909 | 1,824,478 / 1,552,896 / 44,863 | $0.52203720 |
The cost values are harness estimates, not account charges. The broad timing
range also makes a tiny runtime comparison unhelpful. I did not measure the
human time spent designing, reviewing, and maintaining the skill, so this
cannot support a break-even claim.
It does show why "the skill was followed" is insufficient. Procedural overhead
is observable; its outcome value remained unmeasured.
What I Would Change in the Next Evaluation
The next comparison needs a meaningful but honest failure surface.
- Run a difficulty sweep before freezing the main study. The target should not be a benchmark the baseline already solves perfectly.
- Keep uptake and outcome separate. Skill discovery, complete reading, pre-action use, and task success answer different questions.
- Predefine a meaningful effect. A larger study should say what improvement would justify the added procedure and overhead.
- Test boundary cases, not only clean transformations. Ambiguity, conflicting constraints, irrelevant skills, and refusal cases may expose differences without inventing arbitrary traps.
- Match the claim to the evidence. Real safety claims require governed real-world evidence; synthetic sentinel removal is not a substitute.
What Survived From the Original Argument
I still find skills valuable, but the useful claim is narrower:
Skills externalize stable procedures. They do not replace episodic or project
memory, and their value has to be demonstrated against their maintenance and
execution cost.
A repeated instruction is not automatically a skill. Sometimes it should
remain a prompt, project documentation, a template, or deterministic code.
My working decision rule is still:
Formalize a workflow when the expected cost of rediscovering it and repeating
its failures exceeds the cost of creating, testing, loading, and maintaining
the skill.
The pilot did not validate that rule. It exposed what a future evaluation must
measure before I can claim that a skill has earned its cost.
Disclosure: I used ChatGPT Codex to challenge the original claim, design and
run the synthetic pilot, audit the evidence, and help revise this article. The
final publication decision is mine.
When a procedural intervention is adopted by the model but the baseline
already passes every case, how would you redesign the next evaluation to create
a meaningful failure surface without making the benchmark artificially
adversarial?
Top comments (1)
The cleanest way I have found to create a failure surface without resorting to synthetic trick prompts is constraint density.
Single-objective tasks (strip secrets from markdown) are easy for modern models to solve in zero-shot. Failures happen when two orthogonal invariants fight each other. For example, stripping internal hostnames and auth headers while strictly preserving raw code block indentation, or redacting tokens inside tabular markdown without breaking column alignment.
Baseline models usually satisfy the primary cleanup rule and quietly corrupt the layout or over-redact valid public paths. That is where a structured skill with explicit pre-checks and diff verification actually earns its token overhead.