I wanted to write an agent skill that makes coding agents better at data analysis. The kind of mistakes I had in
mind are the ones that produce a confident, wrong number: a "Total" row summed together with the countries, a join
that duplicates orders, averaged p99s, timed-out requests that vanish from the latency percentiles, an A/B test that
was supposed to split 50/50 and came back 50.7/49.3.
Every rule in the skill comes from a paper or official documentation. But before claiming it helps, I wanted to
measure it.
The benchmark
I built 30 realistic cases, each with one planted trap and a correct answer computed by code:
- 19 data-analysis traps (subtotal rows, join fan-out, average of averages,
NAparsed as null when it's Namibia, partial weeks, sample ratio mismatch, segment fishing, Simpson's paradox, …) - 8 performance and reliability traps (closed-loop load tests, a saturated load generator, retries counted as requests, tail-sampled traces, regression to the mean after a "perf sprint", …)
- 3 tasks where the agent measures its own work (fix a flaky test and confirm it, report a speedup, measure logging overhead)
Each case ran with and without the skill on Claude Haiku 5.5 and Sonnet 5.5, graded by an LLM judge (Sonnet,
3 votes per grader).
The result
| With skill | Without skill | |
|---|---|---|
| Haiku 5.5 | 61/61 | 58/61 |
| Sonnet 5.5 | 61/61 | 60/61 |
Today's Claude models already avoid nearly all of these traps, including the harder performance ones. The skill
fixed every baseline miss (-1 timeout sentinels, which both models missed, plus a closed-loop load test hiding a
stall and an overhead number reported without uncertainty on Haiku) and introduced no new failures. But with one
run per arm, that's "catches occasional slips", not a large effect. That's what the README says.
What I learned
1. Skills help most where the model's knowledge is weakest. The same week I wrote a skill for upgrading to
TypeScript 7, which shipped after the models' training cutoff. There the gains were real: graders passed went from
10/18 to 17/18 on Sonnet and 7/18 to 15/18 on Haiku, on small samples with offline agents. Mainstream analysis
practice is already in the model. Post-cutoff releases and exact operational details are not.
2. Check your test cases before you trust a baseline failure. Across both projects, several of the
"failures" I first saw were flaws in my own cases: an effect I hadn't intended in the data generator, a grader
that was too strict, a premise that contradicted itself. Once, Haiku correctly pointed out that a case's premise
was wrong.
3. A skill that doesn't activate doesn't help. In the benchmark the skill activated by itself in 22/30 Haiku
runs. Its description listed data types and trap names. The prompts it missed asked for an action: "should we
roll back?", "confirm it's fixed", "measure the overhead". I rewrote the description to say when to load it ("at
the start of any task whose answer will include a number … before reading files, running benchmarks or editing
code"). In a short trigger test on Haiku (first 3 turns, 2 rounds), activation on 10 queries the description
was never tuned on went from 7/20 to 20/20, with no activations on 10 unrelated queries.
Try it
The benchmark runs against any agent, with or without the skill. If you run it on another model or agent, I'd
like to see the results: https://github.com/boranfurkan/trustworthy-data-analysis
Top comments (1)
Your honesty about the small effect size is the most useful part here, because it reframes skills as insurance against occasional slips rather than a guaranteed lift. The activation finding also rings true: a skill that never triggers is just documentation, so writing the description around when to load it, not just what it does, seems like the highest-leverage edit. One caveat I would add is to keep the benchmark's traps in version control alongside the skill itself, so a later model update can be re-checked without rebuilding the cases from scratch. For readers who want to compare how other skills are structured, a Chinese-language collection of Claude Code skills at github.com/laolaoshiren/claude-cod... is a useful reference for seeing different trigger descriptions and layouts side by side. It helped me think about which of my own skills deserve a benchmark and which are simply conveniences that should stay lightweight.