DEV Community: SmartWord World

My CI check passed every My CI check passed every build for four days. The JSON said {}

SmartWord World — Thu, 24 Sep 2026 17:04:38 +0000

This week I found out the GitHub Action for my open-source tool never failed a single build.

The tool is DocsWatcher. It scans code for API calls that already have a shutdown date (OpenAI models, Stripe API versions, that kind of thing) and fails CI when it finds one. So a check that never fails is the one bug it cannot have.

How the Action decides

The CLI is Java, compiled to a native binary with GraalVM. The Action runs it, reads the JSON report, counts the findings whose severity is breaking, and fails the step if that count isn't zero.

That design was deliberate. The CLI exits 1 when a breaking finding exists, and the Action treats exit 1 as "a result, not a failure". It wants the count from the report, not a bare exit code.

What the binary actually printed

For every finding, this:

[
  {},
  {},
  {},
  {}
]

Four findings, no fields. The count of breaking findings was always zero, so the step always passed.

Why

GraalVM builds a closed world. Anything reached only by reflection is dropped unless it is registered. Jackson writes a Java record by reading its accessor methods reflectively, and my Finding record was never registered. So the native binary serialised every finding as an empty object. No exception, no warning.

The fix is a few lines of reachability metadata:

{
  "type": "dev.docswatcher.engine.Finding",
  "methods": [
    { "name": "severity", "parameterTypes": [] },
    { "name": "change", "parameterTypes": [] }
  ]
}

(The real entry lists every accessor.)

Why the tests missed it

The unit tests run on the JVM, where reflection just works. They were green the whole time.
The release smoke test did run the native binary. It checked that a repository with a breaking finding exits 1. It did. The exit code was right all along. Only the JSON was empty.

What changed

The release smoke test now checks the JSON itself, on each platform's own runner:

printf '%s\n' "$json" | grep -q '"severity": "breaking"' \
  || { echo "::error::the findings carry no severity"; exit 1; }

The Action refuses a report whose findings have no severity, instead of counting it as clean. A broken binary now fails loudly.
The Action's v0 tag points at the fixed release, so existing users get the fix without changing anything.

Looking for the same class of bug found a second one: the native binary crashed on any repository where a finding had more than one location, because an array type (Evidence[]) wasn't registered either.

The lesson

Test the artifact you ship, not the build you test. My tests proved the JVM build was right. Nobody had checked the bytes users actually downloaded.

A question for anyone shipping GraalVM native images or other AOT builds: how do you catch missing reflection metadata before your users do? The tracing agent helps, but it only sees the paths your tests exercise.

DocsWatcher is open source: github.com/jameskomo/docswatcher. The fix is in v0.3.0 and later.

Anthropic shipped an official eval runner for Claude Code. I made my tool run on it the same day. Field notes.

SmartWord World — Sat, 12 Sep 2026 05:38:20 +0000

This week Claude Code 2.1.269 made claude plugin eval public: an official runner for plugin eval suites, with generated starter cases, a no-plugin ablation arm by default, an HTML report and JSON output. I maintain config-drift-checker, a tool that had shipped its own compatible runner while the official one was gated. So release day was migration day, and these are the notes.

The gotchas

1. The frontmatter whitelist rejects, it doesn't warn. My cases carried a custom covers: key for coverage tracking. Under my own lenient parser, fine. Under the official runner, every case carrying it failed to load with "unknown frontmatter key". If you attach custom metadata to cases, put it in a sidecar file next to prompt.md; the runner ignores files it doesn't know.

2. The YAML parser is strict. description: Proves the skill shapes generated Java: envelope, injection dies on the second colon. Two of my six cases failed this way. Quote any description containing ": ".

3. --case matches the name, which defaults to the directory. My case lived in evals/spring-work-triggers-skill/ with a pretty name: Spring work triggers the conventions skill, and filtering by the directory matched nothing. The real fix (credit to a commenter below) is better than any workaround: set no name: at all. It defaults to the folder name, so the filter and the report keys align with your directories. My suite dropped its explicit names the same day.

4. The transcript is ephemeral, but grader verdicts survive it. Tool calls and the full response go to a temp trace file, deleted when the command exits. The JSON you keep has scores, turns, cost, and (another correction from the comments) the per-run verdicts of the cheap graders with their explanations: regex, tool_used and file_exists are computed from the transcript before it vanishes, and lines like "Skill called 1x" persist forever. What you lose is the raw material for anything you didn't write a grader for. If you need that too, run with --keep-temp and copy the trace out before you exit.

What I deleted, and what I didn't

The migration was also a strategy decision. The official runner made my runner a fallback for old versions, and that's fine: the runner was never the product. What the official command deliberately doesn't do (the docs tell you to pin your model so results stay comparable) is remember anything. No stored baseline. No history. No answer to "did this get worse since last month", "is this case just flaky", or "which Claude Code release broke it".

So that's the product: same cases, official format, official runner underneath, and on top a pinned baseline, history across every release, per-case noise bands learned from each case's own past (one of mine naturally swings 0.75, a fixed threshold would either alarm daily or never), a canary track that runs when a Claude Code release actually ships, and a bump PR once a new model proves itself green twice.

Their command answers "does my plugin work right now on my machine". The layer on top answers "did anything stop working since the baseline, across every release, without me watching".

v0.5.0 with all of the above shipped the same day their release did: https://github.com/jameskomo/config-drift-checker/releases/tag/v0.5.0

The repo, with a full comparison table of built-in vs added: https://github.com/jameskomo/config-drift-checker

If you're building your own layer on their JSON instead, I'd genuinely like to compare notes; the noise-band problem in particular is deeper than it looks.

Your CLAUDE.md has zero tests. Here's what happened when I broke mine on purpose.

SmartWord World — Tue, 08 Sep 2026 10:03:18 +0000

Claude Code shipped roughly 25 releases last month. My setup, 400 lines of CLAUDE.md, four skills and a guard hook, had exactly zero tests against any of them.

That bothered me, because this configuration has behavior, and behavior breaks. A new model version ships and suddenly a skill stops triggering. Nothing errors. Nothing goes red. The agent just quietly stops doing the thing you taught it, and you notice three weeks later when the code review comes back weird.

So I built CI for it: config-drift-checker turns your CLAUDE.md, skills and hooks into eval cases and re-runs them on every Claude Code release and every PR that touches the setup.

But a tester you've never seen fail is worthless. So I sabotaged my own setup to test the tester.

Attempt one: the sabotage that did nothing

I deleted the skills entry from my plugin manifest, expecting everything to break.

Nothing broke.

Claude Code auto-discovers the skills directory, so the manifest key does nothing. My first sabotage was a no-op, which is exactly the kind of thing you only learn by trying to break your own system.

Attempt two: the realistic break

This time I rewrote one skill's trigger description the way a careless PR would. The skill still existed, still had all its content, but its description now talked about Terraform instead of Spring services.

	Before	After sabotage
Suite score	1.00	0.36
Tripwire case (did the skill fire?)	1.00	0.00
Content cases	1.00	0.33 to 0.60

The content cases dropped because the agent no longer followed conventions it used to follow. And one case went to exactly zero: the tripwire.

The tripwire

A tripwire case has a single grader:

Was the skill actually invoked?

Not "did the output look right". Just "did the trigger fire". When a trigger breaks, that case reads 0 out of 3 runs, every run, every day, and no amount of model randomness produces that pattern.

The hard part is noise, not scoring

One failed run doesn't mean your rule broke. Models are stochastic; sometimes they just ignore an instruction once. Telling "rule stopped firing" apart from "model ignored it run" turned out to be the real engineering problem.
Three things made the signal trustworthy:
1. Three runs per case. A genuine flake usually recovers within the same three runs. A broken trigger never does.

Noise bands learned from history. Each case's expected spread comes from its own past runs. One of my cases naturally swings by 0.75, so a fixed threshold would either alarm daily or catch nothing. A drop inside the band gets an amber "noisy" flag, never a red check.
Guards so real breaks can't hide in the band. If no current run reaches the baseline, or the case has been below the bar for consecutive runs, it escalates to red anyway. Five failures in a row is not noise.

Likely refusals (zero tool calls, one turn, short reply) get labeled separately, so they don't masquerade as broken rules.

See the failure yourself

The whole broken run is published, unedited:

The red report from the sabotage run

It runs as a GitHub Action on your runner with your key:

- uses: jameskomo/config-drift-checker/action@v0
  with:
    plugin-dir: .

Costs $0 on a Claude Pro or Max plan via a subscription token.

Repo: https://github.com/jameskomo/config-drift-checker

If you've built your own version of this in-house (I keep meeting people who have), I'd genuinely like to compare notes, especially on how you separate "rule stopped firing" from "model ignored it this run".