I Changed One Model Pin and Ten Test Results Went Stale
From the agentdrill v0.1.0 release, part of agentic-suite: open-source, harness-agnostic security tools for AI-agent defenses.
One line changed in the config. The agent model pin went from Qwen3.5-4B-4bit to Qwen3-4B-Instruct-2507-4bit. Nineteen test verdicts sat on disk, all green.
I could not tell you which of them were still true.
That is the actual problem with test evidence. It does not expire on its own. Nothing goes grey. A PASS earned under the old model sits next to a PASS earned under the new model and looks exactly as fresh. The dashboard has no concept of provenance unless you build one.
This happened mid-run in the agentdrill v0.1.0 field test, and the resolution is the part worth stealing. Full details in the field test report, Batch 9.
The instinct that would have wasted a day
The obvious move after a model switch is to re-run everything. Nineteen Track B cases, each one a real agent doing real work against a local LLM server. That is an expensive afternoon, and worse, it is indiscriminate. It re-proves the fresh results to avoid finding out which ones are stale.
The house rule for this run was stricter and better: run only what is failing or stale. So the question became a lookup problem. For each of the 19 cases, when was its verdict written, and was that before or after the pin switch?
Git was the ledger we already had
Every verdict in this field test is committed to git as a verdict.json, together with its captured LLM pairs. The pin switch also landed in a specific commit. That turns the question into arithmetic:
- Find the commit where the new pin landed
- For each case, find the last commit that wrote its
verdict.json - Verdict written before the switch: stale. After: fresh
The answer was 9 fresh and 10 stale. Not 11, which is what the report had been saying from a rougher earlier count. The forensics corrected the report, not the other way around. That correction is in the report now, because a number you cannot trace is a number you cannot defend.
Re-earn exactly what aged
The 10 stale cases were re-run. Nothing else. All 10 green, each with fresh request and response pairs. Then the verification step that matters: for every one of the 19 cases, read the model field out of its most recent captured requests and confirm the new pin is there.
That last check is only possible because of a detail from earlier in the run: every captured LLM request carries meta.model, so each evidence file describes the environment that produced it. Nineteen of nineteen cases now show the new agent pin in their latest capture batch. The mixed state is gone, and the proof is not my word, it is readable out of the files under results/track-b/cases.
The failure mode nobody warns you about
The worst part of this story is not the switch. It is that the first switch half-landed. The pin was changed in one place, but the field runner script still had the old model as its default, so early re-runs quietly kept using the old pin while we believed we had switched.
A config change that half-lands is nastier than no change, because your belief and your evidence diverge without any error message. We only caught it because the captures recorded the model name, and someone actually read them. The fix was to wire the pins through run-local.sh so there is exactly one source of truth.
What I take from this
- Evidence has provenance. A verdict is a claim about a moment: this code, this config, this model. The verdict should say so, or the raw evidence should.
- A change event should age your evidence. Nothing automatic did this for us. We built the habit: config change, then ask which verdicts predate it. Git made the question answerable in minutes.
-
Store the environment inside the evidence.
meta.modelin every capture file is what turned "trust me, it is the new pin" into a checkable fact. - Re-run the stale set, not the world. Discriminate re-runs keep the cost of honesty low. If re-proving everything is the only honest option you have, your evidence is not organized well enough to be cheap about.
Tip: a config change should age your evidence. Find which verdicts predate it before you re-run anything.
The question this leaves me with: when you last changed a model, a temperature, or a prompt, did your old greens get flagged, or are they still sitting on the dashboard looking fresh? Mine were. Ten of them. Green, stale, and indistinguishable from the truth until someone went and checked.
How does your team mark results that predate a config change? I want to hear whether anyone has this automated, because we did it with git one-liners and I suspect there is a better machine for it.
Code and receipts: agentdrill · field test report, Batch 9 · captured pairs with model pins
About agentdrill
This story is one of seven from the agentdrill v0.1.0 field test. agentdrill is the adversarial-testing engine in agentic-suite: open-source, harness-agnostic security tools for AI-agent defenses. It runs versioned attack packs against your deployed stack in CI and publishes honest results, misses included.
pip install agentsec-agentdrill # v0.1.0 is live on PyPI
agentdrill quickstart # first honest result in ~15 minutes, bundled demo
Dig deeper: the v0.1.0 field test report (73/73 units, every pass grounded), the release notes, and the rest of the suite at github.com/agentic-suite: agentdrill drills the defenses, agentwatch records the evidence.
Top comments (1)
The half-landed switch is a useful failure case. One way to make this fail earlier would be to store a fingerprint of the effective runtime configuration with each verdict (model identifier, prompt and relevant tool settings), then compare it at execution and when reading results. If the runner resolves a different model from the requested one, fail the run rather than merely marking an old result stale. How would you handle a provider changing the model behind an unchanged alias?