The benchmark that kept inflating its own score

Author: Mohit Kumar

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

I re-ran the exact same benchmark, on the exact same data, three times in a row. The first run said it had scanned 18 model files. The second said 36. A third said 54. Nothing in the repo had changed between them. A number that is physically incapable of moving was moving — and it was moving up by exactly 18 every time.

The worst part: it was the benchmark for a security scanner. The one number I was using to prove the tool was trustworthy was quietly lying to me.

What the tool is (30 seconds of context)

I build Bulwark — a static security scanner for the AI-agent supply chain. One of its tools, Airlock, inspects machine-learning model files before anything loads them, because the most common model format (Python pickle) can execute arbitrary code the instant it's deserialized. Airlock disassembles the pickle instead of loading it.

To prove Airlock isn't just crying wolf, it ships a benchmark: run Airlock and three other open scanners (picklescan, modelscan, fickling) over a corpus of ~19 real Hugging Face models plus a set of hand-crafted evasive payloads, and report how many each flags. The headline number that matters most is the false-positive count: how many of the 18 benign real models does each tool wrongly flag as malware? The answer is supposed to be 0/18 for all of them. Catching attacks is easy if you cry wolf; not crying wolf is the hard part.

That "18" is the number that started drifting.

The false trail (and why it was reasonable)

My first hypothesis was mundane and, honestly, likely: the corpus just has more pickle files than I thought.

The corpus is built from Hugging Face hf-internal-testing/tiny-random-* repos. Those test repos are messy — they contain sub-folders for different framework variants, and each can carry its own pytorch_model.bin. So "19 repos" does not mean "19 pickle files." When the count came back higher than 19, my brain immediately supplied the comforting explanation: right, the sub-variant folders, of course there are dozens of .bin files. I even started to write that down as a fact.

It's a good example of the most dangerous kind of bug: the one whose wrong number has a plausible innocent story sitting right next to it. If Run 2 had said 72 and stayed 72, I'd have shrugged and moved on. The story would have held.

The turn

What broke the story was determinism. I hadn't touched the corpus, the code, or the models between runs — and yet the count grew by exactly 18 each time. "The repos have many sub-variants" explains a count that's higher than 19. It does not explain a count that changes every time you run the same command. Sub-folders don't breed.

Here's the drift, captured live — the same command, three times, nothing else touched:

$ for i in 1 2 3; do python packages/airlock/scripts/benchmark.py datasets/corpus.txt \
    | grep -oE "real-models \([0-9]+ artifacts\)"; done
  run 1: real-models (18 artifacts)
  run 2: real-models (36 artifacts)
  run 3: real-models (54 artifacts)

So I stopped trusting the aggregate and looked at the actual files on disk:

$ find datasets/corpus -type d -name "_al_*" | wc -l
54

54 directories I never created, with a name I did recognize — _al_ was a prefix from my own benchmark code. And they were nested inside each other:

datasets/corpus/tiny-random-BertModel/
  pytorch_model.bin
  _al_pytorch_model/
    pytorch_model.bin
    _al_pytorch_model/
      pytorch_model.bin
      _al_pytorch_model/ ...

The benchmark was writing into the very corpus it was measuring.

The root cause

To judge each scanner fairly, the benchmark scans one file at a time in isolation. My helper "isolated" a file the laziest possible way — it copied it into a fresh sub-directory next to it, then pointed the scanner at that directory:

# BEFORE — the bug
def airlock_flags_exec(engine, path):
    sub = path.parent / f"_al_{path.stem}"     # a new dir *inside the corpus*
    sub.mkdir(exist_ok=True)
    (sub / path.name).write_bytes(path.read_bytes())   # a copy of the artifact
    result = ModelScanner(engine).scan(str(sub))
    return any(f.category == "M1" for f in result.findings)

path.parent is inside datasets/corpus/. So every run left a copy of every pickle inside the corpus. The next run's corpus walk — a plain rglob("*") — dutifully discovered those copies as new "model files," scanned them, and copied them again, one directory deeper. A benchmark that reads a directory tree and writes into that same tree is a feedback loop. The count didn't represent the corpus; it represented how many times I'd run the benchmark.

The security irony wrote itself. This is a tool whose entire philosophy is never trust your inputs, never let a file you're inspecting change your state. Airlock is fanatical about it — it disassembles pickles instead of loading them precisely so a hostile file can't run code during a scan. And then its own benchmark harness cheerfully mutated the dataset it was inspecting.

The fix (and the hacks I didn't ship)

The obvious patches were all wrong:

• Add _al_* to .gitignore. Hides the mess from git; the files still exist on disk and still get re-scanned. Cosmetic.
• Delete the copies after each run. Now correctness depends on cleanup always running — one crash mid-benchmark and the pollution is back. Fragile.
• Skip _al_* in the corpus walk. A guard clause that treats the symptom, leaving the harness still writing into its own corpus for the next person to trip over.

The real fix was to delete the reason the copies existed. The scanner's loader already accepts a single file — I never needed a directory at all:

# AFTER — scan in place, write nothing
def _airlock_scanner():
    """Airlock scans the file in place (the loader accepts a single file), writing nothing."""
    engine = RuleEngine(load_rules())
    def scan(path):
        result = ModelScanner(engine).scan(str(path))
        return any(f.category == "M1" for f in result.findings)
    return scan

No temp dir, no copy, no write. The harness became a pure reader of its corpus. I also made the corpus walk skip Hugging Face's .cache directory, so only real artifacts are ever counted. Then I deleted the 108 stowaway directories and the count snapped back to a stable 18 and stayed there.

The numbers — and an honest caveat

The count climbed 18 → 36 → 54 across three identical runs — exactly one extra copy per real model each time — then dropped to 18 and stayed there after the fix. On disk after those three runs: 54 nested _al_ directories the harness had created. (That's the clean, controlled reproduction above; the very first time this bit me it had quietly accumulated well past 100 before I noticed the number was even moving — which is the whole point: nobody stares at a "0 false positives" line waiting for its denominator to grow.)

Honesty note, because it matters more than a bigger number would: the log above is captured live from this repo — I reverted the fix, ran the benchmark three times, and pasted exactly what it printed. The incident is written up in docs/DATASETS_AND_TESTING.md §8 so it stays reproducible.

What actually got hurt: the per-model percentages in my published study (100% of models ship pickle, etc.) were fine — those are booleans per repo, immune to duplicate files. But the raw false-positive count — the credibility metric of a security scanner — was silently inflating from 0/18 toward 0/36, 0/54, and up. The number was still "0 false positives," so nothing screamed. It was wrong in the one dimension a reader would have quoted.

The lesson

Not "always validate input." The specific, transferable one:

A test harness that writes into the dataset it reads is a bug, even when every test passes. Isolation-by-copy feels harmless until the copy lands somewhere the next run will discover. If a harness must create scratch files, they belong in a temp dir outside the fixture tree — never beside the fixtures.

And the tell that saved me: a count that shouldn't move but does is a symptom, not noise. The instinct is to round it away — "eh, it's a test repo, whatever, the important percentages are stable." Chasing the drift instead of explaining it away is the whole job. The plausible innocent story is exactly where the bug hides.

---

Airlock and the full benchmark are open source: github.com/mk12002/Bulwark. The fix and the incident write-up live in packages/airlock/scripts/benchmark.py and docs/DATASETS_AND_TESTING.md.