Your model changed overnight and nobody told you. I built a CLI that proves it.

Last week a vendor shipped a new flagship model with record benchmark scores -- and in the same week, users started reporting that the previous model had quietly changed how it thinks. No changelog. No version bump you could see. Just different answers to the same prompts.

Vendor benchmarks are the vendor's own claims. I wanted something else: a way to pin the exact model version I tested against, re-run my own prompts, and get a yes-or-no answer in CI -- did the outputs change, and where?

So I built modeldrift: a zero-dependency TypeScript CLI that diffs two recorded runs of the same prompts and returns a CI verdict.

GitHub: https://github.com/techdevfun/modeldrift

What it does

Four commands, no API key, no server, no YAML:

modeldrift init            # scaffold a 12-prompt suite + pin file
modeldrift pin --model gemini-3.5-flash --run baseline.jsonl
modeldrift diff baseline.jsonl current.jsonl
modeldrift check --pin modeldrift.pin.json --run current.jsonl

diff compares every prompt and reports a verdict per prompt: IDENTICAL, CHANGED, REFUSAL-FLIP, LENGTH-DRIFT, STRUCTURE-FLIP, NEW, REMOVED. The run verdict is STABLE or DRIFT DETECTED, with exit codes made for CI: 0 stable, 1 drift detected, 2 bad input. check fails fast if the model in the run does not match the pin -- "model changed silently -- expected X, got Y".

All signals are deterministic. No LLM judge, no live provider calls. The tool never calls an API; it only reads run files you recorded yourself (JSONL, one line per prompt).

I ran it on real runs -- and the control surprised me

I recorded the same 12 prompts twice through the Gemini API at temperature 0 with a fixed seed: once against gemini-3.5-flash, once against gemini-3.5-flash-lite.

Cross-version diff: 1 identical, 11 changed (91.7%), 1 refusal flip, 1 length drift -- DRIFT DETECTED, exit 1.

Then I recorded the control: the same model a second time, same temperature, same seed. Result: 1 identical, 11 changed (91.7%) -- also DRIFT DETECTED.

The seed does not make this model deterministic. Long free-form answers get reworded on every recording. So the honest reading is: at temperature 0, this prompt set cannot separate version drift from sampling noise. That is exactly why the tool ships with the control step -- record a same-model control first, measure your noise band, and only then read a cross-version diff. The README says this plainly, with the numbers.

One more find from the real data: the single "refusal flip" in the cross-version diff was a false positive. The lite model complied with a story in which a character says "I can't access that much from the drawer" -- dialogue, not a refusal. The detector matched the words, not a refusal. A flip is a signal to read the diff, not a verdict about the model.

Where the rivals beat us

• promptfoo runs live evals across providers, has an LLM-as-judge, a visual diff UI -- and yes, it has its own promptfoo diff command. If you already run evals through promptfoo, use it. modeldrift is for the case promptfoo does not cover: diffing two run files you already recorded, with no eval run, no YAML, no key, no server -- one zero-dependency command plus a pin file with a CI exit code.
• claude-drift replays real Claude Code sessions onto a new model and separates drift from noise with self-replay. That is a stronger noise measurement than mine. But it only works for Claude, needs the claude CLI logged in, and spends live tokens on every replay. modeldrift is provider-agnostic and spends zero tokens -- it diffs runs you already have.
• lm-eval-harness runs standard evals. It does not diff two recorded runs or pin a version.

modeldrift does not replace any of these. It is a thin layer on top of runs you already recorded: pin the model, diff the runs, get a verdict.

Honest limits

• Temperature 0 plus a fixed seed does not guarantee deterministic outputs -- my own control proves it. Always record a control.
• The control I shipped went through a different gateway than the baseline, so the 91.7% noise band is an upper bound mixed with gateway and served-version differences, not a pure noise measurement. A clean same-gateway control is the first follow-up.
• The refusal detector is a public phrase list. It will false-positive on dialogue. The phrase list is in the README and in the code, so you can see exactly what it matches.
• Recorded outputs are ASCII-folded for the repo (curly quotes, em dashes, accented letters). The fold is a fixed mechanical mapping applied identically to both sides, so it cannot create a fake CHANGED -- it can only hide tiny differences.
• Install is from source for now (git clone, npm install, npm run build). npm publish is pending.

Try it

git clone https://github.com/techdevfun/modeldrift.git
cd modeldrift
npm install
npm run build
node dist/src/cli.js diff examples/real/gemini-3.5-flash-20261110.jsonl examples/real/gemini-3.5-flash-lite-20261110.jsonl

The repo ships the real recorded runs I used above, so you can reproduce both diffs yourself. If your model vendor changes behavior silently, this is the cheapest way I know to catch it -- and to prove it with a file, not a feeling.