If you're building on someone else's LLM API, you've probably had this experience: your app has been routing tickets, scoring leads, or gating decisions on a model's answers for months, it's been fine — and then one day it isn't. No error. No changelog. The same input just started coming back with a different answer.
That's what happened to us with TypeSafe's Jev model. We don't control when Jev gets retrained or redeployed. TypeSafe does. So we built a small open-source CLI, jev-watch, to catch it the moment it happens instead of finding out from a support ticket.
The bug that made the case for this tool
Early in building the adapter, we mapped score-type questions onto Jev's noul (yes/no probability) type instead of its native score type. It ran. It returned answers. Nothing crashed. It just silently dropped confidence data and returned worse answers than the model was actually capable of.
We only caught it by comparing live output against a set of answers we knew were correct. That comparison loop — save a known-good answer, re-run later, flag anything that changed — is exactly what jev-watch automates.
How it works
Write a test case as JSON: a scenario, a question, and the answer you know is correct today.
{
"testName": "CRAToolkit refund eligibility",
"state": "Customer says: I was charged twice for my subscription this month and want a refund",
"questions": {
"department": {
"type": "choice",
"instruction": "Which team should handle this?",
"options": {
"billing": "Payment, invoices, subscription issues",
"technical": "Login, bugs, product errors",
"sales": "New purchases, upgrades, demos"
}
}
},
"expected": { "department": "billing" },
"tolerance": 0.1
}
Commit that as your baseline. Re-run it anytime — after a model update, on a schedule, or in CI on every deploy:
$ jev-watch examples
PASS CRAToolkit refund eligibility
PASS CRAToolkit sales lead qualification
PASS CRAToolkit angry customer detection
PASS CRAToolkit support routing
4/4 tests passed
If Jev's answer changes — a choice flips, a score moves past tolerance, confidence drops — you get told exactly what changed, not a vague "something's off":
$ jev-watch examples/support-routing.json
FAIL CRAToolkit support routing
drift [department] choice: expected sales, got technical (tolerance 0.1)
0/1 tests passed
Exit code 0/1, so it drops straight into CI.
Setup
npm install && npm run build
echo "JEV_API_KEY=sk-..." > .env.local # or OPENROUTER_API_KEY, no TypeSafe account needed
node bin/jev-watch.js examples
Works against TypeSafe's API directly, or through OpenRouter if you'd rather not hold a TypeSafe key.
Why this matters beyond Jev
This is really just regression testing applied to model behavior instead of your code. Your code didn't change, your tests still pass, your types still check — but the thing sitting behind the API call did, and none of your existing test suite is watching for that. If you depend on any hosted model you don't control the release cycle of, the same gap exists.
Try it
Repo: https://github.com/akanthed/jev-watch — MIT licensed, v0.1.0, still early. If a model update has ever quietly broken something for you, turn that case into a JSON file and send it as a PR — that's exactly the kind of regression case this project needs more of.
Top comments (1)
The silent type mismatch bug at the start is the right motivation for this kind of tooling. Mapping score-type questions onto noul type, getting back valid-looking responses with no errors, and only catching it by comparing against a known-good answer set — that's the exact failure mode that doesn't show up in your existing test suite because your code is doing the right thing. The model is doing something different than what you specified.
The 'your code didn't change, your tests still pass' framing is exactly right. This is a new class of regression that conventional CI doesn't address. A hosted model you don't control the release cycle of is a dependency, and dependencies change. Treating model behavior as something you regression-test the same way you regression-test code is the correct mental model.
The tolerance parameter is the detail that makes this practical rather than theoretical. A choice answer is binary, but a score or a confidence number has natural variance. Catching a flip from billing to technical is straightforward; knowing when a confidence drift is signal vs noise requires a threshold. The JSON test format expressing this explicitly is cleaner than trying to embed tolerance logic in test code.
The case for committing baselines and re-running on deploy is strong. It's the same intuition as golden-file testing for deterministic outputs — you're not testing against a specification, you're testing against 'this is what it used to do.' When that changes, you want to know before your users do.