My LLM-eval cache made CI free, except the judge ran every time

llm_eval is a small test harness for LLM evals in Dart. You describe cases and checks, run them against your model, and get a report. The feature that makes it usable in CI is a response cache: the first run records each model response keyed by model id and prompt, and later runs read from the cache instead of calling the model. That is what keeps a CI run deterministic and free, so a green suite does not re-bill you on every push.

final cache = FileResponseCache('.eval_cache');

final report = await suite.run(model, modelId: 'gpt-4o-mini', cache: cache);

It worked, and I trusted it, until I looked at a CI run's cost and saw the judge model still being billed on a warm cache.

The judge is a model call the cache never saw

Some checks are exact (Check.contains, Check.isValidJson). One is not: Check.judge hands the model's output to a second model and asks it to score against a rubric. That judge is often a stronger, pricier model than the one under test.

Check.judge(
  judge: judgeModel,
  rubric: 'One sentence, names the version, no marketing.',
  passAt: 0.7,
)

Here is the gap. suite.run caches the call it makes: the model under test, keyed by the modelId I passed. But the judge runs inside the check, as a nested model call the suite never sees, so nothing caches it. On a warm cache the model under test was skipped and served from disk, exactly as intended, and then the judge fired again on every single case, every run. CI was neither free nor deterministic for the one call I most wanted pinned, because the judge is the stronger model and its scores can drift between judge versions.

Worse, the report still labeled those attempts as cached, because fromCache reflected the model-under-test call, not the judge. So the cost showed up on the bill but not in the report.

Give the nested call its own cache path

The fix is to let the judge read and write the same cache, under its own id. ResponseCache gained a wrap that takes a model call and returns one that goes through the cache first, using the same length-prefixed key scheme the suite already uses:

Check.judge(
  judge: cache.wrap(judgeModel, modelId: 'judge-v1'),
  rubric: 'One sentence, names the version, no marketing.',
  passAt: 0.7,
)

Now the judge's responses live in the same cache next to the model-under-test's, under judge-v1, so a warm run serves the judge from disk too and the suite is actually free and deterministic end to end. Giving the judge its own id matters: pin or change one judge and only its entries re-record, the model under test is untouched.

Two smaller decisions came with it. A cache read that throws is treated as a miss and falls through to the real call, the same way the suite already tolerates a broken cache, so a corrupt entry degrades to a live call rather than an error. And I left the report's cached column describing only the model under test, and documented that, rather than trying to fold a nested call's cache state into a number that was always about the top-level call.

The rule I took from it: a cache that wraps your top-level call says nothing about the calls nested inside it. When one of those nested calls is a second, more expensive model, it is the one you most want cached, and it is the one your cache is quietly missing. Wrap it on purpose.