On 6 October, MLflow merged a fix of mine for a model-validation bug in MetricThreshold (mlflow/mlflow#26252). It is a small bug, and a good example of a broader failure mode in evaluation software: a tool can return a perfectly well-formed number or verdict that is not supported by the evidence it claims to measure.
One example
MetricThreshold can gate model promotion on relative improvement over a baseline. The implementation divided the change by the signed baseline value. When that baseline was negative, as R² can be, the sign of the relative-change calculation reversed. A worse candidate could pass while a better one failed.
Nothing crashed. No malformed value appeared. The gate produced a clean answer to the wrong measurement.
The merged fix measures change against the magnitude of the baseline and adds 26 regression cases across positive and negative baselines and both metric directions.
Where it sits
Between 14 and 26 September I read the scoring and gating code of seven open-source evaluation tools: the agent-skills eval harness and a reference gate script, NVIDIA SkillEvaluator, DSPy, LangSmith, DeepEval, Harbor and MLflow. I reproduced 13 defects, 12 of them offline, and filed each upstream with a public reproduction. Twelve carried a proposed fix. At the paper's 30 September cutoff, five fixes had merged and another had been approved. Since then, the MLflow fix has merged as well.
In every case the pattern was the same: the tool returned a result its evidence did not support. The 13 fall into six kinds.
- Absent evidence scored as a measurement
- A result bound to the wrong unit
- A proxy credited as the act
- Stale state read as current
- A sign error in a threshold
- A check aimed at a target the artefact never claimed
The write-up is on Zenodo: Reported, Not Measured: An Empirical Study of Measurement Defects in LLM and Agent Evaluation Tools (CC BY 4.0). It lists each case, the reproduction, the upstream link and its status at the cutoff.
The same standard applied to my own tool
I maintain an evaluation tool myself, so the paper classifies defects from Driftproof's public history under the same six kinds: a parser that converted a non-numeric judge response into a score, a badge that verified generation text but not judge text, and six false-pass cases fixed in 0.12.0. They sit in the catalogue with the same weight as the others.
Why this is not a statistics problem
A noisy measurement can be qualified: more samples, confidence intervals, an effect floor. A well-formed number that was never a valid measurement cannot be repaired that way. More runs of the MLflow gate with a negative baseline give you the same wrong answer with more confidence.
This matters because these outputs sit behind decisions: whether a model is promoted, whether a regression is accepted, whether an agent passes CI, or whether one system is reported as better than another.
The paper proposes nine fields an evaluation record should carry, each paired with the check that would read it, and maps each of the 13 defects to the field and check that could have exposed it. That mapping is a design argument. It has not been tested, and the paper says so.
Try the idea on one of your own skills
Driftproof 0.14.0 also shipped today. If you already have a Claude Code skill, /driftproof:start drafts test cases from the skill with you, writes them only after you approve, and runs a short smoke test. The smoke run is deliberately non-conclusive: its receipt says on its face that it cannot produce a verdict. A full run is required for a measured result.
- Claude Code:
/driftproof:start - CLI:
npx driftproof@0.14.0 - Release notes: v0.14.0
What this is not
The 13 cases are a catalogue of mechanisms found where I looked. Seven tools, chosen by my judgement, over twelve days. It is a lower bound on what exists, not a prevalence estimate, and it says nothing about tools I did not read.
The point is not that evaluation tools are uniquely unreliable. It is that evaluation software deserves the same suspicion we apply to every other measuring instrument: before asking how precise a number is, first ask whether the code measured what the number says it measured.
Disclosure: I build and maintain Driftproof (open source, Apache 2.0).
Top comments (5)
The MLflow
MetricThresholdbug is a perfect specimen — dividing by the signed baseline so a negative R² flips the sign of relative improvement, and a worse candidate sails through the gate. Nothing crashes, the number is well-formed, and the promotion decision is just quietly wrong. That's the scariest category because it never trips an alert.What strikes me reading your six kinds is how many trace back to the eval tool trusting its own arithmetic without a sanity oracle — the gate computes a verdict but never checks that the verdict is even directionally sane. Did you see a recurring root cause across the seven tools (sign handling, empty-denominator, silent type coercion), or were they genuinely independent? And practically: for teams who can't audit every eval dependency, is there a cheap smoke test you'd recommend — something like feeding a known-worse candidate through the gate and asserting it fails? Great work filing all 13 upstream with repros instead of just dunking.
Mechanically they were independent: a sign division, trials joined on task name instead of trial id, missing judge verdicts defaulting to pass, a file read credited as an execution, scores written back to the wrong runs after shuffling. No shared arithmetic class. What recurred was a design habit: the result object does not carry the evidence that produced it, so nothing downstream can notice when the number stops meaning anything. That is why the paper ends in nine record fields paired with the checks that read them, rather than advice about denominators.
Your smoke test is the right instinct. I would make it four asserts, all cheap, no source reading:
Several of the 13 would have surfaced under those four alone.
The sign error is nasty because the number is also unstable in the other direction: with relative change measured against |baseline|, a baseline near zero (R² around 0.01, say) turns a 0.005 gain into a 50% "improvement" that clears any relative threshold. Fixing the sign is necessary but the gate still needs a floor on the denominator, or an absolute-change rule below some baseline magnitude. Did your 26 regression cases include baselines close to zero, or only clearly positive and clearly negative ones?
On your "not a statistics problem" point, I'd split it slightly: more runs can't repair an invalid measurement, but once the gate is valid, a single-run comparison of two candidates still needs the run-to-run spread, otherwise a correct gate promotes noise just as confidently.
Fair catch, and the honest answer is no: the 26 merged cases use baseline magnitudes of 1.8 and 2.0, both signs, both metric directions, nothing near zero. Exactly zero was already handled before my change (the gate falls back to a plain better-or-not comparison), but a baseline like 0.01 goes down the relative path and does what you describe: abs() fixes the sign and leaves the inflation. A magnitude floor with an absolute-change rule under it is the natural next hardening. I kept the PR to the sign defect on purpose, since where that floor sits is a policy choice for the maintainers rather than a correctness fix, but it deserves its own issue.
On the split: agreed, and that is the division of labour I work to. Validity first, then uncertainty. Once a gate computes the thing it claims, a single-run comparison still promotes noise with full confidence, which is why Driftproof verdicts only come from ranges over repeated runs, never one pair of numbers.
Cảm ơn bạn đã chia sẻ chi tiết case study này. Vấn đề "well-formed number ≠ valid measurement" thực sự là nỗi đau lâu năm trong MLOps mà ít team đầu tư đủ vào validation layer.
Đọc qua 13 defects across 7 tools, pattern chung tôi thấy là: hầu hết eval tools đều assume input đã clean, few handle edge cases như NaN/Inf propagation, silent overflow trong floating-point accumulation, hay threshold comparison với precision loss. Bug MetricThreshold của MLflow là ví dụ điển hình — comparison logic không account cho floating-point epsilon dẫn đến false pass/fail ở boundary.
Một lesson học được từ production: luôn inject một validation shim trước khi feed vào bất kỳ eval tool nào (schema check + range check + NaN/Inf guard). Chi phí upfront nhỏ nhưng cứu được rất nhiều debug hour downstream.
Curious: trong 7 tools bạn test, có tool nào handle uncertainty quantification (confidence intervals, prediction intervals) built-in không? Hầu hết chỉ trả point estimate mà quên rằng measurement without uncertainty is incomplete PS: the tool I meant is on labagent .tech