Every CI product in this cluster reduces to the same interface: a process runs, and its exit status decides whether the build is red. Everything you want from an eval gate has to be expressed through that one byte, which makes the design question “what does the runner decide, and what does it merely report?”
The only contract CI understands
Exit 0 is a pass, non-zero is a failure, and nothing else crosses the boundary. Reports, artifacts, comments and dashboards are all downstream of a decision that has already been made inside your process. So the comparison against the threshold belongs in the runner — not in a shell one-liner that greps the log, and not in a CI expression that parses a JSON field, both of which are common and both of which break silently when the output format shifts.
It helps to borrow a convention that already exists. pytest documents six exit codes: 0 when all tests were collected and passed, 1 when tests ran and some failed, 2 when execution was interrupted by the user, 3 for an internal error, 4 for a command line usage error, and 5 when no tests were collected. Notice how much of that vocabulary is about the run not happening properly rather than about tests failing. That is the right instinct for an eval, where “the score is low” and “the scorer never reached the model” are different events that a single non-zero status would flatten together. You can read the list in the pytest documentation.
The threshold is configuration
Put the number in a committed file that contains nothing but numbers. Not in the CI YAML, where changing it looks like a build fix and gets approved as one. Not in the scorer’s source, where it is mixed in with logic and a reviewer skims past it. A dedicated file makes a threshold change a visible, one-line diff that a reviewer has to have an opinion about, which is the entire point — lowering the bar should be a decision somebody made, with a name on it.
{
"suite": "support-triage-v3",
"model": "gpt-4.1-mini-2025-04-14",
"thresholds": {
"schema_valid_rate": 1.0,
"tool_selection_accuracy": 0.92,
"refusal_rate_max": 0.02,
"pii_leak_count_max": 0
},
"min_cases": 120,
"updated": "2026-08-11",
"note": "tool_selection_accuracy raised from 0.88 in PR #4412"
}
Two details in there earn their place. Several thresholds rather than one, because a single blended score hides which property regressed and lets a gain on an easy dimension pay for a loss on the one you care about. And min_cases, which is the subject of the section below.
Note what is not being asserted: no expected sentence, no reference answer to string-match. Every threshold above is over a property that survives paraphrase — does the output parse against the schema, was the right tool named, did a redaction hold. That is what makes the number stable enough to gate on at all, and it is why testing structured output is the easiest place to start.
A runner that does this
import json, sys
FAIL_SCORE = 1 # ran fine, quality below threshold
FAIL_INFRA = 75 # could not complete: provider, network, config
def gate(results: dict, cfg: dict) -> int:
if results["cases_scored"] < cfg["min_cases"]:
print(f"FAIL: scored {results['cases_scored']} cases, "
f"expected at least {cfg['min_cases']}", file=sys.stderr)
return FAIL_INFRA
breaches = []
for name, limit in cfg["thresholds"].items():
got = results["metrics"][name]
ok = got <= limit if name.endswith("_max") else got >= limit
print(f"{'ok ' if ok else 'FAIL'} {name}: {got} (limit {limit})")
if not ok:
breaches.append(name)
return FAIL_SCORE if breaches else 0
if __name__ == "__main__":
cfg = json.load(open("evals/thresholds.json"))
results = json.load(open("reports/eval.json"))
sys.exit(gate(results, cfg))
The naming convention carries the direction of the comparison, so a new metric cannot be added with the inequality the wrong way round — a mistake that produces a gate which passes on exactly the results it should block, and which nobody notices because green builds are not investigated. Printing every metric with its limit, pass or fail, means the log answers the question without anybody opening an artifact.
The gate function takes both structures as arguments and returns a status rather than calling sys.exit itself, which is not fastidiousness: it makes the gate the one part of the harness that is straightforward to unit test. Feed it a results dictionary that should fail and assert it returns 1; feed it one with too few cases and assert it returns the infrastructure status. That test needs no model, costs nothing to run, and is the only thing standing between you and a gate whose comparison logic was never exercised.
Zero cases must not be a pass
This is the failure that survives longest, because it is invisible. Your case file moves, a glob stops matching, a filter excludes everything, a fixture directory is not committed. The suite collects nothing, computes a pass rate over an empty set, and reports either 1.0 or a division that your code helpfully guards to 1.0. The gate goes green. It stays green for weeks.
The defence is the min_cases check above, and it should be an absolute count agreed in the config rather than a comparison against the last run — a comparison against the last run inherits whatever the last run got wrong. It is also worth failing loudly when the count is higher than expected, which catches a duplicated fixture directory being scored twice and quietly halving the effect of any one case. This is the same failure mode that makes a “score exists” check useless as a gate, treated at length in blocking a deploy on a flat or falling eval score.
Why a warning mode does not work
The reasonable-sounding design is three bands: green above the threshold, amber within some tolerance below it, red under that. It fails for a structural reason rather than a cultural one. Amber has no representation in the interface where the decision happens. A pull request has a merge button that is either enabled or not, and any status that does not disable it is, from the point of view of the person merging at five o’clock, a pass with extra text.
The second problem is that amber has no owner. A red build belongs to the author of the change that turned it red; an amber build belongs to nobody, and a signal owned by nobody decays into background noise within about two weeks. If you find yourself wanting an amber band, you are usually saying the threshold is set above the level you are actually prepared to enforce. Set it to the level you will enforce and let it be binary.
There is a legitimate version of the same instinct: report the score and the delta on every pull request, and gate only on the threshold. The information is available to anybody who wants it, and exactly one thing blocks the merge. That is what posting eval results as a pull request comment is for.
Top comments (0)