Run this against whatever you already have.
`from collections import Counter
def pass_all_k(run, tasks, k=8):
"""run(task, variant) -> bool. Returns (pass_all_rate, mean_rate)."""
all_pass, total = 0, 0
for t in tasks:
results = [run(t, variant=i) for i in range(k)]
all_pass += all(results)
total += sum(results)
return all_pass / len(tasks), total / (len(tasks) * k)`
Two numbers come back. The second is what your dashboard shows. The first is what a customer experiences, because customers do not get to retry until it works.
They are rarely close. On the systems I have measured, a mean around 0.85 has sat with a pass-all-8 around 0.45, and I have never once seen the gap go the other way.
Why the gap exists
If failures were independent at rate p, pass-all-k would be (1-p)^k and you could compute it rather than measure it. Failures are not independent, which is the entire point. They cluster by task shape.
That clustering is the actionable part. A 2026 reliability study running 23,392 episodes across ten models and a 396-task benchmark found that degradation was domain specific rather than model specific: a graceful degradation score fell from 0.90 to 0.44 in software engineering as task length grew, while document processing barely moved, 0.74 to 0.71.
So the interesting output of your harness is not the number. It is which tasks are in the failing set.
The variant function is the whole design
This is where most homegrown harnesses go wrong. Eight identical calls measure your cache. You need eight honest variations of the same intent.
`import random
REPHRASE = [
lambda s: s,
lambda s: s.lower(),
lambda s: f"I need to {s[0].lower()}{s[1:]}",
lambda s: f"{s} Please be thorough.",
lambda s: s.replace("?", "").strip() + ", if you can.",
]
def make_variant(task: dict, variant: int) -> dict:
rng = random.Random(f"{task['id']}:{variant}") # deterministic
out = dict(task)
out["prompt"] = rng.choice(REPHRASE)(task["prompt"])
if task.get("states"):
out["state"] = rng.choice(task["states"])
return out`
Seed on task_id:variant rather than on a global counter. Then a re-run of task 7, variant 3 is the same input it was last week, which is the difference between a harness you can compare across releases and one you cannot.
Record per-task, not per-run
The failure mode of a reliability harness is aggregating too early.
`import json, pathlib
def evaluate(run, tasks, k=8, out="reliability.jsonl"):
fh = pathlib.Path(out).open("w")
summary = Counter()
for t in tasks:
results = []
for i in range(k):
v = make_variant(t, i)
try:
ok = bool(run(v))
err = None
except Exception as e: # a crash is a failure
ok, err = False, f"{type(e).name}: {e}"
results.append({"variant": i, "ok": ok, "error": err})
rec = {
"task_id": t["id"],
"shape": t.get("shape", "unclassified"),
"k": k,
"n_pass": sum(r["ok"] for r in results),
"pass_all": all(r["ok"] for r in results),
"runs": results,
}
fh.write(json.dumps(rec) + "\n")
summary[t.get("shape", "unclassified")] += rec["pass_all"]
fh.close()
return summary`
shape is the field that earns its keep. Tag each task with what it is rather than which model ran it: lookup, multi_step, writes_state, long_horizon, needs_tool. Group the failures by shape and the pattern usually falls out on the first run.
An exception counts as a failure. A harness that only counts wrong answers and lets timeouts through will tell you a comforting lie.
What to do with the failing set**
**
Three findings from 2026 tell you where to look first, and each is a check you can run rather than a claim you have to believe.
If a failing shape is multi-agent, test the single-agent version. A study across 180 controlled configurations found that once single-agent accuracy passes roughly 45 percent on a task, adding agents produced negative returns, and that independent agents amplified errors 17.2 times against a single-agent baseline while centralised coordination held it to 4.4 times. Read-heavy work parallelises. Write-heavy work does not, because two agents writing produce two decisions nobody reconciles.
If a failing shape is long-horizon, do not assume a better model fixes it. Same reliability study: capability and reliability rankings diverged, and advanced models showed meltdown rates up to 19 percent, apparently because they attempt harder multi-step strategies.
Before you raise reasoning effort, measure it. Across 21,730 rollouts, higher reasoning effort produced equal or lower accuracy in 21 of 36 model and benchmark combinations. It is a per-task tuning parameter with a real downside, not a quality dial. Three effort levels against the same seed set is an afternoon.
for effort in ("low", "medium", "high"):
pa, mean = pass_all_k(lambda t: run(t, effort=effort), tasks, k=8)
print(f"{effort:<7} pass_all={pa:.2f} mean={mean:.2f}")
Watch for the case where mean rises and pass_all falls. That is a system getting better on average and less dependable, and it is invisible if you only track one of them.
Wiring it into CI
Keep it cheap or it will be deleted within a month.
- name: reliability
run: |
python -m harness --k 8 --tasks tasks/core.jsonl --out reliability.jsonl
python -m harness.gate --min-pass-all 0.60 --baseline main.jsonl
Two rules that have kept this alive on teams I have worked with. Gate on **regression **against the previous run, not on an absolute threshold, because an absolute number gets lowered the first time it blocks a release. And run the full k nightly while running k=3 on pull requests, because a twenty-minute pre-merge check gets disabled.
The honest limits
k=8 is arbitrary. It is enough that luck stops carrying you and few enough that people will actually run it. Use five if five is what gets done.
The variant function encodes your assumptions about what "the same request" means, and reasonable people will disagree about it. That is a feature: it forces the argument to happen in code review rather than after an incident.
And this measures reliability, not correctness. A task that fails all eight times consistently is perfectly reliable and completely wrong. You still need the assertions.
None of this is new thinking. Anyone who has run a payments system or a database already reasons about the worst request rather than the average one. We stopped doing it when the systems started sounding confident.
Sources: arXiv 2603.29231 (31 March 2026, 23,392 episodes); arXiv 2512.08296 (December 2025, 180 configurations); arXiv 2510.11977 (ICLR 2026, 21,730 rollouts).
If you already measure something like this, I would like to know what your variant function does. That part has no established convention yet and I suspect everyone has quietly invented their own.
Top comments (0)