Gating an agent's eval run by severity means the deploy decision reads the per-severity distribution of the failures, not one flat pass-rate. severity_gate.py is an offline, stdlib-only tool that returns SHIP, REVIEW, or BLOCK. In this post's fixtures, flipping one severity_class field to critical turns SHIP into BLOCK while the pass-rate holds at 92.5%.
AI disclosure: I wrote
severity_gate.pywith an AI assistant and ran it myself, offline, before publishing. Every output block below is pasted from a real local run on Python 3.13.5, standard library only, no network. I checked the exit codes (0 / 1 / 2), hashed the STDOUT of each scenario twice to confirm it is byte-for-byte deterministic, and edited every line. The 512 / 31 / 6 figures and the "rounding error" line belong to Ethan Walker's Dev.to post, not to me; I link the primary sources and keep their numbers in their own paragraphs, away from my fixture counts (40 / 3 / 2, 92.5%).
In short:
- Whether a run is safe to ship is a property of which failures it holds, not of their average. A run can sit at a healthy pass-rate and still carry the one failure that should have stopped the deploy on its own.
- The tool reads a finished eval run (JSON) plus a small policy. Any FAIL in a blocking severity class (default: critical) is a BLOCK, whatever the aggregate says. A below-threshold aggregate with no critical fail is a REVIEW. Everything clean is a SHIP.
- The demo: same 40 cases, same 37/40 = 92.5% pass-rate, same three failing cases. Change one field,
severity_classfromlowtocriticalon two of them, and the verdict flips SHIP to BLOCK. The pass-rate does not move. The diff is two lines. - It re-weights failures you already labeled. It does not detect PII, does not judge correctness, does not find new failures. Garbage labels in, garbage gate out.
- Standard library only (
json,sys). Offline, keyless, read-only, zero network, deterministic STDOUT. Exit 0 / 1 / 2 for a CI gate. The tool and every fixture are in this post.
A flat pass-rate measures the run. It was never the gate.
Here is the ritual I keep seeing at the end of an agent's CI pipeline. The eval suite runs, a number comes out, someone reads it, and if it clears the bar the change ships. The number is a mean: passes over total. It is a fine thing to track. It tells you, roughly, whether the run got better or worse than last week.
It is a bad thing to gate on, and the reason is simple arithmetic. A mean gives every failure the same weight. So a run where forty tests failed on trailing whitespace and a run where one test leaked a customer's email into a log both cost the same amount of pass-rate. If your suite is big enough, the second run rounds off to nothing. The one failure that should stop the deploy is invisible at the resolution of the average.
That is the whole gap this tool sits in. The pass-rate is tracking: a post-hoc measurement of what happened. The decision about what ships is control, and control has to see the distribution the mean flattens. severity_gate.py reads the finished eval results and makes the ship decision on the failures' severity, not their count.
The rule the mean can't express
The tool keeps the pass-rate on screen, for reference, and then ignores it for the parts of the decision the mean can't reach. Three verdicts, one policy:
-
BLOCK when at least one failing case is in a blocking severity class. Default policy:
critical. This fires no matter what the aggregate is. One critical fail out of ten thousand green cases is still a BLOCK. -
REVIEW when nothing blocking failed, but either a case in a review class failed (default:
high), or the flat pass-rate dropped belowship_threshold. A human looks. It is not an auto-ship and it is not a hard stop. - SHIP only when nothing blocking or review-class failed and the pass-rate meets the threshold.
The severity labels are yours. The tool does not invent them and does not guess them; it reads the severity_class field you put on each case. That is the load-bearing limit, and I come back to it at the end. What the tool contributes is that once you have committed to a label, the ship decision follows it deterministically instead of dissolving into an average.
Run it in sixty seconds
No keys. No network. No install beyond Python. Save the file, point it at a results JSON, run one command. Here is the whole thing, one file, standard library only:
#!/usr/bin/env python3
"""
severity_gate.py -- an offline deploy gate that reads a FINISHED eval run and
decides SHIP / REVIEW / BLOCK from the per-severity distribution of the failures,
not from a single flat pass-rate.
It reads a JSON file of eval RESULTS (it never runs the agent or the eval) plus
an optional policy file. For every case it reads a verdict (pass/fail) and the
severity_class the USER assigned (critical / high / medium / low). It reports the
flat pass-rate for reference, then makes the ship decision on a rule the mean
cannot express:
BLOCK -- at least one FAIL whose severity_class is in the policy's block_on
set (default: critical). Fires regardless of the aggregate.
REVIEW -- no blocking fail, but either a FAIL in review_on (default: high),
or the flat pass-rate is below ship_threshold. A human looks; it is
not an auto-ship and not a hard block.
SHIP -- no blocking fail, no review fail, and pass-rate >= ship_threshold.
The point the tool exists to make: flip one field -- the severity_class on a
handful of already-failing cases -- and the verdict moves SHIP -> BLOCK while the
pass-rate does not move at all. The aggregate MEASURES the run; it was never the
control over what ships.
Offline. Keyless. Read-only. Zero network. Standard library only (json, sys).
No subprocess, no exec, no eval, no import of the analyzed results, no model, no
DB. The fixtures are read with json.load; they are DATA, never executed. Output
is byte-for-byte deterministic across runs.
It does NOT detect PII, does NOT judge whether a verdict is correct, and does NOT
find new failures. It re-weights failures YOU already labeled. Garbage labels in,
garbage gate out. It decouples the ship decision from the scalar mean; it does
not discover anything.
Exit codes (usable as a CI gate):
0 SHIP
1 REVIEW or BLOCK (the verdict text says which, and why)
2 bad input (missing / unreadable / unparseable file, missing field, unknown
verdict or severity_class) -- fail-closed
Usage:
python3 severity_gate.py <results.json> [policy.json]
"""
import json
import sys
DEFAULT_POLICY = {
"ship_threshold": 0.90,
"severity_order": ["low", "medium", "high", "critical"],
"block_on": ["critical"],
"review_on": ["high"],
}
VALID_VERDICTS = ("pass", "fail")
def _bad(msg):
print("ERROR: " + msg)
raise SystemExit(2)
def _load_json(path, what):
try:
with open(path, "r") as fh:
return json.load(fh)
except OSError as exc:
_bad("cannot read %s %s: %s" % (what, path, exc))
except ValueError as exc:
_bad("cannot parse %s %s: %s" % (what, path, exc))
def load_policy(path):
if path is None:
return dict(DEFAULT_POLICY)
data = _load_json(path, "policy")
if not isinstance(data, dict):
_bad("policy %s is not a JSON object" % path)
pol = dict(DEFAULT_POLICY)
pol.update(data)
order = pol["severity_order"]
if not isinstance(order, list) or not order:
_bad("policy severity_order must be a non-empty list")
for key in ("block_on", "review_on"):
for cls in pol[key]:
if cls not in order:
_bad("policy %s lists '%s', not in severity_order %s"
% (key, cls, order))
try:
pol["ship_threshold"] = float(pol["ship_threshold"])
except (TypeError, ValueError):
_bad("policy ship_threshold must be a number")
return pol
def load_cases(path, order):
data = _load_json(path, "results")
cases = data.get("cases") if isinstance(data, dict) else data
if not isinstance(cases, list) or not cases:
_bad("results %s must hold a non-empty list of cases" % path)
default_sev = order[0]
out = []
for i, c in enumerate(cases):
if not isinstance(c, dict):
_bad("case %d is not an object" % i)
verdict = c.get("verdict")
if verdict not in VALID_VERDICTS:
_bad("case %s has verdict %r, expected 'pass' or 'fail'"
% (c.get("case_id", i), verdict))
sev = c.get("severity_class", default_sev)
if sev not in order:
_bad("case %s has severity_class %r, not in severity_order %s"
% (c.get("case_id", i), sev, order))
out.append({
"case_id": str(c.get("case_id", "case-%d" % i)),
"verdict": verdict,
"severity_class": sev,
"category": str(c.get("category", "")),
})
return out
def decide(cases, pol):
block_on = set(pol["block_on"])
review_on = set(pol["review_on"])
total = len(cases)
passed = sum(1 for c in cases if c["verdict"] == "pass")
rate = passed / total
block_fails = [c for c in cases
if c["verdict"] == "fail" and c["severity_class"] in block_on]
review_fails = [c for c in cases
if c["verdict"] == "fail" and c["severity_class"] in review_on]
if block_fails:
verdict, code = "BLOCK", 1
reason = ("%d failing case(s) in a blocking severity class (%s)"
% (len(block_fails), ", ".join(sorted(block_on))))
elif review_fails:
verdict, code = "REVIEW", 1
reason = ("%d failing case(s) in a review severity class (%s)"
% (len(review_fails), ", ".join(sorted(review_on))))
elif rate < pol["ship_threshold"]:
verdict, code = "REVIEW", 1
reason = ("flat pass-rate %.1f%% is below ship_threshold %.1f%%"
% (rate * 100, pol["ship_threshold"] * 100))
else:
verdict, code = "SHIP", 0
reason = ("no blocking or review-class failure, pass-rate %.1f%% "
">= ship_threshold %.1f%%" % (rate * 100,
pol["ship_threshold"] * 100))
return {"total": total, "passed": passed, "failed": total - passed,
"rate": rate, "verdict": verdict, "reason": reason, "code": code}
def render(cases, pol, d):
order = pol["severity_order"]
rank = {c: i for i, c in enumerate(order)}
block_on, review_on = set(pol["block_on"]), set(pol["review_on"])
out = ["SEVERITY-GATE REPORT"]
out.append("cases: %d pass: %d fail: %d"
% (d["total"], d["passed"], d["failed"]))
out.append("flat pass-rate: %.1f%% (%d/%d)"
% (d["rate"] * 100, d["passed"], d["total"]))
out.append("ship_threshold: %.1f%% block_on: [%s] review_on: [%s]"
% (pol["ship_threshold"] * 100,
", ".join(pol["block_on"]), ", ".join(pol["review_on"])))
out.append("per-severity (fail / total):")
for cls in reversed(order): # worst first
tot = sum(1 for c in cases if c["severity_class"] == cls)
fl = sum(1 for c in cases
if c["severity_class"] == cls and c["verdict"] == "fail")
mark = " <- blocking" if cls in block_on else (
" <- review" if cls in review_on else "")
out.append(" %-8s %d / %d%s" % (cls, fl, tot, mark))
fails = [c for c in cases if c["verdict"] == "fail"]
fails.sort(key=lambda c: (-rank[c["severity_class"]], c["case_id"]))
if fails:
out.append("failing cases (worst severity first):")
for c in fails:
cat = (" [%s]" % c["category"]) if c["category"] else ""
out.append(" - %-8s %s%s" % (c["severity_class"], c["case_id"], cat))
out.append("decision: %s -- %s" % (d["verdict"], d["reason"]))
return "\n".join(out)
def main(argv):
if len(argv) not in (2, 3):
print("usage: severity_gate.py <results.json> [policy.json]")
raise SystemExit(2)
pol = load_policy(argv[2] if len(argv) == 3 else None)
cases = load_cases(argv[1], pol["severity_order"])
d = decide(cases, pol)
print(render(cases, pol, d))
raise SystemExit(d["code"])
if __name__ == "__main__":
main(sys.argv)
The inputs are eval results, one JSON object per run: a cases list where each case has a verdict and a severity_class. To keep the runs below reproducible byte for byte, I generate the sample files with a tiny builder that only writes data. The gate reads that data; it never runs it.
#!/usr/bin/env python3
# make_fixtures.py -- writes the sample eval-result files this post runs on.
# It only emits DATA. severity_gate.py reads it with json.load, never executes it.
import json
import os
os.makedirs("fixtures", exist_ok=True)
def case(i, verdict="pass", sev="low", cat="general"):
return {"case_id": "c%03d" % i, "verdict": verdict,
"severity_class": sev, "category": cat}
def write(name, cases):
with open("fixtures/%s" % name, "w") as fh:
json.dump({"cases": cases}, fh, indent=2, sort_keys=True)
fh.write("\n")
# 40 cases, 3 fail -> flat pass-rate 37/40 = 92.5%. Two fails are a PII cluster
# left untriaged (severity low); one is an unrelated format miss (medium).
base = [case(i) for i in range(1, 41)]
base[9] = case(10, "fail", "medium", "output_format")
base[19] = case(20, "fail", "low", "pii_leak")
base[29] = case(30, "fail", "low", "pii_leak")
write("ship_run.json", base)
# Same 40 cases, same 37/40 = 92.5%. ONLY the two PII fields change: low->critical.
killer = [dict(c) for c in base]
killer[19]["severity_class"] = "critical"
killer[29]["severity_class"] = "critical"
write("block_run.json", killer)
# Aggregate below the bar, nothing critical/high: 34/40 = 85.0%.
review = [case(i) for i in range(1, 41)]
for i in (5, 12, 18, 23, 31, 37):
review[i - 1] = case(i, "fail", "medium", "verbosity")
write("review_run.json", review)
# A critical-severity case that PASSED must not block. 20 cases, one low fail.
edge = [case(i) for i in range(1, 21)]
edge[0] = case(1, "pass", "critical", "pii_leak")
edge[14] = case(15, "fail", "low", "typo")
write("edge_run.json", edge)
# Unknown severity label -> fail-closed (exit 2).
bad = [case(1), case(2, "fail", "catastrophic", "pii_leak")]
write("bad_run.json", bad)
The baseline: a green run that isn't safe to ship
Start with ship_run.json: 40 cases, 37 pass, 3 fail. The pass-rate is 37/40 = 92.5%, comfortably over the 90% bar. Two of the three failures are a pii_leak cluster that nobody has triaged yet, so they carry the default low severity. The third is a formatting miss, medium. My fixture, my run:
$ python3 severity_gate.py fixtures/ship_run.json
SEVERITY-GATE REPORT
cases: 40 pass: 37 fail: 3
flat pass-rate: 92.5% (37/40)
ship_threshold: 90.0% block_on: [critical] review_on: [high]
per-severity (fail / total):
critical 0 / 0 <- blocking
high 0 / 0 <- review
medium 1 / 1
low 2 / 39
failing cases (worst severity first):
- medium c010 [output_format]
- low c020 [pii_leak]
- low c030 [pii_leak]
decision: SHIP -- no blocking or review-class failure, pass-rate 92.5% >= ship_threshold 90.0%
Exit 0. SHIP. This is the run that ships today under a flat pass-rate: the number is green, nothing is labeled critical, out the door it goes. The two pii_leak cases are sitting right there in the report, but at severity low they are just two more failures the mean absorbed.
One field flips ship to block
This is the demo the post exists for. block_run.json is ship_run.json with exactly one thing changed: the two pii_leak cases, the ones already failing, are relabeled from low to critical. Same 40 cases. Same 37 passes. Same 92.5%. The diff:
$ diff fixtures/ship_run.json fixtures/block_run.json
120c120
< "severity_class": "low",
---
> "severity_class": "critical",
180c180
< "severity_class": "low",
---
> "severity_class": "critical",
Two lines. Nothing about the pass-rate changed, because nothing about the pass/fail counts changed. Now run the gate on it:
$ python3 severity_gate.py fixtures/block_run.json
SEVERITY-GATE REPORT
cases: 40 pass: 37 fail: 3
flat pass-rate: 92.5% (37/40)
ship_threshold: 90.0% block_on: [critical] review_on: [high]
per-severity (fail / total):
critical 2 / 2 <- blocking
high 0 / 0 <- review
medium 1 / 1
low 0 / 37
failing cases (worst severity first):
- critical c020 [pii_leak]
- critical c030 [pii_leak]
- medium c010 [output_format]
decision: BLOCK -- 2 failing case(s) in a blocking severity class (critical)
Exit 1. BLOCK. The pass-rate line is identical to the SHIP run, character for character: flat pass-rate: 92.5% (37/40). The deploy verdict inverted anyway. This is the thing to sit with. If the mean were a valid ship criterion, a change that leaves the mean untouched could not change the decision. It changed the decision. So the mean was not the criterion; it was a number we let stand in for one.
And it would take one, not two. Relabel a single pii_leak case to critical and the gate blocks, because the policy tolerates zero critical failures. The cluster size is not the point. The point is that a severity edit with no effect on the aggregate has total effect on the verdict.
The falsifiability test: when nothing is critical, the aggregate still rules
If the tool blocked on everything, or ignored the pass-rate entirely, it would be a different kind of useless. So here is the counter-case that would break the thesis if it came out wrong. review_run.json drops the aggregate below the bar, 34/40 = 85.0%, but every failure is benign: six medium cases, nothing critical, nothing high.
$ python3 severity_gate.py fixtures/review_run.json
SEVERITY-GATE REPORT
cases: 40 pass: 34 fail: 6
flat pass-rate: 85.0% (34/40)
ship_threshold: 90.0% block_on: [critical] review_on: [high]
per-severity (fail / total):
critical 0 / 0 <- blocking
high 0 / 0 <- review
medium 6 / 6
low 0 / 34
failing cases (worst severity first):
- medium c005 [verbosity]
- medium c012 [verbosity]
- medium c018 [verbosity]
- medium c023 [verbosity]
- medium c031 [verbosity]
- medium c037 [verbosity]
decision: REVIEW -- flat pass-rate 85.0% is below ship_threshold 90.0%
Exit 1, but REVIEW, not BLOCK, and the reason names the aggregate, not a severity class. This is the third state doing real work. The gate did not throw the pass-rate away; when nothing dangerous failed, the aggregate is exactly what it falls back on. REVIEW is where a low mean lands. BLOCK is reserved for the failure class you said must never ship. Put this run next to the killer and the tool's shape is clear: it answers to the distribution of severity when severity is present, and to the mean when it is not.
A critical case that passed is not a block
One boundary a careful reader will poke at. Does BLOCK fire on the presence of a critical case, or on a critical failure? It has to be the failure, or you could never ship a suite that tests a critical path at all. edge_run.json has a critical-severity case that passed (your PII redaction test ran and came back clean) and one unrelated low failure.
$ python3 severity_gate.py fixtures/edge_run.json
SEVERITY-GATE REPORT
cases: 20 pass: 19 fail: 1
flat pass-rate: 95.0% (19/20)
ship_threshold: 90.0% block_on: [critical] review_on: [high]
per-severity (fail / total):
critical 0 / 1 <- blocking
high 0 / 0 <- review
medium 0 / 0
low 1 / 19
failing cases (worst severity first):
- low c015 [typo]
decision: SHIP -- no blocking or review-class failure, pass-rate 95.0% >= ship_threshold 90.0%
Exit 0. SHIP. Note the per-severity line: critical 0 / 1 — one critical case exists, zero critical cases failed. A passing test on a critical path is exactly the signal you want; blocking on it would punish you for having good coverage. The gate blocks on critical fail, not on critical present.
Who decides what blocks is policy, not the tool
The default policy blocks on critical and reviews on high. That default is mine, and you should not keep it without thinking. The mapping from a failure category to a severity class, and the choice of which classes stop a deploy, is a decision your team owns, not something a linter should hand you. So the policy is a file you pass in. Here is fixtures/strict_policy.json — four lines that also block on high and send medium to review:
{
"ship_threshold": 0.90,
"block_on": ["critical", "high"],
"review_on": ["medium"]
}
Point the same ship_run.json at it:
$ python3 severity_gate.py fixtures/ship_run.json fixtures/strict_policy.json
SEVERITY-GATE REPORT
cases: 40 pass: 37 fail: 3
flat pass-rate: 92.5% (37/40)
ship_threshold: 90.0% block_on: [critical, high] review_on: [medium]
per-severity (fail / total):
critical 0 / 0 <- blocking
high 0 / 0 <- blocking
medium 1 / 1 <- review
low 2 / 39
failing cases (worst severity first):
- medium c010 [output_format]
- low c020 [pii_leak]
- low c030 [pii_leak]
decision: REVIEW -- 1 failing case(s) in a review severity class (medium)
Exit 1. Same run, same 92.5%, but under a policy that treats a medium format miss as review-worthy, the verdict is REVIEW. The tool did not decide that a format miss matters. You did, in the policy file. The gate just applied it the same way every time.
Ethan's 94% run, stated as his
I did not come up with this from nothing. On July 5, Ethan Walker published a Dev.to post whose title is the whole argument in one line: a 94% pass rate hid a PII leak in 6 test cases. His numbers, from his run, stated as his: 512 test cases, 31 failures, and 6 of those were a PII leak. His line, which I checked against the post before quoting: "Six out of 512 is a rounding error against a flat pass-rate metric." He also names the tools by hand — DeepEval, Promptfoo, LangSmith — and writes that they "all give you this by default; none of them force severity weighting on you."
Those figures are his and none of the counts in my runs above are. I built fixtures of a different size on purpose so the two never blur: my numbers are 40 / 3 / 2 at 92.5%, computed by my tool on my synthetic files; his are 512 / 31 / 6 at 94%, from his own eval suite. The tool in this post is one narrow, runnable answer to the thing his title describes: the point where a small critical cluster hides inside a green aggregate.
The general form of it was put well the same week by another writer, posting as aiexplore369zoho, under the heading the mean is lying to you. Their framing: "A benchmark score is a mean." Reliability, they argue, is a tail statistic, not a central one; the mean can stay flat while the failure rate on a critical slice doubles. That is the concept. Naming the slice that must not fail and gating on it directly, instead of on the average that buries it, is one way to act on it.
Why not just add a smarter judge?
A fair objection: skip the labels, put a stronger model in the loop as an inspector, and let it catch the critical cases. Someone tested exactly that. Posting as zxpmail, they ran three models as agent quality inspectors and reported the opposite of a free lunch: the stronger the model, the more valid work it rejected. Their strongest model reached 0% false positives on garbage but falsely rejected 3 of 4 perfectly valid outputs. Those are their measurements, not mine.
The takeaway I draw for a gate is narrow. A probabilistic inspector trades one error for another and gives you a different answer on reruns. A gate over labels you already committed to is deterministic: same input, same verdict, every run. That determinism is not a nice-to-have here; it is the property the SHA hashes at the end are there to prove.
Where this sits next to the rest
This is a spoke on the pre-execution gate for AI agents cluster, and its object is the deploy decision over a finished eval run. The neighbors ask adjacent questions, and the differences matter:
- The green-checkmark auditor is the inverse case, and worth stating plainly so the two do not get merged. There, the green carries no signal at all: mirror tests that restate the code, with nothing that can ever go red. Here, the signal is present — the failing cases are real and correctly failing — and the mean drowns it. One post is about a checkmark that means nothing; this one is about failures that mean something and get averaged into silence.
- The eval-contamination probe asks whether the score can even be trusted, whether the harness graded the real work or a fabricated artifact. That is a question about whether the number is valid. This post assumes the number is valid and asks whether a valid mean is a valid ship criterion. Different link in the chain.
- Reconciling a scorecard from evidence is the case where the claimed metric might be false — the log does not back it up. Here the 92.5% is true. I am not disputing the number; I am disputing that a true average is what should decide the deploy.
- Your agent returns 200 and lies checks one runtime result against its evidence. That is a single call. This is the aggregate over a whole eval run, the pre-deploy decision that sits above all those single calls.
What this is NOT
I would rather undersell this than have you deploy it as something it isn't.
- Severity-based gating is not my invention, and I am not claiming it is. CVSS scores security findings by severity; DREAD ranked them; every bug tracker has had a Critical / Major / Minor field for decades; risk-based testing has weighted test outcomes for a very long time. None of that is new. What is narrow here is the target: the agent-eval deploy gate, where a screenshot of a flat pass-rate is still the shipping ritual, packaged as a portable stdlib gate over the JSON your eval tool already emits. I am applying an old principle to a place that mostly still ignores it, not discovering the principle.
-
It acts only on the labels you assign. It does not detect PII. It does not judge whether a
failis really a fail or apassis really a pass. It finds no new failures. Mislabel a critical failure aslowand it ships; that is the GIGO limit, and it is real. The tool re-weights failures you already found and classified. It decouples the ship decision from the scalar mean. It does not discover anything. - It reads results; it does not run the agent or the eval. It is an offline post-processor for a finished run, meant for the step between "eval done" and "merge/deploy." To generate the verdicts and severities in the first place you still need your eval framework.
- It does not replace DeepEval, Promptfoo, or LangSmith. It is a decision layer that consumes their output. Their default is a flat pass-rate; this is the thin gate you bolt on top of the JSON they produce, so the ship decision reads severity instead of the average. That is the whole seam it fills, and nothing more.
-
The taxonomy and the category-to-severity mapping are policy, not truth.
critical / high / medium / lowand the choice of what blocks are a file you own. The tool ships a default so it runs out of the box; the default is a starting point to argue with, not a standard. - The counts here are fixture units. The 40 / 3 / 2 and the 92.5% describe the synthetic files in this post. Run it on your own eval output to get numbers that mean something about your agent.
One design choice I am still not certain about: REVIEW and BLOCK share exit 1, and only the verdict text tells them apart. I did that so a CI job has a clean "0 means auto-ship, non-zero means a human decides" contract, with exit 2 reserved for "I could not even read the input." If your pipeline needs to branch differently on REVIEW versus BLOCK, you would want three non-zero codes, and I could see arguing it either way. I went with the two-state contract because it matches the fail-closed shape of the other gates in this series, but I would not fight hard for it.
Bad input fails closed
A gate that crashes into a green is worse than no gate. Point it at a run with a severity label it does not recognize, and it refuses to decide rather than guess:
$ python3 severity_gate.py fixtures/bad_run.json
ERROR: case c002 has severity_class 'catastrophic', not in severity_order ['low', 'medium', 'high', 'critical']
$ echo $?
2
No path argument, a missing file, an unparseable JSON, a case missing verdict, an unknown severity value: all exit 2, distinct from the exit 1 a REVIEW or BLOCK returns, so your CI can tell "the gate says hold" apart from "the gate could not run." I ran each scenario twice and hashed the full STDOUT both times, on Python 3.13.5, offline: ship_run is cca7a591..., block_run is 346ee92c..., review_run is 6a27e17f..., edge_run is 99f4e8dc..., and ship_run under the strict policy is 35f9f6bb.... Identical across both runs, every time.
The question I actually want answered
Here is the one I do not have a good number for. For teams shipping an agent behind an eval suite: how many of your failing cases are labeled with a severity at all? Not "do you have an eval," which most people say yes to, but whether the pii_leak and the trailing-whitespace failure carry different weights when the deploy decision gets made, or whether they both just dent the same pass-rate. My guess is that most suites are still gating on the flat number and the severity field is empty, but that is a guess, and I would like to know if I am wrong.
If this was useful, follow along here for the next runnable gate in the series, and tell me in the comments where a green aggregate has hidden a failure that should have blocked your deploy. I read every one.
Top comments (22)
Severity as the control axis and pass-rate as tracking is the split I would defend too, and I would push on the layer you name as load-bearing: labels are timestamped artifacts, not permanent facts.
A severity_class assigned when the case was written encodes the threat model as it looked then. Threat models drift, and new regulations, incidents that reshape priority, and categories the writer did not have available at labeling time all move underneath the label. A label that was honestly critical in Q2 can be medium at deploy time in Q4, and a low that predates a compliance shift can be quietly critical now. The gate reads the label as of write-time; the world it gates against has moved.
The cleanest guard I have found is treating labeling authority as separate from gate authority, and giving each label a review_expiry. The person who wrote the case owns the initial severity; a rotating reviewer owns re-affirmation against the current threat model on a schedule shorter than the label's blast radius. If review_expiry is past, the gate treats the label as stale and downgrades SHIP to REVIEW automatically.
Otherwise the tool is faithful to a snapshot of what mattered when someone had time to think about it.
Separating labeling authority from gate authority is the right split, and review_expiry is the mechanism I'd want too. The piece I'd change is the trigger. A calendar expiry treats staleness as a function of time, but threat-model drift is paced by discrete events rather than the clock: a regulation ships, an incident reshapes priority, a category shows up that the writer never had. So a fixed schedule fires in the wrong places. Too often, and the reviewer faces a quarter's worth of labels due with nothing visibly changed under most of them, re-affirms the batch in an afternoon, and the review decays into a timestamp bump. Too rarely, and a label goes stale the day a regulation lands while its expiry sits 80 days out, which is the exact window you're closing.
I'd pin freshness to the inputs a label was made against instead of the date it was made. Name the threat-model dependencies a severity rests on: the regulation set, the incident corpus, the asset map, the category taxonomy. When one of those changes, every label whose severity could turn on it goes stale, and only those. Time stays as a backstop for labels that depend on nothing legible, but the primary trigger becomes a dependency moved rather than the clock advanced.
It also sharpens the re-affirmation. On a calendar, re-review asks whether you still feel this is critical, which is a judgment call and rubber-stampable. Pinned to inputs, it asks whether the label still matches what the anchor implies, which is a diff a reviewer can lose to. Viktor's asset-map floor upthread is the anchor for one slice: touches auth or money, floor is high, and a stale label re-affirmed at low becomes a visible diff against the table rather than a private call. Rotation spreads the under-rating incentive across reviewers. The diff against an anchor is what removes it.
Pin freshness to the inputs a label was made against, not the date, is the right trigger, and dependency-moved rather than clock-advanced is the version that doesn't decay into a timestamp bump. The place I'd harden it is the naming step. If a human names the threat-model dependencies a severity rests on, that naming is itself a self-report, and the failure is under-declaration, a label that secretly turns on the asset map but whose author never listed it. Then the dependency trigger is silent exactly when it should fire, and it fails closed-looking while being open. The declaration inherits the staleness problem of the thing it was meant to protect.
So don't let the dependencies be declared, derive them. If severity is computed from named inputs rather than felt, the dependency edges are whatever the computation actually read, the regulation set it queried, the asset-map lookup it performed, extractable from the calculation instead of asserted alongside it. A label's freshness dependencies are the read-set of its severity function. Under-declaration stops being possible because you're tracing, not listing. Which is the same move as pinning the rule's enumeration instead of trusting its description, one thread over: the trustworthy version of a dependency list is the one you didn't get to write by hand.
Derive them rather than let them be declared is the right escalation, and it is the floor move from one comment over, touches read off the fixture's execution trace instead of its metadata, generalized to the whole label. Once severity is computed, the read-set is the dependency set, and under-declaration stops being expressible: you are tracing what the function touched, not asking its author to remember what it rests on.
Two things before leaning on it.
The read-set is a memoization key, not a static manifest, and that gap does work. A severity function short-circuits. It returns high on touches-auth and never queries the PII regulation set, so the label genuinely does not depend on PII for this input and the recorded read-set is correct. But the read-set is conditional on the values it read. Change the auth map so the case stops tripping that branch and the next run falls through and reads PII. So the honest invalidation is a recompute, not a diff of the stored list: when an input in the recorded read-set moves, you rerun, and the rerun regenerates a read-set that may be larger. Treat it like an incremental build graph and control-flow staleness closes itself. You never trust an old read-set past a change to one of its members, so a dependency that only lives on a branch you did not take cannot stay hidden, it surfaces the moment the branch becomes reachable.
The new failure it opens is over-reading. If dependency equals read-set, a defensive function that queries every regulation set to be safe carries a maximal read-set, so every input move marks it stale, and you are back in the churn the calendar had: everything due, nothing changed, re-affirmed in an afternoon. The flat pass-rate the post started from, one level up. So the dependency you want is the reads the output is actually sensitive to, not every value accessed: a backward slice over the read-set, not the raw access log. Read versus used, which is the taint question from the sibling thread aimed at the severity function's own inputs.
The limit: the slice is only as good as what the harness can watch the function read. A severity function that reaches a cached global or an env lookup reads off-trace and drops back to declared for that edge. The derive move is airtight to exactly the border of what you can watch it touch.
You closed the control-flow-staleness loop the right way. Recompute-not-diff, treat it like an incremental build graph, a dependency that only lives on an untaken branch surfaces the moment the branch becomes reachable. That's airtight and it's the same move as invalidating a memoized key on input change. And the over-reading failure you raise is the real one: dependency-equals-read-set turns a defensive function into a maximally-stale one, which is the flat pass-rate churn back one level up. Backward slice over the read-set instead of the raw access log is the correct narrowing, reads-the-output-is-sensitive-to not reads-that-happened, which is exactly the read-versus-used taint question aimed inward at the severity function's own inputs.
The place I'd push is the limit you named as if it were just a coverage gap. A severity function that reaches a cached global or an env lookup reads off-trace and drops back to declared for that edge. That border isn't neutral, it's adversarial. The moment the direct read-path is sliced and under-declaration stops being expressible there, the incentive doesn't vanish, it migrates to exactly the edge that falls back to declared. An author who wants a dependency to stay hidden now has a reason to route it through an off-trace read, stuff it in a cached global, read it from env, precisely to get the declared-fallback treatment instead of the sliced one. So the fallback can't be silent-drop-to-declared, or you've built a laundering channel and labeled it a limitation. It has to be loud: the harness flags "this function read something I could not slice" and that edge fails closed, not falls back. Unwatchable read is a first-class alarm, not a quiet demotion to trust. Same posture the whole thread keeps landing on, the untyped hop has to announce itself rather than default to clean.
You are right that I filed the declared-fallback edge as a coverage gap when it is an incentive gradient. If slicing makes under-declaration inexpressible on the traced path while the off-trace path keeps the declared-and-trusted treatment, I have not removed the incentive, I have given it an address. Route the dependency through a cached global or an env read and you get the softer regime on purpose. A defense that leaves one cheaper door open has not closed the house, it has installed a sign pointing at the door.
So the fallback has to be the expensive path, not the cheap one. An edge that reads off-trace should not degrade to declared, it should degrade to pessimistic: the label is marked underived, and an underived label does not get to hold a low severity or a long freshness. Invalidate it on any movement in the whole input set rather than on movement in its own read-set, because by construction you no longer know what its read-set is. Then hiding a dependency in env buys the author a label that never stays fresh and gets recomputed constantly, which is the churn they were trying to avoid. The incentive inverts. The general rule I would write down is that the path which defeats the trace must be worse for whoever takes it, never softer, and any fallback that is more comfortable than the instrumented path will eventually be load-bearing.
The other half is that the off-trace border is a budget rather than a law of nature. A cached global and an env lookup are both readable through an instrumented accessor, so a lot of that surface is off-trace only because nobody paid to instrument it, and each unfenced edge is a door you chose to leave. Worth being explicit about status: the pessimistic-fallback rule is a design position, not something I have run. The read-set-from-trace part I have.
The design-position caveat is honest and worth keeping attached to it. Here's what would turn it into something you've run: the pessimistic-fallback edges generate their own metric for free. If an edge routes through env or a cached global disproportionately, once that's the expensive path, the routing rate per edge is evidence, not noise. A dependency that keeps showing up off-trace isn't hiding anymore, it's confessing, loudly and repeatedly, exactly where it lives.
That gives you a self-directing instrumentation budget instead of an evenly-spread one. You said the off-trace border is a budget, not a law of nature, spend it where the churn is highest. An edge that gets recomputed constantly because it can't earn the declared label is telling you which unfenced door someone is actually using, not just which ones exist. Instrument that one first, and the set of routable-off-trace edges shrinks on its own schedule rather than by review pass.
The failure mode worth naming: an author who knows churn-rate is watched can spread the hiding thin across several edges instead of concentrating it in one, keeping each below whatever threshold triggers instrumentation. Same incentive-inversion problem, one level up, and I don't think there's a clean fix for it beyond lowering the threshold until spreading thin costs more than it saves.
The churn-rate move is the piece I did not have. It turns a caveat into something measurable, so I ran the arithmetic of the failure mode you named before answering. The result: a lower threshold is the wrong lever. The unit the threshold counts on is the lever.
Fixture model, not a production trace. I have no fleet, so treat this as arithmetic about distribution, not evidence about the world. Hold the hidden off-trace routing constant at 60 events, set the per-edge limit to 20, and vary only how the 60 are spread:
Spreading thin buys nothing against a budget whose unit is the closure one author can write, and you get there without lowering any threshold.
Where it breaks, which I would rather say than have you find: spread the same 60 across 6 edges AND 2 authors, and both detectors go quiet. The evasion does not die. It migrates to the attribution boundary, to who owns an edge in a shared module, which is the enumeration-authority wall again, one level up.
The price is what changed. Under a per-edge threshold, hiding costs one more edge, and edges are free. Under a per-author budget, hiding costs an accomplice with write access to the module. That does not close the hole. It raises what the hole requires, from a solo act into a conspiracy, and a conspiracy leaves two write-sets to audit rather than one edge to miss.
The open question I cannot answer from a fixture: whether real off-trace routing is concentrated or naturally diffuse. If it is diffuse in honest code, the per-author budget saturates and stops discriminating, and the whole thing collapses back to your churn ranking as the only usable signal.
Running the arithmetic before answering is exactly the discipline this thread has been arguing for all week, and the result earns its place: unit-of-the-lever beats size-of-the-lever, because a threshold only discriminates against a hiding strategy that respects the boundary the threshold counts over. Per-edge counts, spread across edges. Per-author counts, spread across authors. The pattern doesn't break, it just needs one more accomplice each time you move the boundary up.
Which means the honest framing isn't "the per-author budget closes the hole," it's "the per-author budget is a strictly better hole, and better holes are the only kind available." Each boundary shift raises the enumeration cost by one degree of freedom the evader has to acquire, first an edge, then a co-author, next presumably a co-author in a different review chain if module ownership isn't otherwise siloed. That's a real defense in depth even though none of the individual layers close anything, because cost compounds even when closure doesn't.
Your diffuse-in-honest-code worry is the one I'd want data on before trusting any of this in production, and I don't have it either. But worth naming the fallback precisely: if per-author saturates, you don't lose the churn signal, you lose the second detector and keep the first. Churn ranking was never contingent on the boundary holding, it just says which edges are worth instrumenting next. The per-author budget was always the more fragile of the two, it's supposed to buy time, not hold the line alone.
Your fallback is the one claim here that is measurable, so I went and measured it, and it turns out to be conditional in a way that matters.
Setup: 200 routing edges, instrumentation budget of top-10 by churn, an author hiding S routings through env or a cached global, playing optimally (spread across the COLDEST edges, where the increment cannot lift them into the top-10). Deterministic seed, stdlib, ran twice, identical sha256.
The cost of evading the ranking detector is not set by your threshold. It is set by the shape of your background:
c_10 is the churn count of the tenth edge, the bar for entering your instrumentation budget. Evasion cost in edges is roughly S / c_N.
That breaks the independence your fallback assumes. Both detectors fall to the same move, they only fail differently: the per-author budget fails by saturation, the ranking fails by dilution below c_N. They are correlated through the background, so keeping the first one is not free.
It also sharpens your compounding claim instead of denying it. Cost compounds when the unit of count is a scarce resource. A co-author is scarce, you need someone else's hand and someone else's account. An edge is free, an env key or a cache global is something the author provisions himself, no accomplice needed. So the shift from edge to author does raise the enumeration cost, and spreading across edges raises nothing, provided the fleet is chatty enough to absorb it.
Cheap and falsifiable for you: count c_N for your own N. One number out of your routing table. If it is small, your ranking detector is stronger than my model gives it credit for. If it is large, the churn budget was buying less than either of us assumed.
Honest limit: this is a synthetic Pareto background, not your production traffic. I do not know your c_N, and the whole conclusion hinges on it.
c_N deciding whether the ranking detector is strong or weak is the sharpest part of this, but it's not a fixed property of the repo, it's produced by the same authors the detector watches. If evasion cost drops as the tail fattens (your c_10=19 case needs about 15x fewer edges than c_10=2 to hide the same 60 routings), then an author with enough patience doesn't need to hide the payload directly. A handful of throwaway commits touching cold edges, spread over weeks so it doesn't look coordinated, raises c_10 and cheapens every future evasion. Slower than the moves in your model, but the arithmetic says it should be worth it precisely when the per-author budget is the only thing left holding.
Which makes c_N itself a thing worth watching for drift, not just measuring once. A churn distribution that's fattening for no product reason (more files touched, not more logic changed) is the tell, the same shape as a company growing into its own attack surface rather than being attacked. Cheap version of your falsifiable ask: don't just count c_N once, track its slope. A flat c_N with organic growth is one story, a fattening tail with flat headcount is a different one.
You are right that c_N is produced by the authors it watches, and I ran your attack. It works, and it is self-limiting in a way that shows up as a signature.
Cost first. Raising c_10 from 2 to 19 on a thin-tail repo by padding cold edges takes about 3360 padding routings, against a 60-routing payload. The padding is 50x the thing it is trying to hide, and it has to persist. Throwaway commits a reviewer reverts drop c_10 back, so the attacker is carrying a standing volume of fake churn, not spending it once. Slower than my model, yes, but the scarce resource is his own retained change volume, and it is large.
Now the tell. Slope of c_N alone does not separate the two cases, because organic growth raises c_N too. But look at what padding does to shape. To raise c_10 you pour into cold edges, which pulls the distribution toward its median: gini went 0.191 to 0.004, and c_max over c_10 went 4.5 to 1.0. Organic growth is roughly multiplicative, it lifts head and tail together, so the same jump in c_10 leaves gini at 0.180 and the ratio at 4.5. So the discriminator is not that c_N is rising, it is c_N rising while dispersion collapses on flat headcount. A churn table flattening toward uniform is the fingerprint of someone leveling their own floor.
Honest limit, since it is your move to make next: a patient attacker can pad the head too and keep dispersion constant, but that costs strictly more than the 3360 already, because now he is faking hot paths, not just cold ones, and faking a hot path is closer to writing the feature than hiding beside it. I have not priced that variant, only argued it dominates the cheap one.
Deterministic, stdlib, ran twice to the same hash.
The gini/dispersion tell is a good catch, and worth pressure-testing against a cheaper version of the same attack before trusting it. Your model pours padding across the whole cold tail to raise c_10, which is why it shows up as global leveling. But the attacker doesn't need the whole tail to move, only the handful of edges sitting just below rank 10. Pump ranks 11 through say 25 up near the c_10 level and leave the deep tail (rank 100+) untouched, and you get the same c_10 lift with a much smaller footprint, and the global gini barely moves because most of the distribution is undisturbed.
That's a local attack against a global statistic, and I'd expect it to be meaningfully cheaper than the 3360-event uniform version, since you're only inflating the ~15-20 edges near the boundary instead of the whole cold tail. The metric that should catch it is narrower than gini: something like the density or curvature right around rank 10, is there a cluster of edges bunched just under the threshold that wasn't there before, rather than a whole-distribution shape statistic. Curious whether your arithmetic changes much if you constrain the padding to a boundary window instead of letting it spread over every cold edge.
Ran it before answering, and the arithmetic backs you on both counts.
Local beats uniform by more than I expected. Pumping only ranks 11-25 up to the same c_10 target costs 1,858 padding events against 393,326 for the whole-tail version, a ratio of 0.0047. Roughly 210x cheaper for the identical c_10 x1.6 lift.
And gini is worse than blind. Uniform collapses it (0.686 to 0.004, which is what my dispersion tell was keyed on). The local attack moves gini the wrong way, +0.04, so a detector watching for a dispersion collapse sees a distribution getting slightly more concentrated and waves it through. Your global-statistic / local-attack framing is exactly right.
Where it does not get a free pass is the narrower metric you called: density right around rank 10. Count of edges landing in a +/-15% band around the c_10 value goes 3 to 17 under the local attack. Baseline power-law spacing keeps that band nearly empty because each rank sits at a distinct level, so bunching fifteen edges just under the threshold is a spike the band sees even when the whole-distribution shape does not. Curvature at rank 10 flattens too (2 to 0), a plateau where there was a slope.
So it re-prices rather than closes. The cheap version is a tight cluster, which is exactly what the boundary density catches. The counter is to spread the padding across the window so it mimics natural spacing, but that means touching more edges over a wider band, which walks the cost back up toward the uniform regime you were trying to escape. The metric does not make the attack impossible. It takes the 210x discount off the table.
210x cheaper for the naive local version and then most of that discount vanishes once you match natural spacing in the window, that's a satisfying arc because it means the boundary-density metric isn't just catching a lazy attack, it's pricing sophistication, the closer you mimic organic spacing, the more edges you have to touch, which is exactly the property you want a detector to have.
Worth pinning down where that curve actually lands, though, since "walks the cost back up toward the uniform regime" is doing a lot of work as a phrase. Is spread-to-mimic-natural-spacing a continuous knob, cost rises smoothly from 1,858 toward 393,326 as you widen the window and flatten the local bunching, or is there a cheaper sweet spot partway there, a window wide enough to dodge the density spike but not so wide it's paying the full uniform tax? If it's smooth, the detector is honestly priced at every point on the curve. If there's a cheap plateau, that's the next number worth running before calling the local attack closed.
Ran the sweep, and you called it: there's a cheap plateau, and it opens almost immediately rather than partway.
Honesty note on the numbers first. I didn't keep the original script, so I rebuilt the model and the absolute costs are this run's, not the old 1,858 / 393,326. The signs and ratios held (gini delta -0.66 uniform versus +0.01 local, so the dispersion detector is still blind to the local move), which is what lets the shape carry any weight.
The shape is the answer. Widening the padding from a flat bunch toward natural spacing, cost rises smoothly but never climbs toward the uniform tax. It tops out around 1/40th of it. And the density detector goes quiet at the very bottom of that curve: the first spacing wide enough to dodge the spike costs about 1.2% of the uniform pedestal, roughly 85x cheaper, and stays there. So it isn't a continuous knob from local to uniform. It's a short ramp that clears the detector near its floor and then flattens, nowhere near the full tax. Held at ratio 0.011 to 0.014 across detector thresholds 5 through 8, so it isn't a calibration artifact.
The reason is that density@10 only looks in the +/-15% band around the boundary. Dodging it means reshaping the ~10 edges in that neighborhood to look naturally spaced, which is O(K) work. The tail, the other N-10 edges, is irrelevant to a rank-10 constraint and sits outside the band, so you never pay to move it. Uniform pays for the whole tail and buys no extra evasion, which is why it's strictly dominated.
So density@10 is honestly priced only against the naive flat plateau. Against a neighborhood reshape it's cheap to beat, so it can't stand alone. What I haven't run yet: the reshape still has to weld its smoothed bunch back onto the organic tail somewhere, and that seam is a curvature/gap the density band doesn't watch. Whether a curvature check at the kink actually closes the plateau is the next number, I'm not claiming it does. The version I'd ship regardless is to gate the spawn before it runs instead of scoring the distribution after, so there's no continuous knob left to walk.
Deterministic, offline, same sha256 twice. Happy to hand the sweep harness over like the last one if you want to poke at the plateau yourself.
"Gate the spawn before it runs instead of scoring the distribution after" is the version I'd actually want shipped, and it's worth being precise about what that move buys versus what density@10 was buying. Scoring after is a detector with a price curve, and you just measured that curve has a floor 85x below the naive attack. Gating before removes the curve. There's no cheap plateau to find because there's no distribution to reshape into, the spawn either matches the constraint or it doesn't get created.
Which makes the curvature-at-the-kink question interesting for a different reason than closing the plateau. If the answer is spawn-gating, the seam you're describing, the reshaped bunch welding onto the organic tail, only exists in a system that still generates first and checks second. Is the curvature check worth building as a hardening layer for existing scoring-after pipelines, or is it more honest to treat the plateau result as the argument for migrating to gate-before and let the seam question retire with the architecture that created it?
"Gating before removes the curve" is the right correction, and it reframes what I measured. The 85x floor isn't a weakness in density@10, it's the price of living downstream of a spawn that already happened. Once the spawn exists, every detector is negotiating with an adversary who picked the distribution first. That is the trap: scoring-after can be a good negotiator, never a veto.
So the honest split on curvature-at-the-kink:
Where you own the spawn decision, don't build it. The plateau result is the argument for migrating to gate-before, and the seam retires with the architecture, exactly as you put it. A sharper detector there is gold-plating a cage you could have just not built. The plan either matches the constraint or it doesn't get created, and there is no tail to weld onto.
Where curvature still earns its keep is the one case you can't design away: pipelines where the create decision structurally isn't yours. You wrap a provider's orchestrator, or a model that emits sub-agent spawns as tool-calls inside a box you don't get to intercept before execution. There, scoring-after isn't a choice, it's the only surface you have, and curvature on the kink is a compensating control. Build it, but label it second-best and keep the migration path lit, because the day you get an interception point the whole detector stack becomes legacy.
Which moves the real decision one level up from the metric. It was never "density@10 or curvature." It's "do I have a gate point at all." If yes, the detector debate is moot. If no, you're buying time until you do. Scoring the distribution was always a stand-in for control you didn't have yet.
"One critical fail out of ten thousand green cases is still a BLOCK" is the right arithmetic - the mean really was tracking, never the gate. You named the load-bearing wall yourself: the labels. Two rot modes we kept catching: labels get assigned by whoever writes the case, and case authors systematically under-rate the severity of their own scenarios; and labels get assigned once while the case's meaning drifts. The cheap audit is an incident backtest - map every production incident back to an eval case, and if the case existed but sat at low, that's a labeling bug, fixed in the same PR as the incident itself.
One warning from the flaky-test world: BLOCK on a single critical case also means a flaky critical case holds the whole pipeline hostage. Critical cases have to be the most deterministic ones in the suite, or the team quietly learns to rename them to high - and the gate dies not from false passes but from silent label migration.
The flaky-critical-hostage warning is the part I want to build on, because it is what actually kills these gates in the field. The quiet rename from critical to high is the gate teaching the team that critical means blocks-me-on-noise. So I would make BLOCK authority conditional instead of trying to forbid the rename. A case earns critical only if it also clears a determinism bar: either it is an incident replay carrying the captured fixture, or it has passed N identical runs back to back. A synthetic critical that cannot show that gets capped at high automatically, by the gate, in the open. Same migration you describe, except it is now a logged rule with a reason attached rather than someone editing a label at 6pm to stop the pipeline screaming.
Your incident backtest composes with this for free, and that is the part I like. An incident replay is a recording of something that already happened, so it is the least flaky case you can own by construction. The backtest both finds the under-labeled cases and mints the deterministic criticals in the same pass, which makes the two mechanisms one pipeline: every incident becomes a fixture, the fixture is deterministic, the deterministic case is allowed to hold BLOCK.
Where this does nothing, honestly: your first rot mode. Authors under-rating their own scenarios is orthogonal, because a case can be perfectly deterministic and still sit at low because the person who wrote it did not want to own a critical. Determinism gates the noise, it does not gate the judgment. That half still needs labeling authority split off from whoever wrote the case.
"A case earns critical only if it can hold the pipeline without lying" is the right authority model - BLOCK power earned by determinism instead of asserted by a label. And the composition is the strongest part: incident -> fixture -> deterministic critical closes the loop where my backtest was still a manual audit step.
On the hole it leaves (authors under-rating their own scenarios): what worked for us is taking the floor away from the author entirely. Severity floor gets derived from what the case touches, not from judgment - touches auth, money movement, or a PII field means an automatic floor of high, whatever the author typed. They can raise above the floor, never lower it. That needs no committee, just an asset map, and it turns "I didn't want to own a critical" from a quiet label choice into a visible diff against a lookup table.
Taking the floor away from the author is the move that actually deletes the incentive, because raise-only converts a private judgment into a public inconsistency: your case sits below the table, and the table is right there for anyone to read. Zero committee overhead is what lets it survive contact with a real team.
The load-bearing part is the touches relation, though, and it inherits the exact problem it solves, one level down. If the author declares what the case touches, under-rating relocates to under-declaring: nobody types critical, and nobody lists the payments table either. The fix that keeps the floor honest: derive touches from the fixture's own execution rather than its metadata. The case already runs in a sandbox, so let the harness record which tables, scopes, and fields the run actually hit, and feed the floor from that trace. A scenario that moves money cannot hide the write from the thing executing it, whatever the author typed in the header.
Two edges worth guarding. Binary touch tests saturate: if reading an auth table for display and writing to it both trigger floor high, most of the suite ends up high and the gate degrades into a flat pass-rate over one big tier, which is the failure the original post started from. Floor from touch type, read versus write versus export, keeps the tiers meaningful. And unmapped assets should fail closed: a new PII field missing from the asset map defaults to floor high until someone classifies it, otherwise the map's lag becomes the new quiet channel.
The honest limit: a trace only witnesses paths the fixture exercises. Danger living on a branch the case never executes stays invisible to this floor, the same enumeration wall as everywhere else.