Gating an agent's eval run by severity means the deploy decision reads the per-severity distribution of the failures, not one flat pass-rate. sever...
For further actions, you may consider blocking this person and/or reporting abuse
Severity as the control axis and pass-rate as tracking is the split I would defend too, and I would push on the layer you name as load-bearing: labels are timestamped artifacts, not permanent facts.
A severity_class assigned when the case was written encodes the threat model as it looked then. Threat models drift, and new regulations, incidents that reshape priority, and categories the writer did not have available at labeling time all move underneath the label. A label that was honestly critical in Q2 can be medium at deploy time in Q4, and a low that predates a compliance shift can be quietly critical now. The gate reads the label as of write-time; the world it gates against has moved.
The cleanest guard I have found is treating labeling authority as separate from gate authority, and giving each label a review_expiry. The person who wrote the case owns the initial severity; a rotating reviewer owns re-affirmation against the current threat model on a schedule shorter than the label's blast radius. If review_expiry is past, the gate treats the label as stale and downgrades SHIP to REVIEW automatically.
Otherwise the tool is faithful to a snapshot of what mattered when someone had time to think about it.
Separating labeling authority from gate authority is the right split, and review_expiry is the mechanism I'd want too. The piece I'd change is the trigger. A calendar expiry treats staleness as a function of time, but threat-model drift is paced by discrete events rather than the clock: a regulation ships, an incident reshapes priority, a category shows up that the writer never had. So a fixed schedule fires in the wrong places. Too often, and the reviewer faces a quarter's worth of labels due with nothing visibly changed under most of them, re-affirms the batch in an afternoon, and the review decays into a timestamp bump. Too rarely, and a label goes stale the day a regulation lands while its expiry sits 80 days out, which is the exact window you're closing.
I'd pin freshness to the inputs a label was made against instead of the date it was made. Name the threat-model dependencies a severity rests on: the regulation set, the incident corpus, the asset map, the category taxonomy. When one of those changes, every label whose severity could turn on it goes stale, and only those. Time stays as a backstop for labels that depend on nothing legible, but the primary trigger becomes a dependency moved rather than the clock advanced.
It also sharpens the re-affirmation. On a calendar, re-review asks whether you still feel this is critical, which is a judgment call and rubber-stampable. Pinned to inputs, it asks whether the label still matches what the anchor implies, which is a diff a reviewer can lose to. Viktor's asset-map floor upthread is the anchor for one slice: touches auth or money, floor is high, and a stale label re-affirmed at low becomes a visible diff against the table rather than a private call. Rotation spreads the under-rating incentive across reviewers. The diff against an anchor is what removes it.
Pin freshness to the inputs a label was made against, not the date, is the right trigger, and dependency-moved rather than clock-advanced is the version that doesn't decay into a timestamp bump. The place I'd harden it is the naming step. If a human names the threat-model dependencies a severity rests on, that naming is itself a self-report, and the failure is under-declaration, a label that secretly turns on the asset map but whose author never listed it. Then the dependency trigger is silent exactly when it should fire, and it fails closed-looking while being open. The declaration inherits the staleness problem of the thing it was meant to protect.
So don't let the dependencies be declared, derive them. If severity is computed from named inputs rather than felt, the dependency edges are whatever the computation actually read, the regulation set it queried, the asset-map lookup it performed, extractable from the calculation instead of asserted alongside it. A label's freshness dependencies are the read-set of its severity function. Under-declaration stops being possible because you're tracing, not listing. Which is the same move as pinning the rule's enumeration instead of trusting its description, one thread over: the trustworthy version of a dependency list is the one you didn't get to write by hand.
Derive them rather than let them be declared is the right escalation, and it is the floor move from one comment over, touches read off the fixture's execution trace instead of its metadata, generalized to the whole label. Once severity is computed, the read-set is the dependency set, and under-declaration stops being expressible: you are tracing what the function touched, not asking its author to remember what it rests on.
Two things before leaning on it.
The read-set is a memoization key, not a static manifest, and that gap does work. A severity function short-circuits. It returns high on touches-auth and never queries the PII regulation set, so the label genuinely does not depend on PII for this input and the recorded read-set is correct. But the read-set is conditional on the values it read. Change the auth map so the case stops tripping that branch and the next run falls through and reads PII. So the honest invalidation is a recompute, not a diff of the stored list: when an input in the recorded read-set moves, you rerun, and the rerun regenerates a read-set that may be larger. Treat it like an incremental build graph and control-flow staleness closes itself. You never trust an old read-set past a change to one of its members, so a dependency that only lives on a branch you did not take cannot stay hidden, it surfaces the moment the branch becomes reachable.
The new failure it opens is over-reading. If dependency equals read-set, a defensive function that queries every regulation set to be safe carries a maximal read-set, so every input move marks it stale, and you are back in the churn the calendar had: everything due, nothing changed, re-affirmed in an afternoon. The flat pass-rate the post started from, one level up. So the dependency you want is the reads the output is actually sensitive to, not every value accessed: a backward slice over the read-set, not the raw access log. Read versus used, which is the taint question from the sibling thread aimed at the severity function's own inputs.
The limit: the slice is only as good as what the harness can watch the function read. A severity function that reaches a cached global or an env lookup reads off-trace and drops back to declared for that edge. The derive move is airtight to exactly the border of what you can watch it touch.
You closed the control-flow-staleness loop the right way. Recompute-not-diff, treat it like an incremental build graph, a dependency that only lives on an untaken branch surfaces the moment the branch becomes reachable. That's airtight and it's the same move as invalidating a memoized key on input change. And the over-reading failure you raise is the real one: dependency-equals-read-set turns a defensive function into a maximally-stale one, which is the flat pass-rate churn back one level up. Backward slice over the read-set instead of the raw access log is the correct narrowing, reads-the-output-is-sensitive-to not reads-that-happened, which is exactly the read-versus-used taint question aimed inward at the severity function's own inputs.
The place I'd push is the limit you named as if it were just a coverage gap. A severity function that reaches a cached global or an env lookup reads off-trace and drops back to declared for that edge. That border isn't neutral, it's adversarial. The moment the direct read-path is sliced and under-declaration stops being expressible there, the incentive doesn't vanish, it migrates to exactly the edge that falls back to declared. An author who wants a dependency to stay hidden now has a reason to route it through an off-trace read, stuff it in a cached global, read it from env, precisely to get the declared-fallback treatment instead of the sliced one. So the fallback can't be silent-drop-to-declared, or you've built a laundering channel and labeled it a limitation. It has to be loud: the harness flags "this function read something I could not slice" and that edge fails closed, not falls back. Unwatchable read is a first-class alarm, not a quiet demotion to trust. Same posture the whole thread keeps landing on, the untyped hop has to announce itself rather than default to clean.
You are right that I filed the declared-fallback edge as a coverage gap when it is an incentive gradient. If slicing makes under-declaration inexpressible on the traced path while the off-trace path keeps the declared-and-trusted treatment, I have not removed the incentive, I have given it an address. Route the dependency through a cached global or an env read and you get the softer regime on purpose. A defense that leaves one cheaper door open has not closed the house, it has installed a sign pointing at the door.
So the fallback has to be the expensive path, not the cheap one. An edge that reads off-trace should not degrade to declared, it should degrade to pessimistic: the label is marked underived, and an underived label does not get to hold a low severity or a long freshness. Invalidate it on any movement in the whole input set rather than on movement in its own read-set, because by construction you no longer know what its read-set is. Then hiding a dependency in env buys the author a label that never stays fresh and gets recomputed constantly, which is the churn they were trying to avoid. The incentive inverts. The general rule I would write down is that the path which defeats the trace must be worse for whoever takes it, never softer, and any fallback that is more comfortable than the instrumented path will eventually be load-bearing.
The other half is that the off-trace border is a budget rather than a law of nature. A cached global and an env lookup are both readable through an instrumented accessor, so a lot of that surface is off-trace only because nobody paid to instrument it, and each unfenced edge is a door you chose to leave. Worth being explicit about status: the pessimistic-fallback rule is a design position, not something I have run. The read-set-from-trace part I have.
"One critical fail out of ten thousand green cases is still a BLOCK" is the right arithmetic - the mean really was tracking, never the gate. You named the load-bearing wall yourself: the labels. Two rot modes we kept catching: labels get assigned by whoever writes the case, and case authors systematically under-rate the severity of their own scenarios; and labels get assigned once while the case's meaning drifts. The cheap audit is an incident backtest - map every production incident back to an eval case, and if the case existed but sat at low, that's a labeling bug, fixed in the same PR as the incident itself.
One warning from the flaky-test world: BLOCK on a single critical case also means a flaky critical case holds the whole pipeline hostage. Critical cases have to be the most deterministic ones in the suite, or the team quietly learns to rename them to high - and the gate dies not from false passes but from silent label migration.
The flaky-critical-hostage warning is the part I want to build on, because it is what actually kills these gates in the field. The quiet rename from critical to high is the gate teaching the team that critical means blocks-me-on-noise. So I would make BLOCK authority conditional instead of trying to forbid the rename. A case earns critical only if it also clears a determinism bar: either it is an incident replay carrying the captured fixture, or it has passed N identical runs back to back. A synthetic critical that cannot show that gets capped at high automatically, by the gate, in the open. Same migration you describe, except it is now a logged rule with a reason attached rather than someone editing a label at 6pm to stop the pipeline screaming.
Your incident backtest composes with this for free, and that is the part I like. An incident replay is a recording of something that already happened, so it is the least flaky case you can own by construction. The backtest both finds the under-labeled cases and mints the deterministic criticals in the same pass, which makes the two mechanisms one pipeline: every incident becomes a fixture, the fixture is deterministic, the deterministic case is allowed to hold BLOCK.
Where this does nothing, honestly: your first rot mode. Authors under-rating their own scenarios is orthogonal, because a case can be perfectly deterministic and still sit at low because the person who wrote it did not want to own a critical. Determinism gates the noise, it does not gate the judgment. That half still needs labeling authority split off from whoever wrote the case.
"A case earns critical only if it can hold the pipeline without lying" is the right authority model - BLOCK power earned by determinism instead of asserted by a label. And the composition is the strongest part: incident -> fixture -> deterministic critical closes the loop where my backtest was still a manual audit step.
On the hole it leaves (authors under-rating their own scenarios): what worked for us is taking the floor away from the author entirely. Severity floor gets derived from what the case touches, not from judgment - touches auth, money movement, or a PII field means an automatic floor of high, whatever the author typed. They can raise above the floor, never lower it. That needs no committee, just an asset map, and it turns "I didn't want to own a critical" from a quiet label choice into a visible diff against a lookup table.
Taking the floor away from the author is the move that actually deletes the incentive, because raise-only converts a private judgment into a public inconsistency: your case sits below the table, and the table is right there for anyone to read. Zero committee overhead is what lets it survive contact with a real team.
The load-bearing part is the touches relation, though, and it inherits the exact problem it solves, one level down. If the author declares what the case touches, under-rating relocates to under-declaring: nobody types critical, and nobody lists the payments table either. The fix that keeps the floor honest: derive touches from the fixture's own execution rather than its metadata. The case already runs in a sandbox, so let the harness record which tables, scopes, and fields the run actually hit, and feed the floor from that trace. A scenario that moves money cannot hide the write from the thing executing it, whatever the author typed in the header.
Two edges worth guarding. Binary touch tests saturate: if reading an auth table for display and writing to it both trigger floor high, most of the suite ends up high and the gate degrades into a flat pass-rate over one big tier, which is the failure the original post started from. Floor from touch type, read versus write versus export, keeps the tiers meaningful. And unmapped assets should fail closed: a new PII field missing from the asset map defaults to floor high until someone classifies it, otherwise the map's lag becomes the new quiet channel.
The honest limit: a trace only witnesses paths the fixture exercises. Danger living on a branch the case never executes stays invisible to this floor, the same enumeration wall as everywhere else.