Every agent I have ever shipped was qualified the same way: someone watched it work once, nodded, and called it production-ready. I did this for y...
For further actions, you may consider blocking this person and/or reporting abuse
The line that production certification is not the staging threshold with a bigger number holds up better than the thresholds do at these corpus sizes. On four tasks the pass rate moves in steps of 0.25, so 0.80 and 0.90 both mean four out of four, and with the same latency budget and no critical failures allowed on either rung, the judge graph gets the same verdict from staging and production for every result it could produce. On six tasks the rungs disagree on one outcome only, five out of six.
In the code, what separates the rungs is the survival gate, 50 completed production runs by default and 0 in the field-test profile. Reading the code after the fix that moved that count server-side, I do not think the gate can be passed by default. The counter moves only when a run completes in production, production admits only certified workloads, and the move to provisional resets the counter to zero. A provisional workload is refused the runs it needs to stop being provisional, and the gate test passes because it increments the counter directly rather than completing a run.
The check is short: default profile, certify at staging, certify at production and watch it defer, submit one production run and get the 403, read the counter. If it holds, the design doc already names the missing piece, runs survived while provisional, so either admission needs a mode that lets a provisional workload into production under that count, or the count has to accumulate somewhere else.
Thanks — this is a careful read, and it's the exact concern the design was built to avoid. The counter is server-side (
increment_production_runs) and does reset to zero onprovisional/quarantined, but a survival-gated provisional workload is explicitly still admissible to production — otherwise, as you say, the run that counts could never happen. There's a regression for precisely that path (test_workload_reaches_certified_with_default_survival_gate), and it asserts admission isTruewhile provisional. Your short check (default profile, certify staging, certify production and watch it defer, submit one run, read the counter) is a good end-to-end probe — if you're seeing that 403 on a default install, I genuinely want the workflow log.That regression and the provisional branch in check_admission both arrived with 0.2.0. My comment read the 0.1.0 code, main as it stood on 28 September, where production admission still required a certified status and the test was not yet in the tree, so there is no log from a current install to send. On 0.2.0 the path looks closed to me.
One hop is still covered only by reading the code. The two workflow tests that reach certified under a nonzero gate move the counter by calling increment_production_runs directly, and the field-test profile sets the gate to zero, so no test drives a provisional workload through a completed production run and reads the counter back. That is the same short check, now expecting an admitted run and a counter of one. The 0.2.0 changelog does not mention the admission change while 0.1.0 is still on PyPI, so one line there would tell anyone on 0.1.0 that the refusal is fixed.
The sentence doing the work is that a manifest change invalidates the old certification. Most gates I have seen treat promotion as a one-time event and then let the thing underneath drift, which means the attestation is a statement about a version that no longer exists. Binding it to model identity and re-verifying on read is the version that survives a model swap someone made on a Friday.
Where I would push is what the manifest does not contain. Owner, adapter, tools, model - all yours. The upstream services the agent calls are not, and they change without telling you. Same prompt, same model, same tool list, a payment API that started returning a new field, and the certification is still green because nothing inside it moved. That is the failure I keep writing about: the check did not fail, it stopped being about the thing it was checking.
Which makes 10/10 the number I would be least comfortable with, and you are already halfway to my point by saying the refusals were the interesting part. A gate that has never blocked a promotion someone actually wanted is the same evidence as a gate that cannot block one. Your four bad fixtures are the right instrument. What I would add is a block rate on real traffic tracked over time, with zero treated as an alarm rather than a pass.
One question, because it is the part I have not solved either. When re-certification fails and the workload quarantines, who is allowed to override it, and does the override land in the attestation chain? Every gate I have watched eventually grows a person who can say yes, and that person is usually where the audit trail stops.
The sentence that lands is the same one that worried me for years: an agent is "production-ready" because someone watched a demo.
A certification ladder (uncertified → provisional → certified, with manifest changes invalidating the attestation) is the right shape. Where I'd push next is what the corpus is allowed to omit.
If the benchmark only scores calm-sky happy paths, the attestation is a statement about a sky that never storms. I'd want at least one arm where:
Upstream drift (arun's point) is the cousin of that: same prompt, same model, same tool list, a dependency that changed shape — certification stays green because nothing inside the manifest moved. Treating a long streak of zero real blocks as an alarm, not a pass, is how you keep the gate honest.
Refusals with named, actionable reasons > "Forbidden." That's the product.
Thanks — "the check did not fail, it stopped being about the thing it was checking" is the sharpest statement of the drift problem I've seen, and it's now a real subsystem: the drift detector shipped in v0.2.0, plus the weekly fleet digest reports quarantined/reinstated/drifting workloads. On the override question — the honest answer is that today the human gate is the approval path for destructive tools (pause → approve → resume), and every refusal and intervention lands in the tamper-evident audit chain, but a dedicated certification override that records into the attestation chain isn't something I've designed yet. You and I both know every gate grows a person who can say yes; that person should be in the chain, and right now that's a gap.
"There is nothing to check" is the real diagnosis. I hit the same wall at the code level: agents working in a large project would pass every demo and still quietly break the architecture. What changed things was the same idea as your ladder: a gate the agent must pass after every change, deterministic, with no appeal. Once the gate exists, agents start satisfying it on the first try. Question about HivePlane: how do you keep the benchmark itself from going stale as the agent's real workload changes? That's the part I find hardest.
Thanks — that's the hardest open question in the whole thing, and I won't pretend it's solved. The corpus is versioned with the manifest, so changing an agent's benchmark is itself a manifest change that forces re-certification; the drift detector (v0.2.0) catches the workload fading away from its certification. But a benchmark that no longer reflects the real workload is a subtler failure than drift, and the honest answer is that today it depends on me noticing. Your framing — the gate the agent must pass after every change — is right; the missing piece is who keeps the gate honest as the work moves.
Agent demos can be so misleading! They always work perfectly in the demo but break in real life. Great that you built a gate to catch this.
For anyone testing AI agents, I've been using jzstoken for my API keys. Makes it easy to test different models without signing up for each one separately. They give you $5 free to start, so you can run real-world tests without any upfront cost.
This really resonates, because agent demos often optimize for the happy path while production systems need a reliable way to reject low-confidence or risky actions. A "gate that says no" is usually what turns an impressive demo into something people can actually trust, especially when decisions have real-world consequences. I wrote a guide on this broader shift toward automation and oversight: gumreads.gumroad.com/l/bgykuc. Curious what signals or thresholds you've found most effective for deciding when the gate should intervene.
The shift from "did it pass the demo" to "is the current artifact certified against the current manifest" is a clean boundary most teams never draw explicitly. The 403 as product output rather than error handling is the part that stuck with me: you're encoding the trust boundary in the protocol itself, not in a post hoc policy document. The HivePlane attestation layer also removes the "but it worked last week" defense because any manifest change voids the cert.
Have you run into cases where the certification policy itself needed to be versioned separately from the manifest, or does keeping them coupled give you the property you actually want?
Thanks — encoding the boundary in the 403 rather than a policy doc is the part I'm most attached to. On versioning: the manifest currently carries the certification corpus and thresholds, so policy and workload are coupled by design — change either and the old attestation is invalid. That coupling is what gives you "no silent path back into production," but you've found its cost: a policy change re-certifies everything even when the workload didn't move. I suspect the right answer is a separate policy version referenced by the attestation, so you get provenance without a blanket re-cert, and I'd rather name that as unsolved than pretend it's settled.