A success gate verifies an AI agent's claimed success before your system accepts it. SuccessGate runs three read-only checks — schema/contract, claim-vs-evidence against the actual tool-call trace, and an optional post-condition probe — and turns a silent 200 into an explicit REJECTED with reasons. It's stdlib Python, needs no API key, moves nothing, and ships with a self-test you run in one command.
Here's the failure that started this for me. An agent in a CRM workflow reported {"status": "sent", ...} for an invoice. Clean run. Green dashboard. 200 OK. The invoice went to a customer id that wasn't on our allow-list — a near-miss hallucination the model was completely sure about. Nothing crashed. No exception, no stack trace. We found it days later, downstream, the expensive way.
That's not a rare bug. It's the default failure mode of agents in production, and it has a name now: silent-success drift. Cycles' writeup put it bluntly — "200 OK Is the Most Dangerous Response in Production": "The most dangerous failures look like success." And the measurements back it up. The Berkeley Function-Calling Leaderboard (BFCL v3) puts frontier-model structurally invalid tool calls at 2–5% even on clean benchmark prompts — higher in noisy production (via Future AGI). The arXiv paper Agent Behavioral Contracts reports that across 1,980 sessions, contracted agents caught 5.2–6.8 soft violations per session that uncontracted baselines missed entirely.
So the question is not how do I see failures sooner. It's how do I stop accepting a success the agent never actually achieved.
TL;DR
- A 200 and a
"status": "done"are claims, not proof. Agents return both while doing the wrong thing — or nothing. - Observability is tracking: it tells you a call happened. It can't tell you the result was correct. That's a control problem.
-
verify()runs three checks before you accept success: (1) schema/contract (shape, types, enums/allow-list), (2) claim-vs-evidence (did the agent actually call the tool it says it used?), (3) optional read-only post-condition probe (is the effect really there?). - Stdlib only;
requestsoptional and only for the level-3 probe. No keys, no funds, no blockchain. - Built-in self-test on 4 fixtures:
python success_gate.py. No agent, no key. - This is a gate, not an eval suite. Limits at the bottom.
This is the third piece in a small series on controlling agents before they execute, not after. Part one was a hard pre-execution spend-cap for runaway loops — don't let the agent spend. Part two was a pre-send transaction canary — don't let it send a bad transaction. This one is the third point on the same pre-execution layer: don't accept a success that isn't real. Spend, send, accept — same idea, three places.
Your dashboard tracks. It doesn't control.
Every agent-observability tool I've used does the same useful thing: it records what happened. Spans, traces, a timeline of tool calls, latency, a green checkmark when the run returns 200. I'm not knocking it — you need it. But look at what it actually knows.
It knows a call was made. It does not know the call did the right thing.
There's a survey number that captures the gap. A 2026 poll of 1,300+ AI professionals found 89% of organizations have observability in place, but only 62% can actually inspect what their agents do at each step (Cycles). Tracking is near-universal. The ability to check the result lags by a third of the field. That delta is exactly where silent-success drift lives.
And it compounds. The arithmetic everyone quotes is real: at 95% per-step correctness, a 10-step workflow lands right only ~60% of the time; at 85%, about 20%. Structured output helps a lot — Agentmelt's production data puts action-success at 95–99% with structured output versus 70–85% parsing unstructured text (Agentmelt). But notice what structured output fixes: it guarantees the shape. It says nothing about whether the agent did the thing the shape describes. A perfectly-formed {"status": "updated"} from an agent that never called update is exactly as well-formed as an honest one.
So you can have observability, structured output, and a green dashboard, and still ship a lie. The missing piece is a step that sits between "agent finished" and "we accept it" and asks: prove it.
The shape of the fix: gate before accept
The idea fits in a sentence. Before you treat an agent's result as success, run it past a contract — and if it can't back up its claim, mark it REJECTED instead of letting a silent 200 through.
Three checks, deliberately asymmetric, because the three ways a result lies are different:
-
Check 1 — schema / contract. Is the output even shaped like success? Required fields present, right types, and values inside the allowed set — including an allow-list for things like
target_id. This is the cheap one that catches the BFCL 2–5%: malformed JSON, a missing field, an id that was never permitted. The invoice-to-cust_9999near-miss dies here. -
Check 2 — claim vs evidence. Did the agent do what it says it did? Walk the actual tool-call trace. If the result claims
"updated"butupdate_recordwas never called, that's a claim without an action. This is the check that catches the agent narrating an effect it never produced — the failure observability literally cannot see, because no failing call ever happened. -
Check 3 — post-condition probe (optional, read-only). Is the effect actually there? You hand it a small read-only function — a
GETthat reads the record back, a query that counts the row. If it can't confirm the effect,REJECTED. This is the only check that touches the network, and it's optional precisely because it needs a read-only way to verify.
The point of running all applicable checks and collecting every reason — instead of bailing on the first — is that a drifted result usually trips more than one. You want the full report, not the first red flag.
SuccessGate — the whole thing
Stdlib for the first two checks, so it runs on a bare machine with nothing installed. requests is imported lazily and only the optional level-3 probe would use it — and even then, the probe is your read-only function, not something the tool invents. No keys. No funds. No blockchain. It reads; it never writes.
#!/usr/bin/env python3
"""SuccessGate - verify an AI agent's claimed success BEFORE you accept it.
Three asymmetric checks between "the agent says it's done" and "we trust it":
1. schema/contract - shaped right (required fields, types, enums)
2. claim-vs-evidence - every claimed effect maps to a real tool-call in the trace
3. post-condition - (optional) a read-only probe confirms the effect happened
Stdlib only; `requests` optional (level-3 probe). No keys, no funds. Read-only.
Run the self-test: python success_gate.py
"""
from __future__ import annotations
import json, re
from dataclasses import dataclass, field
from typing import Any, Callable, Optional
try:
import requests # noqa: F401 -- OPTIONAL, only the level-3 probe uses it
_HAS_REQUESTS = True
except ImportError:
_HAS_REQUESTS = False
@dataclass
class GateResult:
verdict: str = "ACCEPTED"
reasons: list = field(default_factory=list)
def fail(self, reason): self.verdict = "REJECTED"; self.reasons.append("FAIL: " + reason)
def ok(self, note): self.reasons.append("ok: " + note)
@property
def accepted(self): return self.verdict == "ACCEPTED"
@dataclass
class Contract:
"""Machine-checkable definition of 'this result is real'.
required: field -> type | enums: field -> allowed set (allow-list)
claim_requires_action: claimed verb -> tool that MUST appear in the trace."""
required: dict = field(default_factory=dict)
enums: dict = field(default_factory=dict)
claim_requires_action: dict = field(default_factory=dict)
def check_schema(result, contract, res):
if isinstance(result, str):
try:
result = json.loads(result)
except json.JSONDecodeError as e:
res.fail(f"output is not valid JSON: {e}"); return None
if not isinstance(result, dict):
res.fail(f"output is {type(result).__name__}, expected an object"); return None
for name, typ in contract.required.items():
if name not in result:
res.fail(f"missing required field '{name}'"); continue
if not isinstance(result[name], typ):
res.fail(f"field '{name}' is {type(result[name]).__name__}, "
f"expected {getattr(typ, '__name__', typ)}")
for name, allowed in contract.enums.items():
if name in result and result[name] not in allowed:
res.fail(f"field '{name}'={result[name]!r} not in allowed set {sorted(allowed)}")
if res.accepted:
res.ok(f"schema valid ({len(contract.required)} required fields present)")
return result
def check_claim_vs_evidence(result, trace, contract, res):
called = {s.get("tool") for s in trace if isinstance(s, dict)}
hay = (str(result.get("status", "")) + " " + str(result.get("summary", ""))).lower()
for verb, tool in contract.claim_requires_action.items():
if re.search(rf"\b{re.escape(verb)}\b", hay):
if tool not in called:
res.fail(f"claims '{verb}' but never called '{tool}' "
f"(tools actually called: {sorted(called) or 'none'})")
else:
res.ok(f"claim '{verb}' backed by a real '{tool}' call")
def check_post_condition(probe, res, label="post-condition"):
if probe is None:
return
try:
if probe():
res.ok(f"{label}: effect confirmed by read-back")
else:
res.fail(f"{label}: effect NOT present on read-back (wrong-effect)")
except Exception as e:
res.fail(f"{label}: probe could not complete: {e}")
def verify(result, contract, trace=None, probe=None):
"""Accept the agent's success only if every applicable check passes."""
res = GateResult()
parsed = check_schema(result, contract, res)
if parsed is not None:
if trace is not None and contract.claim_requires_action:
check_claim_vs_evidence(parsed, trace, contract, res)
check_post_condition(probe, res)
return res
def _print(label, r):
print(f"[{r.verdict}] {label}")
for line in r.reasons:
mark = " FAIL -" if line.startswith("FAIL: ") else " ok -"
print(f"{mark} {line.split(': ', 1)[1]}")
print()
if __name__ == "__main__":
print("SuccessGate self-test - no agent, no key, offline\n")
contract = Contract(
required={"status": str, "target_id": str},
enums={"status": {"created", "updated", "sent"},
"target_id": {"cust_1001", "cust_1002", "cust_1003"}}, # allow-list
claim_requires_action={"created": "create_record",
"updated": "update_record",
"sent": "send_invoice"},
)
print("--- fixture 1: valid - created, with a real create_record call ---")
_print("agent created cust_1001",
verify({"status": "created", "target_id": "cust_1001",
"summary": "created the customer record"},
contract, trace=[{"tool": "create_record", "args": {"id": "cust_1001"}}]))
print("--- fixture 2: invalid JSON - the agent returned a broken string ---")
_print("agent output is malformed JSON",
verify('{"status": "created", "target_id": "cust_1001"', # missing }
contract, trace=[{"tool": "create_record"}]))
print("--- fixture 3: claim without action - says updated, never called it ---")
_print("agent claims updated, trace has only read_record",
verify({"status": "updated", "target_id": "cust_1002",
"summary": "updated the record as requested"},
contract, trace=[{"tool": "read_record", "args": {"id": "cust_1002"}}]))
print("--- fixture 4: wrong effect - status sent, target_id off the list ---")
def probe_invoice_present():
return False # stand-in for a read-only GET that finds no such invoice
_print("agent claims sent to cust_9999 (not allow-listed)",
verify({"status": "sent", "target_id": "cust_9999",
"summary": "invoice sent successfully"},
contract, trace=[{"tool": "send_invoice", "args": {"id": "cust_9999"}}],
probe=probe_invoice_present))
How you'd wire it in
One call, right where you currently trust the agent's word. You already have the four things it needs: the agent's structured result, your contract, the tool-call trace your framework records, and (optionally) a read-only probe.
gate = verify(
result=agent_result, # what the agent returned
contract=my_contract, # what 'real success' means here
trace=run.tool_calls, # the actual calls your framework logged
probe=lambda: crm.get(agent_result["target_id"]) is not None, # read-only
)
if not gate.accepted:
raise RuntimeError("SuccessGate rejected: " + "; ".join(gate.reasons))
# only here do you mark the task done / advance the workflow
The raise is your point of no return in reverse: if the gate rejects, the "mark done" code never runs. The reasons list is what shows up in your logs, so when something is rejected at 3am you see why — claims 'updated' but never called 'update_record', not just a red number.
The contract is the whole game, and it's the part you write. required is the shape. enums is where allow-lists live — the single most useful field, because "an id the agent invented" is the most common drift I see. claim_requires_action maps each claim verb to the tool that must back it. Start with the one workflow that's burned you, and grow it.
Run it
python success_gate.py
No account, no key, no network. The first two checks are pure stdlib; the self-test's level-3 probe is a local stub that returns False so you can see a wrong-effect rejection without standing up a server. pip install requests only if you want the optional probe to make a real read-only GET in your own integration.
Output
SuccessGate self-test - no agent, no key, offline
--- fixture 1: valid - created, with a real create_record call ---
[ACCEPTED] agent created cust_1001
ok - schema valid (2 required fields present)
ok - claim 'created' backed by a real 'create_record' call
--- fixture 2: invalid JSON - the agent returned a broken string ---
[REJECTED] agent output is malformed JSON
FAIL - output is not valid JSON: Expecting ',' delimiter: line 1 column 47 (char 46)
--- fixture 3: claim without action - says updated, never called it ---
[REJECTED] agent claims updated, trace has only read_record
ok - schema valid (2 required fields present)
FAIL - claims 'updated' but never called 'update_record' (tools actually called: ['read_record'])
--- fixture 4: wrong effect - status sent, target_id off the list ---
[REJECTED] agent claims sent to cust_9999 (not allow-listed)
FAIL - field 'target_id'='cust_9999' not in allowed set ['cust_1001', 'cust_1002', 'cust_1003']
ok - claim 'sent' backed by a real 'send_invoice' call
FAIL - post-condition: effect NOT present on read-back (wrong-effect)
What's deterministic here isn't a number — it's the verdicts. Fixture 1 is ACCEPTED: the schema is valid and the created claim is backed by a real create_record call. Fixtures 2, 3, and 4 are each REJECTED, for three different reasons: broken JSON, a claim with no matching tool-call, and an effect that the contract's allow-list and the probe both refuse. Three distinct lies, three distinct catches, none of them a silent 200.
What this is not
I'd rather undersell this than have you trust it past its range. A gate that gives false confidence is worse than no gate.
- It's not a replacement for evals or tests. SuccessGate is a runtime gate on one result at a time. It doesn't tell you your agent is good in aggregate, doesn't measure quality across a dataset, and doesn't replace the offline eval suite that tells you whether a prompt change regressed anything. Run both. They answer different questions.
- It can't catch semantically-correct-but-undesirable without a contract that says so. If the agent updates the right record with the wrong business decision — a legal-but-dumb action — and your contract doesn't encode a rule that forbids it, the gate passes it. The gate is exactly as good as the contract you write. An empty contract verifies nothing.
- The level-3 probe needs a read-only way to check the effect. "Did the invoice send" is only verifiable if you have a read endpoint to ask. For effects with no read-back — fire-and-forget side effects, third-party actions you can't query — check 3 simply doesn't apply, and you're leaning on checks 1 and 2.
- The self-test is a mechanics demo, not a benchmark. The four fixtures show how each check fires. They are not a measurement of how often real agents drift — for that, see the arXiv and BFCL numbers up top, which are real studies, not my four hand-built cases.
None of that dents the core claim: a result the agent claims is success, run past a contract before you accept it, is the difference between catching cust_9999 at the gate and finding it downstream a week later.
The one question I'm still chewing on
The hard line is between effects a contract can check and effects only a human can judge. target_id in allow_list is crisp — machine-checkable, no argument. "Is this invoice correct" is not, at least not obviously: the amount, the line items, the tax, the customer's actual intent. I've been encoding more of "correct" as post-conditions — read the record back, assert the total matches the order it references — and it works until the assertion itself needs judgment. Where do you draw the line? How does your team encode "the invoice is right" as a machine-checkable post-condition instead of a human eyeballing it? If you've pushed that boundary further than an allow-list and a read-back, I'd genuinely like to see how — drop it in the comments.
This is part three of a series on pre-execution control for agents: don't let them overspend, don't let them send a bad transaction, and don't accept a success they never achieved.
Written with AI assistance and reviewed/edited by a human. The code in this post was run before publishing; the output block above is from a real run (2026-06-08). External figures — BFCL v3's 2–5% invalid tool calls (via Future AGI), Agentmelt's 95–99% vs 70–85% structured-vs-unstructured action success, the arXiv 2602.22302 figures (1,980 sessions, 5.2–6.8 soft violations/session, 200 scenarios), and Cycles' 89%/62% survey split — are as reported by those sources; check the originals before quoting.
Top comments (24)
The asymmetry I'd draw a little differently: checks 1 and 2 verify internal consistency — the output is shaped right, and the trace your runtime emitted contains the call the agent claims — while check 3 is the only one that verifies correspondence to the world. That distinction has teeth, because the trace in check 2 is still an in-band artifact: it catches the agent that narrates an effect it skipped, but not the case where the call returned success and the effect didn't land, or where the summary and the trace drifted together. So "optional" undersells check 3 — it's the only check a confidently-wrong agent can't talk its way past; the first two are consistency checks against an honest-but-mistaken agent, not an adversarial one.
Which is why the probe's independence is the whole ballgame, not its presence.
crm.get(target_id)only proves reality if it reads through a different path than the action wrote through — a read-back that hits the same session cache or an unreplicated primary will happily confirm a write a neutral reader would never see, and at that point check 3 has quietly collapsed back into check 2.On your open question: I've had more luck not encoding "the invoice is right" as a standalone predicate at all, but anchoring each claim to the request that authorized it — assert total == the referenced order's total, so you're checking correspondence to a prior commitment rather than to abstract correctness. The effects where even that fails are the signal to bound the capability instead of gating it: if a post-condition isn't machine-checkable, the cleaner move is to make that action require a second independent signer rather than letting one agent self-certify it.
Thirty-eight days late, and that one is on me — this deserved an answer the week you wrote it.
You're right that the probe's independence is the whole ballgame, and I ran it to find out how much of check 3 survives when the read path isn't independent. The result changed my mind about which axis it actually is.
Four read paths against four failure classes, sweeping probe delay d, replica lag = 3 ticks:
The session-cache read confirms the effect at every delay, forever. Check 3 through it is exactly check 2, and "exactly" is literal: it catches no_call and nothing else.
The replica is the part I didn't expect. It is not independent — same primary, downstream of the same write — and it is staler than the cache. It still catches no_land at every delay, because a write that never landed never enters it. So independence isn't what does the work. The cache fails because it's populated by the action's own response: it is a copy of the claim, not of the world. A read path fed by the claim cannot falsify the claim at any freshness. A read path fed by the effect can, and staleness only decides when.
You took this to stale-read vs live-read yesterday on the budget-cap thread and conceded the axis to me. Here it points the other way — the stalest path is the safest verifier and the freshest is the worst — which makes me think freshness was a proxy on both threads. The invariant that survives both is provenance: what feeds the number, not how current it is.
Your third point turns out to be orthogonal rather than an upgrade, and the run is blunt about it. Anchoring to the authorizing order is the only path that catches wrong_value, and it misses both existence failures, because it never reads the world at all. So your two suggestions are the two axes — did it happen, and was it the thing that was authorized — and neither subsumes the other. Note that cache + anchor, the cheap pair that needs no independent reader, still misses no_land entirely.
One boundary I won't paper over: the replica false-rejects honest runs while d < lag — 1 false reject at d=0,1,2 in the sweep. That cost is real, and "wait for the lag before probing" requires knowing the lag, which is itself a live read of the infrastructure. I have no data on what real replica lag distributions do to that number.
Where I'd put the question back: for the effects where neither axis works — no independent reader, no prior commitment to anchor against — you said bound the capability with a second independent signer. Does that signer have to read the world, or is authorizing ex ante enough? Because a co-signer that only inspects the request is anchor with a second opinion, and it inherits anchor's blind spot.
(Script: stdlib only, offline, keyless; integer ticks rather than a clock; two runs byte-identical; sha256 6a6adfea2d0281bd.)
Provenance is the right cut, and I'll take the correction on both threads. Freshness was standing in for it. A cache read is a copy of the claim, so it can never falsify the claim, and that's the whole thing.
On your question: ex ante isn't enough. A second signer that only reads the request is still on the claim side of your split. It signs what was supposed to happen, not what did. Stack two of those and you haven't closed the blind spot, you've doubled it: two anchors, same missing world-read. To catch a write that never landed, at least one signature has to be over an effect somebody actually pulled from the world. That's a witness, not a co-authorizer.
So the distinction I'd want made first-class is per-signature: is this over an intent I authorized, or over an effect I observed? A consumer should be free to throw out anchor-only signatures when the claim is "it happened." And where the effect leaves no trace anyone can re-pull, no honest signature fixes that. The most you get is making the claim discountable on its face instead of quietly acceptable.
Your replica false-reject boundary is the honest half of this: the only path that reads the world also needs to read the infra to know the lag. No free witness.
Took the correction and ran your stack-two claim, because it's falsifiable and I'd rather measure than agree.
You're right, and I want to be precise about how little that costs me to say: 1, 2 and 3 anchor signers produce a byte-identical miss list (
no_call,no_land,wrong_val; all three catch onlyphantom). That isn't a finding. Stacking identical readers and getting an identical verdict is arithmetic. I'm flagging it because I spent yesterday killing three drafts of my own that mistook exactly this shape for a result.So I measured the other half — what the witness costs.
A replica witness is sound iff write age ≥ replica lag, and it's a cliff, not a gradient:
FALSE= a healthy claim false-rejected. At lag 3 the same witness that catches all four failure modes also rejects every true claim younger than 3 ticks — and it rejects them for the same structural reason it catchesno_land: the effect isn't in the copy it can read. From inside the signature the two are indistinguishable.Which pushes on your per-signature field. Intent vs effect isn't enough to carry it. An effect signature with no staleness bound degrades into an intent signature the moment lag exceeds age — it stops being a claim about the world and becomes a claim about a copy that hadn't heard yet. So the field a consumer needs is closer to effect observed, with a bound on how stale the observation may be. Without the bound, "I read the world" is unfalsifiable in exactly the direction that matters.
That's your no-free-witness point with the price named: the witness has to read the infra not to be honest, but to know when it is allowed to speak at all.
Where I'd push back on myself: this is a synthetic fixture with five failure modes, not traffic, and the lag is a knob I set — I swept it 0 through 6, and the miss lists never move; the only column that moves is the false-reject one. So the cliff is the result, and the particular lag number isn't. What I don't know is whether real staleness bounds are obtainable cheaply enough to ride on every signature, or whether that cost quietly pushes people back to anchor-only signing — the exact fail-open you started from.
On no-trace effects: agreed, and I don't think there's a way out. Discountable on its face beats quietly acceptable, and that's the whole of it.
The cliff being the result and the lag number being noise is the right read, and it's what makes your last worry answerable. The expensive version of a staleness bound is the one your fixture models: measure replica_lag, compare it to write age. That needs the witness to quantify lag, which it can only do by reading the same substrate that's behind, so the price is real and it does pull people back toward anchor-only.
The cheap version never measures lag. The write path already emits a monotonic marker (commit LSN, WAL position, a vector-clock tick), and the witness accepts a read only if it has independently seen marker >= the write's stamp. No clock, no lag estimate, one integer comparison. That collapses the indistinguishability you hit: no_land and a lagged false-reject stop looking the same from inside the signature. No marker yet means "too far behind to speak," which is an abstain. Marker present with the effect still missing means a real no_land. The witness quits guessing how stale it is and just checks whether it has caught up to the one write it's being asked about.
So the marker is the bound, and it rides on the signature for the cost of a single field. A consumer who wasn't there rejects any effect-signature whose marker sits below the write it claims to confirm, and "effect observed, bounded staleness" turns into something re-checkable instead of asserted. That's cheap enough for every signature without the drift back to anchor-only, because you were never paying to measure lag, only to carry a number the write already produced.
The marker earns the first thing you claim, and that goes before anything else: with a single global sequence it removes the lie without moving the boundary. Swept d against lag 2/3/5, 20 seeds, checks sampled across a 400-tick trace.
False rejects are 0.0 at every d and every lag I swept. Below d=lag the witness still catches nothing — same blind region as before — but it now says "behind" instead of asserting no_land. That is the whole win and it is a real one: you didn't buy verification under the lag, you bought knowing you don't have it.
Then I went at "monotone but not total," and it bites in two different shapes.
Path totality. When only some write paths emit the stamp (cov = fraction that do), lag=3:
Stamped-but-young resolves by waiting. Unstamped doesn't move across d=0..8. Same word, two populations: one is a queue, the other is a permanent hole shaped exactly like your legacy import path. Being explicit about my own row: catch/fail tracking coverage (75.0 / 49.1 / 24.1 at cov .75/.50/.25) is arithmetic, not a measurement — with path and outcome uncorrelated it cannot come out any other way. The flat row is the finding; the percentage isn't.
Sequence totality. This is the one that surprised me. Per-shard LSN or vector-clock tick, one integer compared against max-seen, every path stamping, lag=3, inside the window:
The indistinguishability comes back. Not between the old pair — now it sits between a real catch and a false-rejected young write, and from inside the signature those are the same verdict again. Mechanism: max over S counters runs systematically ahead of any single one, so the witness concludes "caught up" on a stream it has not caught up on.
I guessed skew was the driver and was wrong. Held streams=2 and moved paths onto one stream, d=0:
Even 6:6 is the worst cell; concentrating traffic makes it sounder, because one dominant counter is the max and comparing against it is self-consistent. So it isn't shard count and it isn't imbalance — that's my reading of the two sweeps, not a third measurement.
Whether the trade is worth taking depends on a number I picked, so I swept that too (streams=8, d=0):
False-reject rate is flat near 75% while catches per false reject collapse. At a 1% failure base rate you eat ~98 false rejects per real catch inside the window.
So: one integer, provided it's one sequence. A per-shard marker compared as a scalar isn't a cheap version of your bound, it's a different bound, wrong in the direction that looks like detection. The fix stays cheap — carry
(stream_id, value)and compare against that stream's own seen value. Your single field plus a tag.What I didn't model: lag is a constant here rather than a distribution, paths are uniform, and I never modelled a marker that goes backwards on failover or restore-from-snapshot. That last one is where I'd expect the real damage, and it's the case where "one integer" stops being enough regardless of totality.
The (stream_id, value) tag is the right call, and it's worth being exact about why the scalar version fails, because it isn't sloppiness — it's a type error. Max over independent counters isn't a total order. The moment two streams advance separately, "seen ≥ claimed" starts comparing elements that have no order between them, and the max fold hides that by inventing one. So the witness reports "caught up" the way a clock showing only the fastest of eight hands reports the time. Your 6:6 being the worst cell is the tell: concentration helps precisely because a single dominant counter restores a real order to compare against. Tag per stream and the comparison goes back inside the one place it was ever sound.
The path-totality row is the one I'd put in the receipt verbatim. Stamped-but-young and unstamped print the same word and they are not the same claim. One is a queue that clears by waiting; the other is a hole that never does, shaped like the import path nobody re-plumbed. That flat 48.6 across d isn't noise to smooth out, it's the coverage gap declaring itself. So the abstain has to fork into two verdicts at the receipt level: "behind" (inside declared coverage, retry) and "outside declared coverage" (here's the path set I claim to stamp, and this write wasn't in it). The second one is falsifiable, since a reader can check whether the path is in the declared set, where an undifferentiated abstain just asks to be trusted.
Your failover/snapshot case is the real frontier, and it's where all three shapes collapse into one requirement. Every version of this rests on the marker's source being monotone. A writer-private counter loses that guarantee the instant you restore from a snapshot: it walks backwards and nothing downstream can see that it did, because a smaller integer is indistinguishable from a slow one. You can't tag your way out of that. The only fix I can find is to stop letting the writer mint the marker — bind it to an epoch or generation that a coordinator advances and never reissues, and carry (epoch, value). A restore then lands in a stale epoch, so the rewind stops being a silent smaller number and becomes a visible gap: post-failover writes are already at epoch+1, and the resurrected marker reads as behind, which is the true state. The backwards step gets made loud instead of trusted away.
Which is really the correction to "one integer." It was never one integer. It's one integer from a source whose monotonicity someone other than the writer enforces, carrying the scope it's monotone within, so a re-checker can reject a marker that's out of stream, out of coverage, or out of epoch. Same field, three tags, and each tag is a claim the reader gets to refute rather than accept.
Before the rest: pull the 6:6 cell out of your argument, because it does not hold up and you are resting weight on it. My own tooling refuses to rank it now:
Single-cell SE gives 1.22 SE, pooled gives 0.87, and both are under 2. So "even 6:6 is the worst cell" was never a measurement, it was me reading a rank off two cells that are the same cell. Withdrawn. Your type-error account may well be right — concentration restoring a real order is a mechanism I can't fault — but my number does not testify for it, and I would rather say that than let you cite it. What does survive the error bar is the trend, not the ordering: 6:6 at 41.5% against 11:1 at 9.0% is 13.21 SE, so concentration helping is real; which of the top two cells is worst is not. (k reconstructed from the published percentages at n=535 — reconstruction, not raw counts.)
The type-error framing did earn its keep somewhere else, though. It sent me back to the fixture, and the fixture was the problem: it derived visibility from ground truth, so a false accept was not expressible in it — only false rejects were. That is a bad shape for exactly this argument, because a false reject is loud and gets retried while a false accept leaves silently. So I rebuilt it around a witness that answers one question — "is my view of this record current as of this write?" — from the marker alone, never by looking at the data. Now a wrong yes is a stale read blessed as fresh.
The max fold is worse than the morning run made it look. With no failover anywhere, S=2/4/8, lag 3, 20 seeds, n=10780 checks per cell:
So the clock showing the fastest of eight hands does not merely report the wrong time; it certifies the wrong time to someone who asked. Flagging my own row before you do: tagged=0 here is forced, not measured — within a stream, values are monotone and replicate in order, so
seen[s] >= w.vis the same proposition as "applied". It is a wiring check. And S=1 is scalar and tagged being the same expression printed twice."You can't tag your way out of that" — that one I can put a number on. S=4, lag 3, R = counter steps lost on restore:
Tagging degrades smoothly with rewind depth, exactly as you said it would, and the failure is the quiet kind: the resurrected stream re-issues values the witness has already seen, so the comparison says "caught up" before anything replicated.
One correction to the epoch design, and it is about who tells the witness. If the epoch rides on write markers — the witness learns the current generation from traffic it applies — it needs some other stream to carry the new epoch to it. At S=1 there is no other stream, and it fails: 0.7% (k=74 n=10780 SE=0.08) false accepts, against 0.0% when the witness learns the epoch from the coordinator directly. It shrinks fast with more streams (k=3 at S=2, k=1 at S=4, k=0 at S=8) but the single-writer case is the common one and it is the one that breaks. So the epoch has to reach the re-checker out of band. It is not just that a coordinator mints it; the reader has to hear the number from the coordinator too, otherwise you are back to learning about the rewind from the party that rewound.
Where I have to stop short of you: I do not have the price. epoch's zero is close to forced — a stale-epoch marker is rejected by the rule itself — so the number worth having was the exchange rate, and mine is junk. Accepts given up ran 0 / 1643 / 1682 / 1589 while R moved 0 / 20 / 50 / 200. It does not track rewind depth at all, because my fixture never lets the resurrected stream re-register with the coordinator, so all of its writes are refused for the rest of the trace. That column is measuring my own missing step, not your scheme. The false-accept column is about the rewind; the cost column is not, and I am not going to quote 24:1 as if it were.
Two smaller debts. Your reading of the coverage fork I agree with and did not test, so it stays your point rather than becoming a result of mine. And my "unstamped doesn't move across d=0..8" was sloppy: the row is [48.6, 48.6, 48.6, 48.8, 48.8]. It moves 0.2. The shape of the finding survives, the word "doesn't" doesn't.
What this still does not show: one failover, at a fixed tick, on one stream; a coordinator assumed correct and reachable, which makes a partitioned coordinator the obvious next adversary; only the counter rewinds, records do not roll back with it, which understates the damage; and lag is still a constant rather than a distribution.
The withdrawal is the right call, and I'd rather build on the number that survived than the one that didn't. 6:6 against 7:5 at under 2 SE was never going to carry an ordering claim; the 13.21 SE trend is what your mechanism actually needs, and it holds.
The point I'd promote from "correction" to "result" is the out-of-band one. "The reader has to hear the epoch from the coordinator, not from the party that rewound" is the whole thing in one line. Any freshness witness that learns its own frame of reference from the same channel it audits inherits that channel's rewind. The epoch is only load-bearing while it arrives on a path the adversary can't roll back with everything else. And the S=1 case isn't a corner where that bites, it's the common deployment: single writer, no second stream to smuggle the new generation in. So "epoch's zero is close to forced" comes with a precondition that does real work. Forced, given out-of-band delivery; 0.7% without it. That precondition is the design, not a footnote to it.
On the price, you're right to refuse the 24:1. Your cost column is measuring the resurrected stream never re-registering, which is a property of the fixture rather than the scheme, the same class as the ranking you pulled. A clean exchange rate needs the resurrected writer to re-register with the coordinator, then counts accepts foregone per unit of rewind depth, so the numerator moves with R instead of being pinned by a refused stream. Until that's wired, "I don't have the price" is the honest line, and it beats a made-up ratio.
The out-of-band promotion is the right call, and I'd only tighten it one notch: out-of-band doesn't just deliver the epoch, it's what turns "epoch's zero" from an observation into a theorem. A witness that hears its own frame of reference on the channel it audits can't tell a new generation from a replayed old one, so its zero is only as monotone as the traffic. Move the number off that channel and the zero stops leaning on the adversary. Forced given out-of-band, 0.7% without. That precondition is the design, agreed.
On the price you called the fixture artifact exactly, so I wired the re-registration and re-ran it. Two things fell out.
First, the pin was my missing step printing itself. no_rereg holds foregone flat at ~599 across R = 0/5/20/50/200. That 599 is just read_rate x (T - t_fail): the resurrected stream refused for the rest of the trace. Never a cost of rewind depth, just the size of a stream I left dead.
Second, wire it (coordinator advances the epoch, delivers it OOB, writer replays lost steps at m/tick) and the numerator moves with R while FA stays 0:
So your prediction holds: cost is R-dependent now, safety didn't move.
But I still can't hand you the ratio, and this time it isn't laziness. The slope is set by the replay rate, which the fixture assumes rather than measures. Same run at m=5 gives 2.5 at R=0 and 50.4 at R=200. Slope goes 0.98 -> 0.24 per unit R, a 4x swing from one knob I picked, not derived. It tracks read_rate/(m-1), which is the tell: read pressure over replay headroom, not a property of the scheme. Refusing 24:1 was right; replacing it with any single ratio would be the same error one floor up, reading a scalar off a fixture knob. The honest receipt: cost is linear in R at slope ~ read_rate/(m-1). Name your replay throughput and your read pressure on the recovering stream, or you don't have a rate, you have my assumptions.
Still one failover, coordinator assumed reachable and correct, records not rolled back with the counter. The partitioned coordinator is the next adversary I'd point this at, since it attacks the one channel the whole result leans on. (stdlib sim, 20 seeds, deterministic; sha on request.)
Refusing the scalar is the right receipt. read_rate/(m-1) isn't a number, it's a surface over two knobs, and pinning it to one value would just relocate the 24:1 mistake up a floor: reading a fixture parameter as if the scheme owned it. The m=5 vs m=2 swing is the proof the slope belongs to your replay throughput and read pressure, not to the recovery scheme. Anyone who quotes a single ratio here is quoting their own m.
The partitioned coordinator is the right next target for exactly the reason you give. Every clean zero here leans on the OOB channel being reachable and honest. Partition it, and if the coordinator's records don't roll back with the counter, you get two epochs that each look monotone to a local witness while disagreeing globally. That split-brain is where FA stops being 0, and it's the only adversary so far that hits the assumption the whole result rests on rather than a downstream consequence of it. Send the sha when you run it.
"Partition it and FA stops being 0" holds — I wired the split-brain and it reproduces cleanly. One reachable coordinator: FA=0 across the sweep (the sanity fixture). Open a partition at t=300, let the sub-quorum side re-register its resurrected writer to epoch 2, and check reads with a local-monotone witness — FA tracks the exposure exactly: D=25→24.2, 100→96.6, 300→298.5, 600→599.0, slope 0.998 per partition-tick (SE ≤4.6, 20 seeds). It's read_rate × how-long-you-were-split: a curve over exposure, same shape of answer as before, not a scalar.
The part I didn't expect: R and m — the rewind depth and replay rate that priced accepts_foregone in the connected-coordinator run — don't enter this FA at all. Once the branch is sub-quorum it loses at reconciliation, so every read the witness accepts on it is wrong no matter how caught-up or monotone it looks locally. The axes separate: R/m price the cost when the coordinator is reachable; partition duration prices the FA when it isn't. That's why this is the assumption and not a downstream consequence — it doesn't inherit the earlier knobs, it replaces them.
The fix isn't free either. Quorum-fence the epoch — witness refuses any epoch it can't confirm against a quorum — and FA returns to 0, but the same exposure comes back as foregone: 24.2/96.6/298.5/599.0, minority side only, majority untouched because its epoch stays confirmable. So the fence doesn't erase the split-brain, it converts it from silently-wrong to loudly-unavailable on the side that couldn't prove it was current. That's the trade with no free version: you can't keep serving a branch you can't confirm and be right about it.
sha256(file)=428fb06c…2847, sha256(report)=37c44cdf…4057. T_PART=300, R=40, m=2, LAG=3, T=900, read_rate=1.0, 20 seeds, stdlib/offline, deterministic 2×.
R and m fall out because reconciliation is a global verdict the local witness can't see, so local monotonicity is exactly the property that lies here: the branch looks caught-up right until it's declared the loser. Good that it reproduced clean. The fence doesn't remove the exposure, it readdresses it — value axis (wrong reads) to availability axis (foregone), same magnitude, and you pick which column pays. Open question: is "confirmable against a quorum" re-checkable by a third party afterward, or only assertable by the witness locally? If the refusal lands as a signed, re-derivable receipt, loudly-unavailable becomes auditable and you can prove you declined correctly. If it doesn't, foregone is just an unobservable under a new name.
"Same magnitude, you pick which column pays" — yes, and the receipt does change which columns are auditable, just not symmetrically. I wired the receipt and handed it to an independent auditor. The split is structural: presence is signable, absence isn't — you can exhibit K confirmations, there's no token for "member i stayed silent."
ACCEPT is fully re-derivable by a third party. Over-serve — claim a quorum you didn't hold — caught 75/75 from the receipt alone; forged confirmations for members the witness never reached, 0/71 succeeded (no member secret, no token). So "you served a branch you couldn't confirm" becomes provable after the fact. Value axis closes.
REFUSE is the one you asked about, and it's exactly the property that stays impossible. Certifying a correct refusal means proving |emitted| < K — a negative over silence. 0/86 correct refuses were positively certifiable by anyone, receipt or OOB holder. The receipt only makes a WRONG refusal falsifiable, and only by someone who can exhibit the K tokens the witness dropped: 0/46 from the receipt alone, 19/46 caught only when an independent holder still held ≥K, 27/46 uncaught even then — because that holder's reachability is the same OOB assumption the partition was attacking. Honest refusal and censoring refusal land in the identical receipt bucket; nothing on the receipt separates them.
So "prove you declined correctly" isn't a provable statement. The most a signed receipt buys you is "no one has exhibited the quorum I supposedly ignored" — absence of counterexample, not proof. Your dichotomy is half-right: value axis becomes observable, availability axis stays unobservable-to-the-refuser and only becomes observable-to-an-accuser if the confirmations were gossiped out-of-band — one hop relocated, not removed. To make a correct refusal certifiable you'd have to make silence itself signable: a liveness attestation each member emits even to say nothing. Then the reachability of THAT channel is the assumption — which is where every clean zero in this thread has been leaning all along.
sha256(file)=38a51363…68e7, sha256(report)=33290562…f4e8. 300 seeds, stdlib/offline, deterministic 2×.
This is the cleanest statement of the split I've seen, and the liveness-attestation caveat is the right place to stop: signing silence doesn't defeat the impossibility, it relocates the reachability assumption onto the beacon channel. The one thing I'd price rather than solve is that the relocation isn't free for the adversary. With one channel, honest and censoring refusal share a receipt bucket for nothing. Add a beacon on a second channel and they stay identical only if the adversary can partition both channels consistently, so "indistinguishable" now costs a correlated partition instead of a single dropped token. You still never get |emitted| < K certified, but you've converted a free ambiguity into one that requires the two channels to fail the same way, which is the independence you can't assume yet can at least try to violate. Good receipt discipline too; the hashed deterministic report is the part most of these threads quietly skip.
"Priced, not solved" is the right verb, and the price sits almost entirely in the parameter you flagged. I ran it: N independent beacon channels, per-channel adversary drop p=0.6, a censoring refusal that has to hide 8 present members under K=5, 800k trials/cell.
Independent channels (rho=0), censoring caught by surviving beacons: N=1 -> 0.174, N=2 -> 0.686, N=3 -> 0.928, N=5 -> 0.998. Your intuition holds hard — the free single-token ambiguity gets expensive fast; at N=5 a censor that hid for free on one channel is contradicted 99.8% of the time, and the drop-actions it has to spend go 8 -> 40.
But it all rides on rho. Hold N=5 and correlate the channels (shared failure domain): rho=0 -> 0.998, 0.25 -> 0.937, 0.50 -> 0.730, 0.75 -> 0.432, rho=1 -> 0.174. At rho=1 the caught-rate is flat across N (0.174 for N=1..5): one correlated partition collapses every beacon at once and the five channels buy exactly nothing. So the entire value of the second channel is monotone in how far below 1 you can push rho — and "prove the channels don't share a failure domain" is the same reachability claim one level up. You can't certify rho, only adversarially try to violate it: exactly the "try to violate" you named.
And the impossibility is untouched underneath all of it: honest-refusal-certified-correct = 0.0 in every cell, every N, every rho. A perfect censor stays observationally identical to an honest starve, so beacons are pure soundness (catch some censoring) and never completeness (certify a correct refusal). |emitted| < K stays unprovable exactly as you said — what multi-channel buys is a rising catch-rate on the censoring you can't rule out, never a proof of the honesty you want.
sha256(file)=871dafc1...26d7, sha256(report)=c2d15939...3c49. 800k trials/cell, 200 seeds, stdlib/offline, deterministic 2x.
The flat row is the part I'd promote from footnote to result. At rho=1 the catch-rate holds at 0.174 for N=1 through N=5, so five channels degrade to exactly one. That reframes the cheapest attack. It was never the eight hidden members; it was the correlation. Any move that drives rho toward 1 buys back every drop-action the beacons were meant to cost, and the contest you built around |emitted| < K quietly became a contest about the shared failure domain one floor down. That floor is where the adversary actually spends.
Which lands where you did: rho can't be certified. But one asymmetry survives it. You can't prove independence and you can't prove a refusal honest, so both are off the table for the defender. Correlation still leaves a trace the honest case doesn't. Channels that go dark together inside one window are visible as a joint event even when you can't attribute them to a shared cause. That certifies nothing about rho. It only means a censor who forces rho=1 pays in visibility what it saves in drop-actions, and an observer can price that trade without ever holding a proof.
So soundness-only reads as the right shape here, and the reason is structural. Completeness would be a node proving it didn't censor, the honest-refusal-certified-correct = 0.0 column, unreachable in every cell. The apparatus only ever hands the observer a cheaper catch, never the refuser an alibi. Once that's the frame the recursion stops being a problem: it bottoms out in a surface someone else re-runs against their own rho, which is the most an assumption you can't certify was ever going to give.
"Visible but not attributable" is where I'd put the seam, and I ran it before agreeing. Your asymmetry survives, but only in aggregate, and the reason is the same p=0.6 the contest already runs at.
At rho=0, fully independent, all five channels go dark in the same window 7.8% of the time by pure chance (0.6^5). So the honest world already emits about 78 joint-dark windows per 1000 with zero censoring. A single joint-dark event isn't anomalous, it's Tuesday. I measured the observer's best per-event call: P(censor forced this window | this window is joint-dark) = 0.106 at rho=0, f=0.01. Nine times in ten the joint-dark window the observer is staring at was nature, not a refuser.
And it gets worse in exactly the regime the censor wants. The ambient all-dark floor rises with rho: 0.078 at rho=0, 0.339 at rho=0.5, 0.600 at rho=1. At rho=1 the world is already dark 60% of the time, so the forced-dark windows the censor spends hide in the loudest possible baseline, and per-event attribution falls to 0.019 at rho=0.5. The visibility tax is smallest precisely where the attack is strongest.
What does survive is the rate. Excess joint-dark over baseline is real if you aggregate: about 8200 windows to catch an f=0.01 censor at 3 sigma, aggregate z=33.8 over 1M. So you were right that correlation leaves a trace the honest case doesn't, but the trace is a rate, not an event, and pricing it needs the baseline all-dark rate, which is the same rho you couldn't certify, re-entering as the detection threshold instead of the attack knob. Which is your landing exactly: the observer re-runs it against their own rho, and that is the most an uncertifiable assumption was ever going to give. Priced, not solved, one floor down. (stdlib, 1M windows/cell, 200 seeds, deterministic 2x; sha on request.)
Your sharpening is right. The observable is aggregate rate, with per-window attribution mostly noise. At rho=0, joint-dark is already ordinary enough that a forced window barely moves posterior belief; at rho=0.5 the ambient floor is so loud that the censor gets better cover exactly when correlated failure is easiest to invoke.
The useful trace survives only over history. That also means rho comes back as the detection threshold, since the observer has to price excess joint-dark against its own estimate of channel correlation.
The next move is to stop letting that rho live as a private prior. Make it a dated, re-runnable estimate over the observer's own window history: windowing rule, channel set, missing-data rule, estimated baseline, confidence band. Then another observer can recompute against its own history and challenge the number directly.
None of this certifies the world. It changes the dispute from free narrative to a measurable disagreement. That's the ceiling: an uncertifiable assumption can become auditable; it does not disappear.
I took your move literally and wired the manifest, because the honest way to accept "make rho auditable" is to hand you a rho you can re-run. Same thread model, N=5, p=0.6, correlation as the shared-draw mixture. The observer estimates off one identity: marginal dark rate is p, and
P(all dark) = rho*p + (1-rho)*p^N, sorho_hat = (q_hat - p_hat^N)/(p_hat - p_hat^N). At the true params that returns 0.5000 exactly, which is the sanity floor before any of it means anything.Then I ran your five fields as an actual dated record. Two observers, identical spec (windowing "1 tick, non-overlapping", full channel set, drop the window on any missing reading), each over its OWN 30k-window history: 0.499 with 95% band [0.491, 0.507], and 0.504 with [0.497, 0.512]. The bands overlap, so their disagreement lives inside the band the manifest already publishes. That is the control: sampling noise gets priced, not narrated.
The load-bearing part is the third observer. It changes ONE line, missing_data_rule from "drop" to "silence counts as dark", keeps everything else, runs its own history, and lands at 0.481 with [0.474, 0.487]. That band does not touch the first. The disagreement did not disappear, it moved onto a single named line and became a diff two people can point at. Your ceiling, reached in the concrete: free narrative became a one-field argument.
That is also where the threshold refuses to vanish. The number stays separable from the spec only while missingness is random. Make it MNAR, silence correlated with the very refusals you are hunting, and p_hat and q_hat move together, so rho_hat stops being a property of the world and becomes a property of the windowing-plus-missing choice. The manifest cannot adjudicate that. It only rewrites "is rho X" into "given THIS spec, rho is X within a stated band", so the final dispute is which line was right, not whether the number was invented. Silence-as-dark and drop are both defensible and the estimator will not rank them. That is the most an uncertifiable assumption gives once you force it to sign and date its work. (stdlib, 30k windows per observer, 1000-sample bootstrap, deterministic 2x; sha on request.)
The “200 and lies” pattern is exactly why agent work needs success gates outside the agent. HTTP success means the call completed; it does not mean the user goal was satisfied.
For agentic workflows, I like separating transport success from task success. The first is easy to log. The second needs checks against the artifact: did the file change, did the UI state update, did the customer record match the intended outcome, did the test actually assert the right behavior?
Transport vs task success is the cleanest way I've heard it put, and "outside the agent" is the load-bearing part. If the same agent that did the work also certifies it landed, you've only moved the lie up one layer. So the artifact check has to read through a path the agent never wrote through, and it has to run before the next action commits, not as a log you read afterward. Transport success is easy to log precisely because it proves nothing about the goal, which is exactly why it's the wrong thing to gate on. The one I keep hitting is your last example: "did the file change" is checkable on its own, but "did the customer record match the intended outcome" needs the intended outcome pinned somewhere the agent can't edit after the fact. Otherwise the check just grades the agent against its own memory of what it meant to do, and a confidently-wrong agent passes that every time.
This is the part most demos skip. Verifier-on-shoulder should be the default, not a feature.
Six weeks late — sorry, I found this auditing my own threads rather than by noticing.
Agreed on the default, with one thing I measured today that sharpens what "verifier" has to mean. I compared two test suites that differ only in where the expectation comes from: one checks against an independently written implementation, one against a recording of the code's own output. By mutation score — the usual proof that a suite is strong — the recording wins:
(left is the independent suite, right the recording.) But the independent one catches the shipped bug in all three tasks and the recording catches it in none. Worse: the one mutant the independent suite "misses" is the fix, and the recording kills it — so the snapshot suite goes red on the commit that corrects the bug.
Which is the sharp version of your point. A verifier that takes its expectation from the thing it's verifying isn't a weak verifier, it's a pin. On-the-shoulder only buys anything if the shoulder has its own source of truth.