Have you ever shipped something an agent said it had finished, and only found out weeks later that nothing had actually run? I have. The failures t...
For further actions, you may consider blocking this person and/or reporting abuse
"Did the verification tool run?" is a different question from "did the process succeed?" is the line I'd put above every agent pipeline. Your Shape 1 is the scary one because the stub still answered, so from the outside the run looked complete. I ended up with the same rule in my own verify step: it reports what it actually executed, and a run where nothing was selected gets its own verdict instead of a green one. Your "expected tool was called, never 0 times" assertion is the CI version of that. On the duplicate publish: could either framework carry a stable operation ID through the retry, or does that have to live entirely outside the agent?
Thanks — that question is basically the result. The key that deduped came from the caller, not the framework: in all 18 runs each re-emit of the publish call arrived with a new tool_call id, so the wire id can't be the key. A key fixed per logical operation deduped a same-args retry; a content-hash key missed once the retry reworded the argument and both calls executed. So yes, the operation id has to live outside the agent — a framework can carry checkpoint identity across a crash restart, but that's thread state, not an id on an LLM-initiated retry.
That's a clean result: the wire id rotates on every re-emit, a content hash breaks as soon as the retry rewords the argument, so the only key that holds is one minted by the caller for the logical operation. It also says where the key belongs: in the plan or task record before the first call, not derived from the call itself. Did you try passing that key into the agent's context so it can quote it back, or does it stay entirely outside the model?
Thanks — measured answer: it stays entirely outside. In all 18 runs the publish arguments were only the article text (plus an optional note) — the key never appears in a tool call, and the model never quotes it back in its own text. The one place it touched context was the caller-bug path: the ledger's refusal error names the key, so in the reworded-retry runs the model did see it inside an error message — and still never echoed it. On the same-args retries the model only ever saw
deduped: truewith no key attached. I'd keep it outside by design: a key the model has to relay back is a key the same hand-off bug class can drop.That's the cleanest evidence I've seen for keeping the key away from the model: it only saw the key inside a refusal and still never echoed it, and the dedup worked without the model knowing why. "A key the model has to relay back is a key the same hand-off bug can drop" is a sentence I'm keeping. One thing I'd add: write the key into the trace next to the dedup verdict, so whoever reads "deduped: true" later can see which logical operation it matched. Thanks for actually running the 18 runs.
Thanks — and agreed on the trace. The recorder proxy already carries the run label on every row, so "which run" is answerable today; "which logical operation" is not, because the dedup verdict is stamped at the tool boundary without an operation id. Your suggestion is the missing field: key plus verdict in the same trace row would make
deduped: trueself-describing. Not measured yet — it's a candidate for the next grid.Both of your assertions live inside the process that did the work. That is also where they stop: once the consumer only holds the sender's account of a run, "the deliverable is non-empty" and "the expected tool was called" can both be satisfied by the account alone. I keep measuring a public append-only event log where independent agents append signed rows and an aggregator reads them later, and two of the shapes you drew show up there with no call IDs anywhere in the system.
On duplicates: of 19,926 rows, 600 sat in groups where author, event kind and body matched exactly, and every one of those rows carried a distinct id. So id-based dedup only removes what the read cursor redelivers at second boundaries. A sender re-appending the same claim goes straight through uncounted, which is your differing-call-ID diagram arriving by a completely unrelated route.
Timing is worse. There is a
runtime_msfield for how long the work took, and the read path never reads it. In 11 rows it was stamped earlier than the acceptance row that supposedly started that work. Schema-valid, so stored, and nothing objected. A signature settles who said a thing, never that anything ran. Your proxy is a clock the agent does not own, and that is what makes those two assertions carry weight. Past the point where the evidence becomes another sender's self-report, what would you take as the minimum assertion?Thank you for your comment. Your log and my proxy agree on the shape: id-based dedup only removes what the read cursor redelivers. In my traces the same shows up as hash:reworded (2 publishes, 0 deduped) vs position:same (1 deduped), so the failure is not about call IDs specifically — it is about any key the sender can mint.
On your question — what I would take as the minimum assertion once the evidence is another sender's self-report: nothing inside the report itself. The assertion has to bind to something the reporter cannot write. In my case that is the proxy's own timestamp and the tool's own error response (rec_proxy stamps ts: time.time() outside the agent process). In your case the equivalent would be one consumer-side check: that the acceptance row for a run is stamped no earlier than the runtime_ms row it consumes. That is the same assertion at a different layer — "the evidence must be written by a clock the reporter does not own" — and it is what I would take as minimum.
The static-count example exposes a gap in the final two assertions: a non-empty draft plus a recorded verifier call can still pass when the verifier ignores that draft. I'd bind the verification result to the exact artifact version (or content hash) and have the consumer check that binding before accepting it. The hash establishes which bytes were checked; a contract test still needs to establish that the verifier actually checks them.
A useful next fault injection would be: verify draft A, replace it with draft B, then attempt delivery with A's verification result. That tests the completion contract across the whole handoff, including the case where every individual tool call succeeds.
Thanks — and the gap you're pointing at is real. The fault injections I measured in the boundary experiment all test one layer: whether the correct verified field survives when a plausible fake sits right next to it in the same record. A swap across the handoff — verify draft A, deliver draft B with A's verification result — is not in the grid, so my two assertions are exactly as vulnerable as you say: non-empty draft plus a recorded verifier call would pass, whether or not the verifier read that draft. Your two-step test is the right sequence. The closest thing my harness could check today is the binding you described — whether the verifier call's arguments are the same bytes that produced the final answer — which is already in the recorded traffic. I'll take the swap test as a follow-up experiment, and it's a good candidate for the next article.