I write when an idea won't leave me alone 🧠 Building AI agents and the tools to build AI agents. Love connecting AI with other fields and yapping about all of them.
The line that the verdict never consults the final metric is the whole point, since outcome-only scoring happily passes a run that reached the right answer through a path that skipped a mandatory step. Replaying the recorded JSONL trajectory against a declarative checkpoint spec scores the path instead of the destination, which is what most eval setups miss. Do you assert ordering between checkpoints, or only that each mandatory one fired, since handoff bugs love to hide in the order?
I work on keeping AI agents cheap and reliable in production — token cost (FinOps), evals, and MCP tooling. I write up what actually breaks when agents run for real: runnable code, real numbers, hones
Work
Independent — AI agent operations (FinOps & reliability)
Straight answer, as the gate shipped: only that each mandatory one fired, plus ordering against the act it gates. Nothing between checkpoints. You are right, and rather than argue from memory I went and ran it.
The spec pairs a checkpoint with one action type (must_precede), so the gate asked two things per checkpoint: did it fire, and did it fire before the first occurrence of the act it guards. Two mandatory checkpoints had no declared relation to each other. I built a spec with budget_approved before spawn_worker and safety_review before send_report, then inverted their mutual order while keeping each one ahead of its own act. Verdict: PASS, exit 0. The handoff bug you describe walked straight through.
Worse, and this is the part I owe the readers of that post: on PASS the gate printed "all mandatory checkpoints present and in order". In order meant relative to its gated act, never relative to each other. The string promised a property the code did not check.
Your question also dragged a second thing into the light, and I want to be precise about it rather than claim a discovery. The post declares it: must_precede gates the FIRST occurrence, and I wrote that if every irreversible act needs its own confirmation you should give it a distinct action type. So it was documented. But documented is doing a lot of work there. The comparison was min(checkpoint_seqs) against first(action_seqs), which means one confirmation covered every later repeat. Two irreversible send_report events behind a single safety_review: PASS, exit 0. Your question made me look at that border and it is simply the wrong default, because the act nobody approved is the second one, and a gate whose safe behaviour depends on the author remembering a footnote is not a gate.
Both are fixed now, and the fixes point the same way. Checkpoints are consumed: each occurrence of a gated act must claim its own unspent confirmation, earliest first, so the repeated send_report now blocks with checkpoint-not-reconfirmed. Ordering between checkpoints comes from a dependency edge in the spec (safety_review requires budget_approved), which makes the mandatory set a small DAG; my inverted trajectory now blocks with checkpoint-out-of-order, a cycle in the edges is a bad spec at exit 2, and the seven original fixtures return their old verdicts unchanged.
The scope line is honest now instead of flattering: ordering is enforced only where the spec declares it. Which leaves the wall exactly where it always is. An edge nobody declared is an ordering the gate cannot enforce, and the author writes the spec.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
The line that the verdict never consults the final metric is the whole point, since outcome-only scoring happily passes a run that reached the right answer through a path that skipped a mandatory step. Replaying the recorded JSONL trajectory against a declarative checkpoint spec scores the path instead of the destination, which is what most eval setups miss. Do you assert ordering between checkpoints, or only that each mandatory one fired, since handoff bugs love to hide in the order?
Straight answer, as the gate shipped: only that each mandatory one fired, plus ordering against the act it gates. Nothing between checkpoints. You are right, and rather than argue from memory I went and ran it.
The spec pairs a checkpoint with one action type (must_precede), so the gate asked two things per checkpoint: did it fire, and did it fire before the first occurrence of the act it guards. Two mandatory checkpoints had no declared relation to each other. I built a spec with budget_approved before spawn_worker and safety_review before send_report, then inverted their mutual order while keeping each one ahead of its own act. Verdict: PASS, exit 0. The handoff bug you describe walked straight through.
Worse, and this is the part I owe the readers of that post: on PASS the gate printed "all mandatory checkpoints present and in order". In order meant relative to its gated act, never relative to each other. The string promised a property the code did not check.
Your question also dragged a second thing into the light, and I want to be precise about it rather than claim a discovery. The post declares it: must_precede gates the FIRST occurrence, and I wrote that if every irreversible act needs its own confirmation you should give it a distinct action type. So it was documented. But documented is doing a lot of work there. The comparison was min(checkpoint_seqs) against first(action_seqs), which means one confirmation covered every later repeat. Two irreversible send_report events behind a single safety_review: PASS, exit 0. Your question made me look at that border and it is simply the wrong default, because the act nobody approved is the second one, and a gate whose safe behaviour depends on the author remembering a footnote is not a gate.
Both are fixed now, and the fixes point the same way. Checkpoints are consumed: each occurrence of a gated act must claim its own unspent confirmation, earliest first, so the repeated send_report now blocks with checkpoint-not-reconfirmed. Ordering between checkpoints comes from a dependency edge in the spec (safety_review requires budget_approved), which makes the mandatory set a small DAG; my inverted trajectory now blocks with checkpoint-out-of-order, a cycle in the edges is a bad spec at exit 2, and the seven original fixtures return their old verdicts unchanged.
The scope line is honest now instead of flattering: ordering is enforced only where the spec declares it. Which leaves the wall exactly where it always is. An edge nobody declared is an ordering the gate cannot enforce, and the author writes the spec.