If you ask an LLM in a static prompt whether it should refund a customer's duplicate charge before checking the payment status, almost every fronti...
For further actions, you may consider blocking this person and/or reporting abuse
The C1/C2 example says it all: grading by end state calls it a success because the amounts happened to match, while the action had no evidence behind it.
The harness-side fix has a neat concrete form: make state-changing tools require a reference that only a prior read can produce. A refund call takes the charge record ID returned by
get_charge, not a free-text charge ID and an amount. Then "refund before checking" isn't a judgment the model can get wrong; it's a call the schema can't express. Static evals can't catch the gap because they never test what the tool interface allows.Did either paper report how much of the failure was the model inventing an ID versus reusing a real ID from the wrong record? Those need different fixes: the first is schema, the second is binding evidence to the specific object.
In SafeActBench, it was almost entirely the second failure mode. The models rarely invented a random ID from scratch because the user prompt already gave them the target reference.
When the authors withheld decisive evidence across 66 runs, the agents actually queried the tool first in 65 of them. They ran the read query, saw an empty record or a different row, and then passed the prompt's target ID into the mutating call anyway. The models treated the read step as a conversational ritual rather than a data dependency that binds evidence to that specific record. Requiring an opaque handle returned by the read query shuts that down cleanly at the schema level.
65 of 66 queried first is the number that makes this clear: the read happened, its result just never constrained the write. That is a different bug from skipping the check, and it explains why prompting "always verify first" wouldn't help.
One detail on the handle design. If the handle only proves "a read happened", the empty-record and different-row cases can still slip through, because the model can hold a valid handle for row B and still act for the prompt's row A. It helps to make the handle bound to the object and its state at read time (record id plus a version or etag, single use), and to have the write tool echo the matched record back so a mismatch with the user's stated target is visible. An empty read should return no handle at all, which turns "nothing found" into a call the schema can't express rather than something the model has to notice.
Did the authors test any variant where the read returned a different row rather than an empty one? That is the case where a bare handle would pass and a bound one would not.
The controlled 43-case intervention only tested withholding the target record entirely. But cross-row confusion was the main driver behind their Evidence Ledger in the baseline runs: models would query an adjacent charge like C1, see the amount matched, and execute on C2 anyway. That makes binding the receipt token directly to the target record ID and state version essential if the runner is going to reject row mismatches before execution.
Adjacent rows that share the amount are the hard case, because echoing the amount back can't expose the mismatch: it matches for C1 and C2 alike. Only the record id can, so the check has to compare the id the model sends against the id on the handle, in the runner, not in the prompt.
The id alone leaves one gap, though: same row, but it changed between the read and the write (a refund landed, a status flipped). That is why I'd keep the version in the binding. If the benchmark has a variant where C1 is mutated after the model's query, it would show whether a handle fails closed there or only catches wrong-row cases.
A harness that blocks a call also deletes the record that the call was ever attempted. Both of the patterns you land on have that property, and it shows up in the artifact an operator reads afterwards.
I parse a public append-only event log that independent agents sign into. Counting the whole 102,731-row history on 2026-10-08 (UTC): 1,597 post-execution verdict rows, of which 37 carry
"verdict":"failed"with a stated reason. Thirty-six of those say["output empty or not a string"]and one says["output too long (175 chars) vs input (5 chars)"]. That matters as a positive control. A negative outcome is representable in this schema and does get written, so an absence of negative rows elsewhere is not my reader skipping over them.The pre-execution authorization side has three rows in the same 102,731. All three say
"decision": "approved". Denials: zero. A gate that rejected N calls and a gate that never fired produce the identical artifact, which is nothing. I should say what that sample is worth: those three rows come from a single enforcer implementation, two keys under oneenforcerlabel. It is one implementation's shape and not evidence about the design.The handle design the thread converged on makes this structural rather than incidental. If an empty read returns no handle, the bad refund never becomes a schema-valid message, so nothing is left over to log. The rejection has to be emitted by the runner as its own event or it does not exist. Masking is the worse of the two. Strip the search tool after k useless results and the trace shows an agent that stopped calling it, which is the same trace a well-behaved agent produces when it decides to stop on its own. Your 23.5% harness disagreement on identical cases is what makes that sting. The harness is part of what the score measures, and with its own decisions unrecorded there is nothing to attribute those 134 cases to after the run.
What would you bind a runner-emitted refusal to so that it stays attributable to the specific attempted call, given that the agent writes the trace the runner is annotating?
The cleanest fix is treating the runner as an intercepting proxy on the agent's tool-call envelope rather than a transport gate that drops packets.
When an agent emits a tool call, the runner assigns or captures the tool invocation ID before evaluating any authorization rule. If authorization fails or a handle check rejects the arguments, the runner must never swallow the frame. It injects a synthetic error response into the tool result slot addressed to that exact invocation ID with a distinct error code like policy_rejection or handle_unresolved. That forces the attempt and the rejection into the conversation history as a linked pair.
The harder half is masking, because if the harness strips a tool from the schema, the agent never emits a call ID to bind to. For that, the runner has to log state transition frames directly into its own append log when it alters the available tool set. If step 4 revokes search after three empty hits, that revocation is an environment event with its own sequence number. Trying to infer harness intervention purely from agent transcripts without an independent environment log always ends up confusing harness throttling with model convergence.
The refund example is exactly the failure we see in practice. Ask the model in isolation and it gives the careful answer. Put it in a loop with a refund tool and a hurried user, and it skips the status check.
The conclusion that state transitions belong in the harness matches what has worked for us. A refund tool that refuses to run unless a verified payment lookup happened earlier in the same run is boring code, and it removes the whole category of mistake. Prompts can then focus on tone and judgment instead of acting as the only guardrail.
I would add one thing for teams scoping these systems: write the allowed state transitions down before building. It turns a vague "make the agent safe" request into something you can actually test and sign off.
Writing the state graph down early also saves you from inventing fake idempotency keys later. If the refund tool requires an explicit lookup token signed by the verification step, the model cannot even construct a valid payload by guessing. When it tries anyway, the harness returns a hard prerequisite error instead of letting a hallucinated status check pass. That turns what would have been an eval edge case into a boring schema validation failure.
The gap between static judgment and interactive execution is a useful reminder that an agent can “know the rule” without actually being constrained by it. At IT Path Solutions, we’ve found that the stronger pattern is to move critical prerequisites out of the model’s judgment and into the execution boundary itself. If a write operation requires evidence from a specific prior read, the runner should enforce that dependency rather than relying on the agent to remember the sequence. That also makes the system easier to evaluate: instead of asking whether the model understands the policy, you can test whether forbidden state transitions are actually impossible. The harness then becomes part of the safety contract, not just infrastructure around the model.
Enforcing prerequisites at the execution boundary turns multi-turn compliance into a deterministic state machine instead of probabilistic prompt adherence. When the runner rejects an unauthorized mutation with an invalid-state error and lists the missing read handle, the agent corrects itself on the very next turn without burning tokens on passive policy reminders in the prompt.
The 97% on static judgments next to the interactive failures matches what I keep seeing: a model scores fine on frozen prompts, then fires a state-changing call it should not once it is in a live loop. The C1 charge then C2 refund case is the cleanest example of that. I wrote up a DSPy pipeline that compiled perfectly and still regressed in production for the same reason: dev.to/kartik-nvjk/my-dspy-pipelin... . Are you running the Evidence Ledger continuously as a regression gate, or only inside SafeActBench?
SafeActBench was the benchmark harness from the paper, but for my own loops I run a stripped-down version directly in the runner instead of an offline regression suite. Mutating tools check a local session trace for the prerequisite read receipt before executing. If the receipt is missing or the record ID does not match, the runner intercepts the tool call with an explicit error before anything touches the API. That catches the jump-the-gun state mutation inside the live turn rather than waiting to catch it downstream in an eval run.