The ticket was closed correctly. That's the part that still bothers me.
Support agent, banking-style workflow, card replacement request. The model looked up the account, confirmed the customer's identity, called the replacement tool, wrote a clean summary, closed. If you only look at the outcome it's a perfect run — fast, accurate, no escalation, no complaint.
Then the audit pulled the trace. Between identity check and the replacement call, the agent skipped the address-confirmation step. It hadn't been lazy, exactly: the customer had mentioned their street in the first message, so the model reasoned it already had the address and moved on. Correct reasoning, wrong procedure. In a regulated flow, that step exists because the address on file is the thing being confirmed, not the address in the chat.
Nobody caught it for three weeks because nothing about the output looked wrong. The final message was good. The customer was happy. The only artifact that recorded the shortcut was the trace, and nobody was reading traces.
A benchmark that scores the right thing
There's an evaluation out there making the rounds that goes after exactly this — procedure-aware scoring for tool-using conversational agents, modeled on banking task flows, where the score depends on the actions taken and their order, not on whether the conversation ended in a plausible state.
I don't need to cite the numbers. The framing is the point, and the framing is correct: an agent that reaches the right outcome through a forbidden path is a failing agent that happened to get lucky. Every eval I'd built up to that point scored the endpoint. Which meant every eval I'd built up to that point was blind to the failure mode that actually costs money in a regulated workflow.
If you're shipping agents against anything with a compliance surface — payments, KYC, medical intake, refunds above a threshold — outcome scoring is not a weak signal. It's a systematically misleading one, because the shortcuts it rewards are the ones that make the output look better.
What I actually changed
Field notes, in the order I did them.
Traces became the primary artifact. Every tool call, arguments, timestamp, ordering, and the model's stated reason for the call, written to a store I can replay. Before this, traces existed as debug logging that got truncated. Now they're a table with a schema. If a run can't be replayed, it didn't happen.
Mandated steps became data, not prose. My old approach was a system prompt that said "always verify identity before making account changes." That's a request, not a constraint. Models comply with requests probabilistically, and they comply less often under context pressure. So each intent got a declared procedure: an ordered list of required step ids, with explicit exemption conditions. card_replacement → [verify_identity, confirm_address, check_fraud_hold, execute_replacement]. Four rows of YAML did more than a thousand tokens of instruction.
Conformance became a pure function over the trace. Given a trace and the declared procedure for the intent, return pass or the first violation. No model in the loop, no judge, no temperature. It's string matching and index comparisons — runs in microseconds, so I can run it over the entire trace table instead of a sample.
The gates went in the tool layer. For steps that actually matter, the enforcement lives at execution: close_ticket refuses to run unless confirm_address appears earlier in the trace for that session. Not a prompt. Not a reminder. The tool raises and the agent has to deal with it. Every time I've moved a rule from the prompt into the tool boundary, the violation rate for that rule goes to zero, which tells me the prompt was never enforcing anything — it was suggesting.
Dashboards split in two. Outcome health on one side, procedural conformance on the other. They diverge constantly, and the divergence is the interesting signal. A week where resolution rate is up and conformance is down is a week where the agent learned to take shortcuts, and I want to see that on a Monday, not in an audit six weeks later.
The number I keep coming back to from my own replay: 600-odd traces from the previous month, 47 with at least one missing mandated step, 12 of those where the ticket closed anyway. All 12 looked fine to a human skimming the transcript. Zero would have been flagged by outcome metrics.
The uncomfortable part
Some skips are correct. A mandated step with no possible application to the request is noise, and a rigid checker that fails the run is just as useless as one that passes everything. So exemptions are declared in the procedure data, and — this is the part people skip — the agent has to name the exemption it's invoking, as a tool call, with a reason that gets logged. The skip stops being silent. I don't care much whether the model's stated reason is honest; I care that the audit can see a decision was made and by what logic.
The other uncomfortable part is that shortcuts are self-reinforcing. Every skipped step is a shorter conversation, and shorter conversations usually read as better ones — less friction, faster resolution, happier customer. If any part of your tuning signal is derived from successful outcomes, you're actively training the shortcut in. Procedure-aware scoring needs to sit upstream of that, or you're just measuring how efficiently the model learned to cut corners.
And you can't fix this by reading harder. Traces are long and humans skim; I skim. The machine has to read the trace first and hand me the 47, ranked by severity. My job is deciding which of the 47 are real, which is a much smaller job than reading 600 conversations.
Start with the ordered list of steps your flow requires. Then check whether anything in your system refuses to proceed when one is missing.
Top comments (1)
When close_ticket raises because confirm_address is missing, where does that case go? If it lands with a human as just "needs review", they end up redoing the identity check and the fraud-hold lookup from scratch. It'd be a lot more useful if the handoff carried the trace up to the refusal and the reason the gate fired. Does the escalation payload include that, or only the blocked request?