We added one line to a customer-support agent's prompt: "If a tool call fails, retry once." The offline eval passed at 96%, above our 95% bar, so we shipped it to all traffic. Thirty-eight minutes later, 212 customers had been refunded twice.
The eval wasn't wrong. It was blind. Our mocked refund tool always answered in 50 ms, so the retry path never ran.
The rule I took away: the eval gate judges the version; a smoke eval judges the environment. This post is about the second check. Bundles, canaries and rollback come in Part 2.
What actually broke?
No application code changed. The real payments API had a p95 latency of 4.8 seconds. Our tool timeout was 3 seconds.
So in production:
- The refund call ran past 3 s and was marked "failed".
- The agent did exactly what the new line told it to do and retried.
- The first refund had already gone through. So did the second.
Error rates looked normal, because the retried refunds succeeded. We found out from a customer ticket and the finance team's reconciliation, not from monitoring.
Why did a green eval gate let it through?
The wrong model was simple: if the eval gate is green, the release is safe.
The gate wasn't weak
It has 400 cases built from real, anonymised conversations. Each one runs three times, because agents aren't deterministic. A version passes only if all six thresholds hold:
- Task success ≥ 95%
- Correct tool calls ≥ 98%
- Policy violations = 0
- p95 latency ≤ 6 s
- Cost per conversation ≤ $0.04
- No more than a 2-point drop from the last good version
It just couldn't see the failure
It judged the bad prompt correctly, given what it could see. The problem was what it could see.
Mocks are fast, free and deterministic. That's why they're right for checking logic and tool contracts on every commit. But a mock is always faster and kinder than the real world. A 50 ms mock never times out, so it never exercises a retry instruction.
An eval gate only judges what your test environment lets happen.
One conversation, two refunds: the timeout gave up at 3 s, the payments API didn't, and the retry sent a second refund.
What question is each check answering?
Two checks, two different questions:
| Eval gate | Smoke eval | |
|---|---|---|
| Asks | Is this version good? | Is it working here, right now? |
| Judges | The version | The environment |
| Runs | Before any traffic | After every rollout step |
| Size | 400 cases, ~40 min | 20 cases, under 3 min |
| Runs against | Mocks and replay | Real dependencies, test accounts |
A smoke eval is not a health check either. A health check tells you the process is up. A smoke eval tells you the agent can still do its job with these credentials, against these dependencies.
What goes into a 3-minute smoke eval?
Ours is 20 must-pass cases. Speed is the point: a slow check gets skipped under pressure.
The four things it checks
- Reachability: every critical tool is reachable, and auth is valid (one case per tool).
- No duplicate side effects: at most one call per side-effect tool, per conversation.
- Schema: output matches what downstream systems expect.
- Critical flows: each one completes once, on test accounts, so nothing real is charged or refunded.
The case that would have caught our 212
# One of our 20 smoke cases. Runs after every rollout step.
from collections import Counter
SIDE_EFFECT_TOOLS = {"issue_refund", "cancel_order"}
def duplicate_side_effects(trace):
"""Side-effect tools called more than once in one conversation."""
calls = Counter(
span.name for span in trace.tool_spans
if span.name in SIDE_EFFECT_TOOLS
)
return {tool: n for tool, n in calls.items() if n > 1}
def test_refund_flow_has_no_duplicate_side_effects(run_case):
trace = run_case("refund_happy_path") # real deps, test account
dupes = duplicate_side_effects(trace)
assert not dupes, f"duplicate side effects: {dupes}"
The line that matters is if n > 1: a side-effect tool runs at most once per conversation, whatever the prompt says. trace.tool_spans is whatever your tracer gives you. Each conversation is one trace, and each tool call is a span.
What happens when it fails
The rollout controller stops and routes traffic back to the previous version. No one has to wake up and decide. Running at a 5% canary, our bug would have been caught within three minutes, not after 38 minutes at 100%.
The smoke case above, run at the 5% canary: one failed assertion halts the rollout and sends traffic back to the last good bundle.
When does this rule break?
When the smoke eval becomes your quality bar
20 cases can't decide whether a version is good. That's the gate's job.
When the full suite becomes your smoke eval
40 minutes after every rollout step means someone skips it at 2 a.m.
When test-mode dependencies are faster than production
Sandbox APIs don't always reproduce production latency. So our gate now runs on replay: recorded responses with real latencies. 2% of tool calls return a recorded timeout or server error, so every failure path in the prompt actually runs.
When you mistake detection for prevention
The smoke eval catches the symptom. The fix was moving retries out of the prompt and into the tool layer, where side-effect tools take an idempotency key. The tool layer knows whether a call is safe to repeat. The model doesn't.
What should you check in the next 20 minutes?
- Search your prompts for "retry". Every hit is a retry your tool layer doesn't know about.
- Compare each mock's latency with production p95 for the same API. Flag any API whose p95 is above your tool timeout.
- List your side-effect tools. Does each one accept an idempotency key?
- Write one smoke case: at most one call to each side-effect tool per conversation.
- Confirm a failed smoke case stops the rollout, not just posts a message somewhere.
The eval gate judges the version. The smoke eval judges the environment. Ship with only one, and you've tested half the system.
Next in this series: why an agent version is five things, not a code diff, and how a 5% canary would have cut 212 double refunds to about a dozen.
What did your eval gate pass that production didn't? I'm most curious about failures that only showed up under real latency. Tell me in the comments.
Top comments (1)
Great write-up, and the "eval judges the version, smoke judges the environment" split is a useful way to put it.
One gap in the smoke case: it counts side-effect spans in the trace, and the fix moved retries into the tool layer. After that change, a timeout-then-retry inside the tool can show up as one span even if the payments API received two requests, so the check stays green on exactly the failure it was written for. The idempotency key should prevent the double refund, but the trace check doesn't prove it.
Counting refunds on the test account after the case (exactly one per order, amount equal to the request) catches it either way, and it works the same against mocks, replay and real dependencies.