The support agent looked fine during testing.
It understood refund requests, found the correct order, explained the policy, and asked for confirmation before taking action.
Then it reached production and refunded the same order twice.
The first reaction was predictable: rewrite the prompt.
Add “never issue the same refund twice.” Put it in capital letters. Repeat it near the tool instructions. Lower the temperature.
None of that fixed the real problem.
The payment API had completed the first refund, but its response timed out. The workflow interpreted the missing response as failure and retried the tool call. The model followed its instructions both times.
This was not a prompt bug. It was a distributed-systems bug wearing an AI costume.
The Prompt Is Only One Part of the System
A demo often looks like this:
user request -> model -> answer
A production agent looks more like this:
request -> context -> model -> tool -> external API
-> state update -> model -> approval -> final response
Every arrow is another place where the workflow can fail.
The wrong customer record may enter the context. A tool may receive valid JSON with the wrong business meaning. Authentication may expire halfway through a run. A background worker may retry after the external action has already succeeded.
Prompt changes cannot repair those failures.
Tool Calls Create Real Side Effects
Generating a poor paragraph is inconvenient. Sending an email, cancelling an order, or updating a CRM record changes something outside the model.
That means consequential tools need the same protections as any other production service: input validation, authorization, timeouts, retry policies, idempotency, and audit logs.
A simplified refund tool might protect repeated calls with an action identifier:
async def refund_order(order_id: str, action_id: str):
previous = await actions.find(action_id)
if previous:
return previous.result
return await actions.run_once(
action_id,
lambda: payments.refund(order_id),
)
The important instruction is not hidden in the prompt. It is enforced by the application: one logical action should produce one external effect.
The production implementation still needs atomic storage and protection against concurrent requests, but the boundary is clear. The model can propose the action. The system decides whether that action is safe to execute.
Traces Are More Useful Than Chat Logs
A chat transcript shows what the user and model said. It rarely explains why an agent selected a tool, what arguments it generated, how long the API took, or which state was restored after a retry.
An end-to-end trace should connect the model turn, tool call, approval, API response, state change, and final answer under one run identifier.
The OpenAI Agents SDK tracing documentation reflects this approach. Its traces can capture generations, function calls, handoffs, guardrails, and custom events as parts of one workflow.
This makes a useful debugging question possible:
Where did the actual behavior first diverge from the expected behavior?
That is much more precise than asking why the model was “confused.”
Some Actions Should Pause
Not every decision should be automated simply because the agent can call the necessary tool.
Reading an order status may be low risk. Issuing a large refund, deleting a record, publishing content, or changing account permissions may require human approval.
The human-in-the-loop guidance for the OpenAI Agents SDK demonstrates a pause, approve or reject, and resume pattern for sensitive tool calls. The specific framework is optional. The control is not.
Experienced AI engineers define these boundaries before production incidents define them instead.
Evals Should Test the Workflow
Testing ten prompts against ten expected answers is useful, but it does not test the complete system.
Workflow evaluations should also ask whether the correct tool was selected, whether its arguments were valid, whether unauthorized data was excluded, whether duplicate actions were prevented, and whether low-confidence situations were escalated.
Every production failure should become a regression case. Over time, the evaluation set becomes a record of how the system has failed in the real world.
At Spaculus Software, this is why AI engineering work extends beyond model integration. Reliable systems require workflow design, tool boundaries, evaluation, observability, and safe recovery paths around the model.
The prompt still matters. It guides intent, tone, reasoning, and tool selection.
But when an AI application breaks in production, the most valuable engineer in the room is usually not asking, “How should we reword this?”
They are asking, “What exactly happened between the request and the result?”
What workflow failure has been hardest for you to diagnose?
Top comments (0)