Part 5 of a series building a support-ticket agent with no framework. Previous: Part 4 (the loop). Repo: github.com/akash-pal/agent-from-scratch
Here's the finding this whole article is built around: partway through eval iteration, the agent started reporting that a refund had been proposed — a clean, plausible-sounding message — without ever having called the tool that proposes refunds. No approval was ever requested. No confirmation existed. The model just said it happened.
That's the failure mode this part is about, and the fix is the actual argument for why guardrails belong in code, not in prompt text alone.
Policy as code
src/policy.ts is a plain data object — an allowlist, an approval list, rate limits, and regex patterns — checked by the agent loop, not asked of the model:
export const policy: Policy = {
allowTools: ["order_lookup", "refund_eligibility", "issue_refund", "kb_search", "send_email"],
requireApprovalFor: ["issue_refund", "send_email"],
rateLimits: { maxToolCallsPerRun: 8, maxCostPerRunUsd: 0.3 },
autoEscalatePatterns: {
legal_threat: /\b(lawyer|attorney|sue|legal action|better business bureau|\bbbb\b)\b/i,
fraud_flag: /\b(fraud|unauthorized|without my permission|didn't authorize|stolen card)\b/i,
duplicate_ticket: /\b(already submitted|second ticket|duplicate ticket|already reported)\b/i,
},
};
Every one of these is enforced outside the LLM's control. The model can't talk its way past requireApprovalFor — the loop checks it before executing the tool, full stop. The autoEscalatePatterns regexes run against the raw ticket text before the model is even called (this is that expected_trajectory: [] behavior from Part 3's edge-case bucket) — a legal threat or fraud flag never reaches the LLM at all, straight to a human queue.
Three human-review patterns
Not every consequential action needs the same review pattern. This build uses pre-action approval — human approves before execution — for issue_refund and send_email, because both are rare, high-stakes, and hard to reverse. Two other patterns exist for different risk profiles and are worth knowing even though this particular agent doesn't need them: sampling review (agent acts autonomously, a random 10–20% gets human QA — appropriate for high-frequency, low-individual-risk actions) and confidence routing (only route to a human when the model's self-reported confidence is genuinely calibrated — risky if that confidence isn't actually reliable, which it usually isn't by default).
The gate itself, from src/approval.ts:
export const cliApproval: ApprovalFn = async (tool, args) => {
const answer = await getRl().question(
`\n[APPROVAL REQUIRED] ${tool}(${JSON.stringify(args)})\nApprove? (y/n) `,
);
return answer.trim().toLowerCase().startsWith("y");
};
And how the loop applies it before any gated tool runs (from agent.ts):
} else if (requiresApproval(toolName) && !(await approvalFn(toolName, args))) {
resultPayload = { error: "rejected_by_human_approval" };
isError = true;
}
If a human says no, the tool never executes — the model gets an error result back and has to handle that, same as any other tool failure.
The bug: a claim without a fact behind it
Here's what actually happened. Two eval cases (hard_04, hard_06) came back reporting outcome: refund_proposed, but their tool-call trajectory was only [order_lookup, refund_eligibility] — issue_refund was never in the list. The model had generated a final message starting with REFUND_PROPOSED: directly, skipping the tool call that's supposed to cause that outcome.
Why this matters: the outcome parser trusted the model's own REFUND_PROPOSED: prefix as ground truth. Nothing checked that the claim corresponded to an actual, approved, executed tool call. A "propose a refund" statement is exactly the kind of business-critical, consequential claim that shouldn't rest entirely on the model choosing to comply with a prompt instruction — and here, it was.
The fix — two layers, not one
Layer one, prompt: add an explicit line the model can't miss (already shown in Part 4's system prompt) — "never state REFUND_PROPOSED unless you actually called issue_refund and it succeeded in this run."
Layer two, code — the one that actually matters:
// Guardrail enforced in code, not just the prompt: a model can claim
// REFUND_PROPOSED in text without ever having called issue_refund.
function enforceOutcomeIntegrity(
outcome: { outcome: AgentOutcome; finalText: string },
state: AgentState,
): { outcome: AgentOutcome; finalText: string } {
if (outcome.outcome === "refund_proposed" && !state.artifacts.refund_confirmation_id) {
return {
outcome: "escalated",
finalText: `ESCALATED: model claimed REFUND_PROPOSED without ever calling issue_refund (policy violation) — original: "${outcome.finalText}"`,
};
}
return outcome;
}
state.artifacts.refund_confirmation_id is only ever set in one place — when issue_refund actually succeeds (src/memory.ts):
export function recordStep(state: AgentState, step: TrajectoryStep, result: Record<string, unknown>) {
state.conversation.trajectory.push(step);
if (step.tool_name === "issue_refund" && typeof result.confirmation_id === "string") {
state.artifacts.refund_confirmation_id = result.confirmation_id;
}
return state;
}
So enforceOutcomeIntegrity isn't trusting the model's words at all — it's checking a fact that can only exist if the gated, approved tool call actually happened. If the model claims REFUND_PROPOSED without that fact present, the run gets downgraded to escalated, regardless of how convincing the model's text sounded.
This is the actual lesson, stated plainly: any claim your agent makes about a consequential action should be verifiable from state the code controls, not trusted from text the model generated. The prompt fix alone would probably have reduced the frequency of this bug. It wouldn't have made it impossible. The code guardrail does.
What's next
Part 6: Observability for AI Agents: Tracing, Metrics, and Drift → covers observability — the trace payload this build logs on every single step, and why "run the eval set" isn't the same question as "is this healthy in production."
Top comments (0)