DEV Community

Saurav Bhattacharya
Saurav Bhattacharya

Posted on

Right Tool, Wrong Arguments: The Agent Failure Your Evals Wave Through

Your agent picked the correct tool. It routed to refund_order when the user asked for a refund. Your eval suite went green. And then it issued a $4,200 refund on order #0 because the argument extraction fumbled and defaulted the ID to zero.

This is the failure mode nobody instruments: right tool, wrong arguments. Most eval setups check whether the agent chose the correct action, then wave the arguments through because validating them looks tedious. That gap is where real money leaks.

Tool selection is the easy 20%

Routing to the right tool is a classification problem, and modern models are good at it. The hard, dangerous part is the arguments — the resolved IDs, amounts, filters, and paths the agent synthesizes from messy context. Those are structured claims the agent authored, and they fail in boring, expensive ways:

  • A refund amount pulled from the wrong line item.
  • A file path hallucinated from a plausible-looking directory that doesn't exist.
  • A date filter off by a timezone, silently returning an empty set that reads as "no results."
  • An order_id that got coerced to 0 or null and still satisfied a loose schema.

None of these are subjective. None of them need a model to grade them. They are Tier 1 problems — externally observable proof the agent can't forge — and they're getting handed to a model-as-judge (or worse, to production) because teams conflate "did it call the right tool" with "did it call the tool right."

The independence axis, applied to tool calls

The tier doctrine agent-eval is built on ranks evidence by independence, not cost:

  • Tier 1 — proof the agent can't forge. The arguments parse as valid JSON, satisfy the tool's schema, the referenced order_id actually exists, the amount is within the order total, the path resolves. Deterministic, ~$0, runs in the hot path, can block the call before it executes.
  • Tier 2 — statistical signal against a baseline the agent didn't author. The extracted amount is within a sane distribution for this merchant; the argument set actually changed from the previous turn; the resolved entity embeds close to the entity named in the user's request.
  • Tier 3 — model-as-judge. "Given the conversation, does this refund feel justified?" A signal, never a verdict, and offline only — metered, slow, non-deterministic, no business sitting in your payment path.

The mistake is asking Tier 3 to do Tier 1's job. A judge model evaluating whether order_id: 0 is correct is circular reasoning: it shares a substrate with the agent that produced the argument, and it has no independent ground truth about your database. Whether order 0 exists is not an opinion. Query the database.

Gate the arguments before the tool fires

Tier 1 for tool calls is a validation layer that sits between the model's proposed call and execution. Here it is against a proposed refund:

import { z } from "zod";

const RefundArgs = z.object({
  orderId: z.string().regex(/^\d+$/),
  amountCents: z.number().int().positive(),
});

type ToolCall = { name: string; args: unknown };

async function gateRefund(call: ToolCall, db: OrderStore) {
  // Tier 1a: does it even parse to the contract?
  const parsed = RefundArgs.safeParse(call.args);
  if (!parsed.success) {
    return { ok: false, tier: 1, reason: "args_schema_invalid" };
  }
  const { orderId, amountCents } = parsed.data;

  // Tier 1b: does the referenced entity actually exist?
  const order = await db.find(orderId);
  if (!order) {
    return { ok: false, tier: 1, reason: "order_not_found" };
  }

  // Tier 1c: is the amount observably impossible?
  if (amountCents > order.totalCents) {
    return { ok: false, tier: 1, reason: "amount_exceeds_total" };
  }

  // Tier 2: statistically weird, even if legal?
  if (amountCents > 3 * order.merchant.medianRefundCents) {
    return { ok: false, tier: 2, reason: "amount_outlier" };
  }

  return { ok: true };
}
Enter fullscreen mode Exit fullscreen mode

Every check here is proof or statistics, not opinion. Schema, existence, bounds, distribution. The agent cannot talk its way past order_not_found, because the check consults a source the agent didn't get to write. That's the whole point of the independence axis: the gate is only worth anything if the agent couldn't author the evidence it's judged against.

This is the 80% you ship first. Malformed args, hallucinated IDs, empty result sets, out-of-bounds amounts — the failures that actually page you at 2am — are all caught at Tier 1+2, deterministically, before execution. The subjective ~20% ("was a refund the right call socially?") is the only thing you route to a judge, clearly labeled opinion.

You can't gate what you can't see

This all assumes you have the resolved arguments — what the tool actually received after the model's output was parsed, defaulted, and coerced. Most logging captures the user prompt and the final response and nothing in between. When the refund fires on order 0, your logs show a happy path.

This is where the two halves of the workflow lock together. AgentLens captures the trace: every model step and tool step, the resolved inputs the tool actually saw, and the raw outputs — unforged, agent-didn't-author records of what happened. agent-eval scores and gates against that trace using the tier doctrine above. AgentLens gives Tier 1+2 something real to validate; agent-eval turns it into a red/green decision that can block the call.

Without the trace, your evals are grading the story the agent tells about itself. With it, you're grading the arguments it actually passed — and stopping the $4,200 refund on order 0 before it clears.

The takeaway

Stop congratulating your agent for picking the right tool. Tool selection is classification; argument synthesis is where the risk lives. Validate the arguments as proof, not opinion — schema, existence, bounds — in the hot path, before execution. Reserve the judge for the genuinely subjective tail. And trace the resolved inputs, because you cannot gate what you never recorded.

Top comments (0)