Nearly half our agent's tool-call failures were our assertion, not our agent

We had a number on a dashboard and it would not move. Twenty-two percent of the tool calls our agent made were scored as failures by our eval suite. We spent most of a quarter on it: better tool descriptions, a stricter schema, few-shot examples of correct calls, a model change for the planning step. The number moved between 20 and 24 and settled back around 22.

Then somebody on the team did the obvious thing that none of us had done, which was to read the failures.

She pulled a hundred of them at random and went through them by hand against the tool definitions. Forty-seven were correct calls. The agent had asked for the right thing from the right tool and our assertion had scored it wrong.

I want to be careful here, because the interesting part is not that we had a bug in a test harness. Everybody has had that. The interesting part is what a metric does to a team when a known fraction of it is noise, and how specifically an exact-match assertion is wrong.

What the assertion was

The eval case looked like most agent eval cases I have seen, including ones in framework documentation:

{
  "input": "refund the duplicate charge on Dana Whitfield's March invoice",
  "expected_tool": "search_invoices",
  "expected_args": {"customer_id": "C-4471", "month": "2026-03", "status": ["open", "disputed"]},
}

and the check was, in effect, actual["name"] == expected_tool and actual["args"] == expected_args. A dict comparison. It is the natural thing to write, it is what I would have written, and for a tool surface of four stable tools it is fine.

We had forty tools.

The four shapes of a false failure

The 47 sorted cleanly, which was itself informative. In descending order:

One: equivalent encodings, 18 of 100. "2026-03" against an expected "2026-03-01", both accepted by the tool and both meaning March. amount: 4200 against amount: 42.00 on a tool that took either cents or dollars on the same parameter (that one was our fault twice over). Durations as "PT30M" and as 1800. The largest pile, and in hindsight the most predictable: we had written schemas that accept two encodings per field, so we received two encodings per field, and then compared them byte-for-byte.

Two: argument ordering, 14 of 100. status: ["disputed", "open"] against an expected ["open", "disputed"]. The parameter is a set in every sense that matters to the tool, and a list in JSON. Our assertion compared ordered lists. Around half of these came from one tool that takes three list-valued filters, which is why one badly-shaped tool can dominate a metric.

Three: defaults stated explicitly, 9 of 100. The agent passed limit: 50 where 50 is the documented default and the expected args omitted it. The call is byte-for-byte different and behaviorally identical. A stricter schema, which we shipped in month two of the quarter, made this pile bigger rather than smaller: telling the model to be explicit about parameters is good advice that our assertion punished.

Four: a different valid path, 6 of 100. The agent called get_customer and read the invoice list off the response instead of calling search_invoices. One tool call instead of the expected one, same information, arguably better. Our case had one right answer because writing a case with two right answers is more work.

That leaves 53 real failures out of 100, which puts the true rate near 11.7 percent rather than 22. The quarter of work we had done was not wasted, but we had been measuring it through an instrument that nearly doubled the number it reported, which is why nothing we did looked like it worked.

There is a smaller thing in that paragraph I should own. One of the changes we made that quarter was swapping the planning model, and I have argued in public that function-calling robustness is a model-selection problem more often than teams admit. The swap moved nothing. That is not evidence against the position, because a broken comparator would have absorbed a real improvement too, but it is a reminder that I could not have told the difference at the time, and I was fairly confident anyway.

The second error, which I think is worse

An exact-match assertion is not only too strict. In one specific way it is too permissive, and this is the part I did not see until we went looking for the opposite failure.

The assertion checks that the call matches. It does not check that the call was the right thing to do at that point in the run.

Finding that meant sampling the other way round, because a false pass cannot appear in a pile of failures. So she went back and took 120 runs the suite had scored clean end to end, every tool call in them green, and read those instead. In 11 of the 120, the agent had produced an exactly-matching call for a step it should not have taken at all, usually because it had already retrieved that information two steps earlier and was looping. Argument equality passed every one of those calls, correctly, because each one was right in isolation. A trajectory that repeats a correct call four times is a bad trajectory made of good tool calls, and dict comparison has nothing to say about it.

I have written before that you should grade the trajectory rather than the final answer. This is that argument arriving from the other side: it is not only that step-level grading sees more, it is that per-call grading actively certifies steps a trajectory view would fail.

Then the human effect compounded it. Once you have triaged twenty tool-call "failures" and found that half are nonsense, you stop triaging them. Two engineers on our team had independently stopped opening that dashboard section. The 53 genuine failures were sitting in a list nobody read. I think that is worse than having no metric at all, because nobody goes looking for a replacement while the dashboard still has a number on it.

What we replaced it with

Three levels, checked in order, and the important design choice is where the rules live.

_MISSING = object()

def ident(v):
    return v

NORMALIZERS = {
    "unordered_list": lambda v: sorted(v) if isinstance(v, list) else v,
    "month":          lambda v: to_utc(v).strftime("%Y-%m"),   # NOT str(v)[:7]. see below
                                           # to_utc: parses ISO with offset, returns tz-aware UTC
    "money_cents":    lambda v: int(round(float(v) * 100)) if isinstance(v, float) else int(v),
    "duration_secs":  to_seconds,          # accepts "PT30M" | 1800 | "30m"
}

# equivalence declared per PARAMETER on the tool, not per test case
search_invoices.eval_norms = {"status": "unordered_list", "month": "month"}

def args_equivalent(tool, actual, expected):
    norms = getattr(tool, "eval_norms", {})
    def norm(k, v):
        return NORMALIZERS.get(norms.get(k), ident)(v)

    # normalize the DEFAULTS too, or a restated default fails on encoding
    defaults = {k: norm(k, v) for k, v in tool.defaults().items()}
    a = {k: norm(k, v) for k, v in actual.items()}
    e = {k: norm(k, v) for k, v in expected.items()}

    # a stated default is equivalent to an omitted one, in both directions.
    # _MISSING, not .get(k), so "no default declared" never collides with
    # "the default is None" and silently swallows a real difference.
    a = {k: v for k, v in a.items() if not (k not in e and defaults.get(k, _MISSING) == v)}
    e = {k: v for k, v in e.items() if not (k not in a and defaults.get(k, _MISSING) == v)}
    return a == e

Level one is tool identity, unchanged. Level two is that function. Level three, for the eight tools with a queryable effect, ignores the call shape entirely and asserts on the world afterwards: did a refund of this amount exist against this invoice. Effect assertions are the ones I trust, and they are the only ones that survive a tool being refactored, which happened twice while we were doing this.

Two of those normalizer names carry a warning. money_cents only works because both encodings arrive on one parameter. Equivalence rules normalize values under a key, they do not alias one key to another, so a tool exposing both an amount and an amount_cents parameter is a schema problem no comparator will fix for you. And month is the one that bit us, which I will come to.

Case four, the alternate valid path, is not fixed by any of that. We changed the case format to accept a set of acceptable tool sequences, and to prefer an effect assertion where one exists:

{
  "input": "refund the duplicate charge on Dana Whitfield's March invoice",
  "accept": [
    [{"tool": "search_invoices",
      "args": {"customer_id": "C-4471", "month": "2026-03", "status": ["open", "disputed"]}}],
    [{"tool": "get_customer", "args": {"customer_id": "C-4471"}}],
  ],
  "effect": lambda world: world.refunds.exists(invoice="INV-9912", cents=4200),
}

We wrote alternates for 11 eval cases where a second path obviously existed, which is a different 11 from the looping runs above, and accepted that we will keep discovering more. That pile is the one where I think honest measurement is genuinely hard rather than merely neglected.

What it cost, and what it got wrong

We wrote normalizers for 9 of 40 tools. Those nine carry 81 percent of call volume, which is the only reason this was a two-week job and not a quarter. The measured failure rate went from 22 percent to 12.4, against a hand-counted estimate of 11.7. I read the gap as mostly the 31 un-normalized tools, and I would rather report both numbers than pretend the estimate and the measurement agreed.

One normalizer was wrong in a way that should worry anyone doing this. Our first version of month was str(v)[:7]. Feed it "2026-03-01T00:40:00+01:00" and it returns "2026-03", so it compares equal to an expected March. The instant that string names is 23:40 UTC on the 28th of February. It is a February call wearing a March prefix, and the assertion said fine. A rule written to remove false failures had quietly created a false pass.

We caught it because the effect assertion on that tool disagreed with the argument assertion, which is a decent argument for keeping both even where they overlap. Exact-match fails loudly and wrongly. A normalizer fails quietly and wrongly. I would still take the normalizer, but only with something checking it.

Where I'd push back on this

The strongest version of the other side is not that exact-match is easy. It is that exact-match is the only assertion in the list that cannot lie to you about a passing call, because it contains no code of your own. Every equivalence rule I add is an untested hypothesis about my own tool. Our timezone incident is exactly the failure the sceptic predicts, and we found it by accident.

If your tool surface is small, your parameters are scalars, and your schema admits one encoding per field, exact-match is correct and the rest of this post is overhead you should not buy. Most of our 47 came from three things we chose: list-valued filters, permissive schemas, and forty tools. Tightening the schema to one encoding per parameter would have removed the largest pile at the source, and it is a better fix than normalizing around it. We did some of that afterwards and I would do it first next time.

What I would not concede is the inference people draw from a stable bad number. For a quarter we treated 22 percent as a property of the agent. It was a property of the agent and the assertion together, and we had never separated them. Reading a hundred failures by hand cost one engineer two days. The 120 clean runs cost her three more, because a whole run takes longer to read than a single call. Five engineer-days total, and it was worth more than everything else we tried that quarter.