The tool-call number I published has expired. The claim it supported has not.

Function-calling robustness is a model selection problem masquerading as a prompt problem. I proved that with a percentage, which was the mistake.

TL;DR. Two months ago I posted a number in a comment thread: same tool description, 1,000 enterprise queries, one model called the tool correctly 96% of the time and another managed 88%. People liked it. Someone quoted it back at me in a design review last week to justify a model choice, and the quote was doing work the number could not support any more. The claim underneath it is structural and still true: if your agent ignores a tool, the cause is more often the model you picked than the words you wrote. The evidence I gave for it has a shelf life of about one release cycle, and I published it with no expiry date on it. This is what I should have shipped instead: the harness, the metric definition, and the decision rule. Your numbers, not mine.

The design review

We were picking a model for a new extraction flow. Document in, structured record out, one tool call to fetch the customer's contract template before the extraction runs. Ordinary work.

Halfway through, an engineer on the team pulled up my comment. He had found it while searching for exactly this decision, which is how these things go. "O'Connor got 96 versus 88 on this, we should just use the same model."

He was quoting me correctly. That is the part that bothered me.

The test he was quoting ran in late 2025. It compared two models that are now two releases back on both sides. It used our tool descriptions, our schema depth, our document mix, and a definition of "called the tool correctly" that I never wrote down anywhere he could read. He had none of that context. He had a percentage, my name attached to it, and a decision to make on a Tuesday.

I have been on the other side of this. I have cited someone's benchmark from a blog post because it was the only number I could find and the meeting was in ten minutes. Everyone does it. The number is not lying to you, exactly. It is answering a question that was asked somewhere else, about something else, a while ago.

So this post is partly a correction and partly an apology to anyone who has quoted that comment since May.

What the number actually was

I want to be precise about it, because the imprecision is the whole point.

Late last year we shipped a contract-extraction agent. It had one tool worth arguing about: fetch_template(customer_id, doc_type). The agent was supposed to call it before extracting, so that extraction ran against the right schema. When it skipped the call, extraction ran against a generic schema and quietly produced worse output. Not an error. Just worse.

We were seeing skips. The team's instinct, and mine at first, was that the tool description was bad. So we did what everyone does. We rewrote it. We made it more imperative. We added "ALWAYS call this first." We moved it up in the tool list. We added an example. Each change bought a couple of points and cost a week.

After about three weeks of this I got annoyed enough to run the comparison properly. Same tool description, byte for byte. Same 1,000 queries sampled from real traffic. Two models. The gap was eight points, and it was larger than everything the prompt rewrites had bought us put together.

We hard-routed that flow to the model that won and stopped iterating on the prompt. That was the right call and I would make it again.

Then I wrote "96% versus 88%" in a comment box and walked away from it.

Three things that broke the number

1. The models moved, and they did not move in parallel

This is the obvious one and it is still worse than people expect.

The intuition most teams carry is that model releases lift all boats, so a gap measured last year roughly holds this year, just at higher absolute numbers. That intuition is wrong in a specific way: tool-calling behaviour is shaped by post-training choices that are not on the same schedule as general capability. A vendor can ship a model that reasons better and calls your tool less often, because they retuned how eagerly it reaches for tools, or changed how it handles a tool whose description sounds optional.

I have watched a model get better at the task and worse at the tool call in the same release. If you are only tracking the end metric, that shows up as noise. If you are tracking the tool call separately, it shows up as a decision.

Which direction any given release moved is a question about this month, and if I answer it here that answer rots too. That is not me dodging. It is the actual finding.

2. It was my workload, not yours

The eight-point gap was measured on one tool, in one schema, in one document domain, with one tool-list length.

Every one of those is load-bearing:

• Tool count. One tool is a different problem from fourteen. Selection pressure between similar-sounding tools is a separate failure mode from whether the model reaches for a tool at all, and models that are good at one are not automatically good at the other.
• Schema depth. A flat two-field schema and a nested schema with optional sub-objects do not fail the same way. (I have written about optional fields before. They remain the sharpest edge in this whole area.)
• Description ambiguity. Our description was decent. If yours is genuinely ambiguous, you have a prompt problem sitting on top of your model problem, and my number tells you nothing about the ratio.
• Domain. Contracts. If you are routing support tickets, the retrieval-shaped context is different enough that I would not transfer the number across.

So the honest scope of "96 versus 88" is: for this tool, in this schema, on this traffic, at that time. Anyone outside that scope is reading a number that was never about them.

3. The metric was underspecified, which is my fault

Here is the one that embarrasses me.

"Called the tool correctly" is at least four different metrics:

1. Did it call the tool at all, when it should have?
2. Did it call the right tool, when several were plausible?
3. Did it pass arguments that validate against the schema?
4. Did it pass arguments that were semantically right, meaning valid and also correct?

My 96 and 88 were mostly metric 1, with a bit of 3 folded in because a call that failed validation got counted as a miss. I did not say that. Someone reading the comment could reasonably assume I meant 4, which is the one they actually care about and the one where the gap between models is different again.

That gap between 3 and 4 is where most production pain lives. A tool call can validate perfectly and be wrong. My number does not speak to that at all, and it was quoted at me as if it did.

What I should have published

Not a percentage. A harness that produces yours.

Here is the shape we use now. It is deliberately small. The point is not that it is clever, it is that it takes under an hour to point at your own traffic, and after that you never have to cite a stranger's blog post in a design review again.

from enum import Enum
from typing import Any, Callable, Sequence

from pydantic import BaseModel, ValidationError


class Outcome(str, Enum):
    """The metrics people collapse into 'it worked'. Keep them apart."""

    NO_CALL = "no_call"                  # metric 1: should have called, didn't
    SPURIOUS_CALL = "spurious_call"      # should have called nothing, called anyway
    WRONG_TOOL = "wrong_tool"            # metric 2: called something else
    INVALID_ARGS = "invalid_args"        # metric 3: args failed schema validation
    VALID_BUT_WRONG = "valid_but_wrong"  # metric 4: schema-valid, semantically wrong
    CORRECT = "correct"


class Case(BaseModel):
    """One row of your traffic, with the answer you'd have wanted."""

    query: str
    expected_tool: str | None            # None = the model correctly calls nothing
    expected_args: dict[str, Any] | None


class Result(BaseModel):
    case: Case
    model: str
    outcome: Outcome


def grade(
    case: Case,
    tool_name: str | None,
    raw_args: dict[str, Any] | None,
    arg_schema: type[BaseModel],
    semantic_check: Callable[[Case, BaseModel], bool],
) -> Outcome:
    if tool_name is None:
        return Outcome.CORRECT if case.expected_tool is None else Outcome.NO_CALL
    if case.expected_tool is None:
        # It reached when it should have sat still. That is not tool *selection*.
        return Outcome.SPURIOUS_CALL
    if tool_name != case.expected_tool:
        return Outcome.WRONG_TOOL
    try:
        parsed = arg_schema.model_validate(raw_args or {})
    except ValidationError:
        return Outcome.INVALID_ARGS
    # The step almost everyone skips. Schema-valid is not the same as right.
    return Outcome.CORRECT if semantic_check(case, parsed) else Outcome.VALID_BUT_WRONG

And the runner, which is the boring part that matters:

def run_matrix(
    cases: Sequence[Case],
    models: Sequence[str],
    call_model: Callable[[str, str], tuple[str | None, dict[str, Any] | None]],
    arg_schema: type[BaseModel],
    semantic_check: Callable[[Case, BaseModel], bool],
    repeats: int = 3,
) -> list[Result]:
    """Same cases, same tool description, every model. repeats>1 because
    these are sampled, not deterministic, and a 2-point 'gap' across a
    single pass is usually just temperature."""
    out: list[Result] = []
    for model in models:
        for case in cases:
            for _ in range(repeats):
                tool_name, raw_args = call_model(model, case.query)
                out.append(
                    Result(
                        case=case,
                        model=model,
                        outcome=grade(
                            case, tool_name, raw_args, arg_schema, semantic_check
                        ),
                    )
                )
    return out

Three things about this that are not obvious until you have run it wrong once.

repeats defaults to 3 and should probably be higher. These calls are sampled. Run 200 cases once against two models, see 94 and 91, and you have learned almost nothing. I have watched a "gap" evaporate on the second pass. If the difference you are chasing is inside the run-to-run spread, you have not found a difference, you have found the noise floor.

semantic_check is yours and nobody can write it for you. For fetch_template ours is roughly "does this customer_id belong to the customer the ticket is about, and is doc_type one this customer actually has." It is not glamorous. It is also the only part that measures the thing you care about, which is why every generic harness stops at validation and leaves the interesting metric on the floor.

Keep the tool description byte-identical across models. The first time we ran this we had per-model prompt tweaks left over from the three weeks of iteration, so we were comparing prompt-plus-model against prompt-plus-model and calling it a model comparison. That run told us nothing and we nearly shipped a decision off it.

The decision rule

The harness gives you a distribution across six outcomes per model. What you do with it:

Mostly NO_CALL, and it varies a lot by model. Model selection. Stop editing the prompt. This was us. If the spread across models is wider than the spread you can buy with prompt changes, you have your answer, and the prompt work you were about to do is a tax you are choosing to pay.

Mostly NO_CALL, and every model does it equally. Now it is a prompt problem, and specifically a description problem. The models agree with each other and they all disagree with you. That is information about your description, not about them.

Mostly SPURIOUS_CALL. The model reaches when it should sit still. Same axis as NO_CALL pointing the other way, and it moves between releases for the same reason. Same rule: check the spread across models before you rewrite the description.

Mostly WRONG_TOOL. Usually a naming and boundary problem, and it usually does not go away by switching models. Two tools that sound alike will confuse a better model too. I have written about naming before and I still think it is underrated relative to how cheap it is to fix.

Mostly INVALID_ARGS. Constrained decoding or a validation-retry loop, and bound the retries. A retry loop with no ceiling turns one bad request into a bill.

Mostly VALID_BUT_WRONG. The hard one. No model switch saves you. This is a precheck and eval problem, and it is where I would spend the time if the other four buckets are clean.

The rule I would have written into that comment if a comment box made you write rules instead of numbers: run the matrix before you touch the prompt, because the matrix is an afternoon and the prompt iteration is a quarter.

Why I am not publishing new numbers

I re-ran this on current models before writing this. I am not going to print what I got.

Not coyness. Two reasons.

The first is that it would recreate this exact post in six months. Someone would quote it in a design review in January, the models would have moved twice, and the number would be doing the same unearned work.

The second is that my re-run is still my workload. Same tool, same schema, same domain. Publishing it dressed up as a general finding is precisely the error I am describing, and doing it knowingly would be worse than doing it by accident in a comment box.

What I will say is about the method, not a leaderboard, and you can check it against your own run instead of taking it from me. Nothing forces a model's buckets to move together. A model can reach for the tool more often and be worse at argument semantics, and averaged into a single "correctness" percentage those two cancel. That is what my original number did. It collapsed buckets that were telling different stories into one figure that hid the decision instead of informing it. Which is a fifth thing broken about it, and arguably the worst one.

Jason Liu has been saying a version of this for years in the Instructor docs and around them: the interesting work is in the schema and the validation, not in the sentence you write above it. I read that, agreed with it, and then went and published a sentence-level percentage anyway. Anthropic's "Building Effective Agents" makes a nearby point about tool documentation deserving the same care as an API you hand to a junior engineer. Both hold up better than my comment did.

Where I'd push back on this

The steelman is real, and it deserves a proper hearing.

Published numbers are how anyone starts. If every practitioner refused to publish measurements because they might be misused, nobody would have a prior, and every team would rediscover from zero that model choice matters here. My 96 versus 88 did do work: it made people run their own tests. That is a real contribution, and a purist "measure it yourself" position is a bit rich coming from someone who benefited from other people's benchmarks for years.

I will concede that. What I would not concede is publishing it naked. A number with its scope, its metric definition, its date, and its models attached is a contribution. The same number in a comment box with none of that is a liability with my name on it.

And "just run it yourself" is not free. An afternoon of engineering time is an afternoon you do not have, and 200 labeled cases with a real semantic_check is more like two days than one afternoon if you are honest about the labeling. For a team of three shipping on a deadline, taking a stranger's number and moving on is a defensible call. I have made it. The thing I would ask is that you know you are making it, and that you write down which model and which month the number came from, so that when it stops being true you can find out.

The strongest objection is that the claim itself might not survive. "Function-calling robustness is a model selection problem" is a statement about a period in which models differ a lot on this axis. If tool-calling reliability commoditizes and every frontier model lands within a point of the others, my claim becomes a historical note and the prompt people were right all along, just early. I do not think that has happened as of July 2026, because I keep measuring gaps that are wider than my prompt work can close. But I hold the claim more loosely than I hold the method. The method survives either way. If the gap goes to zero, the matrix tells you that too, and then you go and fix your description with a clear conscience.

The number expires. The harness does not.

Comments

[Luis Cruz]

I was particularly intrigued by the idea that function-calling robustness is a model selection problem rather than a prompt problem, and how the 96% vs 88% comparison was used to justify a model choice without considering the context and shelf life of the data. The fact that the models have moved and didn't move in parallel is a crucial point, and it highlights the importance of continuously monitoring and updating benchmarks. It's also interesting to note that the decision to hard-route the flow to the winning model was based on a comparison that was only valid at that specific point in time. What strategies do you think teams can use to ensure that their benchmarks and comparisons remain relevant and useful over time, especially in rapidly evolving model landscapes?