DEV Community

Михаил
Михаил

Posted on • Originally published at agentlabjournal.online

LLM Function Calling: Comparing Schemas and Tool Selection Across Providers

#ai #automation #agents

LLM Function Calling: A Reproducible Provider Comparison | Agent Lab Journal

  Agent Lab Journal

    Guides
    Glossary
Enter fullscreen mode Exit fullscreen mode

Practical evaluation · Intermediate

LLM Function Calling: Comparing Schemas and Tool Selection Across Providers 

      35 min read
      Intermediate
      Updated August 1, 2026




      Give several models the same function description and they may choose different
      tools, omit different arguments, normalize values differently, or recover from
      errors in incompatible ways. A reliable integration therefore cannot be judged
      by one successful demo. It needs a repeatable test that separates model behavior,
      provider protocol, schema validation, and application-side recovery.
Enter fullscreen mode Exit fullscreen mode

Contents

  • Why identical tools behave differently

  • The concrete test case

  • Define a provider-neutral contract

  • Create the test dataset

  • Normalize provider responses

  • Build the benchmark runner

  • Score selection, arguments, and recovery

  • Test error handling

  • Verify reproducibility

  • Common failure cases

  • Limitations

Why identical tools behave differently 

      Function calling is a protocol in
      which a model requests that the host application execute a named operation with
      structured arguments. The model does not run the function itself. It produces a
      proposed call; your application validates it, performs the operation, and may
      return a result to the model.




      In practice, llm function calling is not one universal wire
      format. Providers differ in request fields, supported
      JSON Schema features, tool-choice
      controls, response envelopes, streaming events, parallel-call behavior, and the
      messages required to return a tool result. Models add another layer of variation:
      they interpret descriptions and ambiguous user requests differently.




      The surrounding llm api also matters. A provider may reject an
      unsupported schema keyword before inference, silently ignore it, or accept the
      schema while the model later violates it. Those outcomes should not be collapsed
      into a single “function calling failed” metric.
Enter fullscreen mode Exit fullscreen mode

A useful comparison separates four stages:

  • Request acceptance: did the provider accept the tool schema?

  • Tool decision: did the model call, abstain, or choose the wrong tool?

  • Argument construction: were the supplied values structurally and semantically correct?

  • Recovery: after a validation or execution error, did the model repair the call appropriately?

      Core rule: record raw requests and responses, but score a
  normalized representation. Without both, provider syntax can distort the
  comparison and normalization bugs can become invisible.

The concrete case: a support operations assistant 

      We will test three read-only tools for a fictional support workflow. No real
      customer data, credentials, or measured provider results are included. The
      benchmark produces its own evidence when you run it against the models you are
      authorized to use.
Enter fullscreen mode Exit fullscreen mode 
{
  "tools": [
    {
      "name": "lookup_order",
      "description": "Find one order by its exact public order ID.",
      "parameters": {
        "type": "object",
        "properties": {
          "order_id": {
            "type": "string",
            "pattern": "^ORD-[0-9]{6}$",
            "description": "Public order ID, for example ORD-104821."
          }
        },
        "required": ["order_id"],
        "additionalProperties": false
      }
    },
    {
      "name": "search_orders",
      "description": "Search orders when an exact order ID is not available.",
      "parameters": {
        "type": "object",
        "properties": {
          "email": {
            "type": "string",
            "description": "Customer email address."
          },
          "status": {
            "type": "string",
            "enum": ["processing", "shipped", "delivered", "cancelled"]
          },
          "limit": {
            "type": "integer",
            "minimum": 1,
            "maximum": 20,
            "default": 5
          }
        },
        "required": ["email"],
        "additionalProperties": false
      }
    },
    {
      "name": "get_shipping_quote",
      "description": "Estimate shipping price and delivery window for a destination.",
      "parameters": {
        "type": "object",
        "properties": {
          "country_code": {
            "type": "string",
            "pattern": "^[A-Z]{2}$",
            "description": "ISO-style two-letter uppercase country code."
          },
          "postal_code": {
            "type": "string",
            "description": "Postal code as user-provided text; preserve leading zeros."
          },
          "weight_grams": {
            "type": "integer",
            "minimum": 1,
            "maximum": 50000
          },
          "service": {
            "type": "string",
            "enum": ["standard", "express"]
          }
        },
        "required": ["country_code", "postal_code", "weight_grams"],
        "additionalProperties": false
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode 
      This tool set exposes common problems without requiring destructive actions:
      overlapping lookup and search functions, formatted identifiers, enums, bounded
      integers, optional defaults, uppercase normalization, and postal codes that must
      remain strings.




      The distinction between lookup_order and
      search_orders is especially valuable. A model that merely notices
      the word “order” cannot pass consistently; it must decide whether the user
      supplied an exact identifier.
Enter fullscreen mode Exit fullscreen mode

Define a provider-neutral contract first 

      Do not begin by copying one provider’s request object throughout the application.
      Define a small internal representation, then translate it at the boundary of each
      llm api. This gives the benchmark one source of truth.
Enter fullscreen mode Exit fullscreen mode 
export type CanonicalTool = {
  name: string;
  description: string;
  parameters: Record<string, unknown>;
};

export type CanonicalToolCall = {
  id: string | null;
  name: string;
  arguments: unknown;
};

export type CanonicalOutcome =
  | { kind: "tool_calls"; calls: CanonicalToolCall[] }
  | { kind: "text"; text: string }
  | { kind: "refusal"; text: string | null }
  | { kind: "provider_error"; code: string; message: string };
Enter fullscreen mode Exit fullscreen mode 
      Keep parsed arguments as unknown until validation. If an adapter
      casts "750" to 750, removes an unexpected property, or
      supplies a default before scoring, it hides the model’s actual output.
Enter fullscreen mode Exit fullscreen mode

Use the schema intersection for the main comparison 

      Providers support different schema subsets. Start the primary benchmark with
      conservative constructs: object, string, integer, number, boolean, array,
      properties, required, enum, and simple bounds. Keep advanced keywords in a
      separate compatibility suite.




      Maintain two schema files:
Enter fullscreen mode Exit fullscreen mode

  • tools.core.json contains the portable schema used to compare models.

  • tools.compatibility.json probes keywords such as patterns, unions, nested constraints, and strictness controls.

      If a provider cannot accept pattern, for example, remove it only in
  that provider’s compatibility adapter and record the transformation. Do not then
  claim that all models received semantically identical constraints.

Freeze the instructions 

You are a support operations assistant.

Use a tool only when the user's request requires information supplied by that tool.
Never invent missing required values.
If exactly one required value is missing, ask one concise clarification question.
Use lookup_order only when an exact public order ID is present.
Use search_orders when the user supplies an email address but no exact order ID.
Do not call tools for greetings, explanations of policy, or unsupported actions.
Enter fullscreen mode Exit fullscreen mode 
      Store this text in a versioned fixture. Whitespace changes, extra examples, and
      provider-specific instructions are experimental variables, not harmless edits.
Enter fullscreen mode Exit fullscreen mode

Create a diagnostic test dataset 

      A test row needs more than an expected function name. It should state whether a
      call is expected, the exact acceptable arguments, and why alternative behavior
      is wrong.
Enter fullscreen mode Exit fullscreen mode 
{"id":"select-lookup-01","prompt":"Where is order ORD-104821?","expected":{"kind":"tool","name":"lookup_order","arguments":{"order_id":"ORD-104821"}},"tags":["selection","exact-id"]}
{"id":"select-search-01","prompt":"Show my shipped orders. My email is sam@example.test.","expected":{"kind":"tool","name":"search_orders","arguments":{"email":"sam@example.test","status":"shipped"}},"tags":["selection","enum"]}
{"id":"abstain-01","prompt":"Hello! What can you help me with?","expected":{"kind":"text"},"tags":["abstention"]}
{"id":"clarify-01","prompt":"How much is express shipping to Germany for 750 grams?","expected":{"kind":"clarification","missing":["postal_code"]},"tags":["missing-required"]}
{"id":"types-01","prompt":"Quote standard shipping for 750 grams to US postal code 02139.","expected":{"kind":"tool","name":"get_shipping_quote","arguments":{"country_code":"US","postal_code":"02139","weight_grams":750,"service":"standard"}},"tags":["types","leading-zero"]}
{"id":"normalize-01","prompt":"Shipping quote for 1 kg to ca, M5V 3L9.","expected":{"kind":"tool","name":"get_shipping_quote","arguments":{"country_code":"CA","postal_code":"M5V 3L9","weight_grams":1000}},"tags":["normalization","unit-conversion"]}
{"id":"unsupported-01","prompt":"Cancel every order on my account.","expected":{"kind":"text"},"tags":["unsupported","safety"]}
{"id":"conflict-01","prompt":"Find ORD-104821. My email is sam@example.test.","expected":{"kind":"tool","name":"lookup_order","arguments":{"order_id":"ORD-104821"}},"tags":["selection","overlap"]}
Enter fullscreen mode Exit fullscreen mode 
      Use a reserved domain such as example.test and synthetic identifiers.
      The fixture does not need a live order database because the first evaluation
      stage stops after the proposed call.
Enter fullscreen mode Exit fullscreen mode

Cover five behavior classes 

  •     Positive selection: the request clearly maps to exactly one
    function.

  •     Confusable selection: two tools share vocabulary, but only one
    satisfies the request.

  •     Abstention: no tool is necessary or available.

  •     Argument pressure: values require conversion, normalization,
    or preservation of string formatting.

  •     Missing information: the model must ask instead of guessing a
    required value.

  Add paraphrases, but keep each original prompt. Replacing cases over time makes
  historical comparisons impossible. A practical convention is
  dataset_version: "1.0.0" in every run manifest.

Normalize provider requests and responses 

      Each provider adapter should have three responsibilities:
Enter fullscreen mode Exit fullscreen mode

  • Translate canonical tools and messages into the provider request.

  • Extract text, refusals, errors, and every proposed tool call.

  • Preserve the unmodified request and response for diagnosis.

      Keep the rest of the runner provider-agnostic. The following interface is enough
  for a first benchmark:

export interface ModelAdapter {
  provider: string;
  model: string;

  run(input: {
    system: string;
    prompt: string;
    tools: CanonicalTool[];
    toolChoice: "auto" | "required" | "none";
    temperature?: number;
  }): Promise<{
    outcome: CanonicalOutcome;
    rawRequest: unknown;
    rawResponse: unknown;
    latencyMs: number;
    usage?: {
      inputTokens?: number;
      outputTokens?: number;
    };
  }>;
}
Enter fullscreen mode Exit fullscreen mode

Do not compare provider controls as if they were identical 

      A mode named auto can allow either text or a call. A mode named
      required may force any tool, a particular tool, or at least one
      tool, depending on the provider. Document the actual semantics beside every
      adapter.




      Run the main selection test with the closest available equivalent of
      auto. Run forced-tool behavior as a separate experiment. Otherwise,
      a provider with a forced call may appear to have better recall while producing
      unusable calls on abstention cases.
Enter fullscreen mode Exit fullscreen mode

A note on function calling openai integrations 

      When implementing function calling openai support, isolate the
      current request and response translation in its own adapter. Do not make the
      canonical format depend on provider-specific message roles, call IDs, strictness
      switches, or streaming event names. Apply the same rule to every other provider.
      This article intentionally avoids fixed endpoint and model identifiers because
      those operational details can change; pin the exact values available in your
      environment inside the run manifest.
Enter fullscreen mode Exit fullscreen mode

Normalize calls without repairing them 

export function normalizeArguments(value: unknown): unknown {
  if (typeof value !== "string") return value;

  try {
    return JSON.parse(value);
  } catch {
    return {
      __parse_error: true,
      __raw: value
    };
  }
}
Enter fullscreen mode Exit fullscreen mode 
      Parsing a JSON string is protocol normalization. Converting units, fixing enum
      spelling, removing unknown keys, or changing types is semantic repair. Score
      those repairs separately if your production stack performs them.
Enter fullscreen mode Exit fullscreen mode

Build the benchmark runner 

      The runner executes every case against every configured model, repeats cases,
      writes append-only records, and never calls a real business tool during the
      selection stage.
Enter fullscreen mode Exit fullscreen mode 
type TestCase = {
  id: string;
  prompt: string;
  expected:
    | { kind: "tool"; name: string; arguments: Record<string, unknown> }
    | { kind: "text" }
    | { kind: "clarification"; missing: string[] };
  tags: string[];
};

for (const adapter of adapters) {
  for (const testCase of cases) {
    for (let repeat = 0; repeat < config.repeats; repeat++) {
      const startedAt = new Date().toISOString();

      try {
        const result = await adapter.run({
          system: fixtures.system,
          prompt: testCase.prompt,
          tools: fixtures.tools,
          toolChoice: "auto",
          temperature: 0
        });

        await appendRecord({
          runId,
          startedAt,
          datasetVersion: config.datasetVersion,
          provider: adapter.provider,
          model: adapter.model,
          caseId: testCase.id,
          repeat,
          expected: testCase.expected,
          outcome: result.outcome,
          latencyMs: result.latencyMs,
          usage: result.usage,
          rawRequest: redact(result.rawRequest),
          rawResponse: redact(result.rawResponse)
        });
      } catch (error) {
        await appendRecord({
          runId,
          startedAt,
          provider: adapter.provider,
          model: adapter.model,
          caseId: testCase.id,
          repeat,
          harnessError: serializeError(error)
        });
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode 
      A temperature of zero can reduce variation, but it does not prove deterministic
      behavior. Providers may change serving infrastructure, model revisions, or
      decoding internals. Repeat each case enough times to observe instability rather
      than assuming it away.
Enter fullscreen mode Exit fullscreen mode

Suggested project layout 

function-benchmark/
├── fixtures/
│   ├── system.txt
│   ├── tools.core.json
│   ├── tools.compatibility.json
│   └── cases.jsonl
├── src/
│   ├── adapters/
│   │   ├── provider-a.ts
│   │   └── provider-b.ts
│   ├── normalize.ts
│   ├── validate.ts
│   ├── score.ts
│   └── run.ts
├── results/
├── package.json
├── tsconfig.json
└── benchmark.config.json
Enter fullscreen mode Exit fullscreen mode

Minimal local commands 

mkdir function-benchmark
cd function-benchmark
npm init -y
npm install ajv
npm install --save-dev typescript tsx @types/node
npx tsc --init
npx tsx src/run.ts --config benchmark.config.json
Enter fullscreen mode Exit fullscreen mode 
      Install provider SDKs only for adapters you actually use, and pin their versions
      in the lockfile. Prefer environment-variable names scoped to each provider.
      Never write secrets into fixtures, result files, console snapshots, or version
      control.
Enter fullscreen mode Exit fullscreen mode 
{
  "datasetVersion": "1.0.0",
  "repeats": 5,
  "temperature": 0,
  "timeoutMs": 30000,
  "concurrency": 1,
  "models": [
    {
      "provider": "provider-a",
      "model": "replace-with-pinned-model-id"
    },
    {
      "provider": "provider-b",
      "model": "replace-with-pinned-model-id"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode 
      Start with concurrency one. It makes rate limits and response ordering easier to
      diagnose. Increase concurrency only after the baseline is stable, then record it
      as part of the experiment.
Enter fullscreen mode Exit fullscreen mode

Score selection and arguments separately 

      A single pass rate conceals important differences. A model can select the right
      tool but invent a required argument; another can produce perfect arguments for
      the wrong operation. Treat these as separate failures.
Enter fullscreen mode Exit fullscreen mode

1. Request acceptance 

      Record whether the llm api accepted the request. Recommended
      categories are:
Enter fullscreen mode Exit fullscreen mode

  • accepted

  • schema_rejected

  • authentication_error

  • rate_limited

  • timeout

  • provider_error

  • harness_error

      Do not count authentication, rate-limit, or harness failures as incorrect model
  decisions. Report them as coverage loss.

2. Tool selection

Build a confusion matrix with these outcomes:

  • correct tool;

  • wrong tool;

  • tool called when abstention was expected;

  • no tool when a call was expected;

  • multiple calls when exactly one was expected;

  • clarification when the fixture expected clarification;

  • clarification when sufficient information was already present.

      The last distinction matters. Asking a safe but unnecessary question may avoid a
  malformed call while still degrading the user experience.

3. Structural argument validity 

      Validate the untouched model arguments against the canonical schema. With Ajv:
Enter fullscreen mode Exit fullscreen mode 
import Ajv from "ajv";

const ajv = new Ajv({
  allErrors: true,
  strict: false,
  useDefaults: false,
  coerceTypes: false,
  removeAdditional: false
});

export function validateCall(
  schema: Record<string, unknown>,
  args: unknown
) {
  const validate = ajv.compile(schema);
  const valid = validate(args);

  return {
    valid: Boolean(valid),
    errors: validate.errors ?? []
  };
}
Enter fullscreen mode Exit fullscreen mode 
      The disabled coercion, default insertion, and property removal are deliberate.
      They prevent the validator from improving the output before it is measured.
Enter fullscreen mode Exit fullscreen mode

4. Semantic argument correctness 

      Schema validity is necessary but insufficient. weight_grams: 1 is a
      valid integer but is wrong when the user said “1 kg.” Compare values against the
      fixture after validation.
Enter fullscreen mode Exit fullscreen mode 
export function compareExpected(
  expected: Record<string, unknown>,
  actual: Record<string, unknown>
) {
  const missing = Object.keys(expected).filter((key) => !(key in actual));
  const wrong = Object.entries(expected)
    .filter(([key, value]) => key in actual && actual[key] !== value)
    .map(([key, value]) => ({
      key,
      expected: value,
      actual: actual[key]
    }));

  const unexpected = Object.keys(actual)
    .filter((key) => !(key in expected));

  return {
    exact: missing.length === 0 && wrong.length === 0 && unexpected.length === 0,
    missing,
    wrong,
    unexpected
  };
}
Enter fullscreen mode Exit fullscreen mode 
      Some optional arguments admit multiple correct outputs. Encode explicit
      assertions rather than forcing exact object equality:
Enter fullscreen mode Exit fullscreen mode 
{
  "arguments": {
    "requiredEquals": {
      "email": "sam@example.test",
      "status": "shipped"
    },
    "optional": ["limit"],
    "constraints": {
      "limit": { "integer": true, "minimum": 1, "maximum": 20 }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

5. Stability 

      For each model and case, group repeated normalized outcomes by a stable hash. A
      simple stability measure is:
Enter fullscreen mode Exit fullscreen mode 
stability = count(most_common_normalized_outcome) / successful_repeats
Enter fullscreen mode Exit fullscreen mode 
      Report the numerator and denominator. A value without the number of successful
      repeats can be misleading, especially when provider errors removed observations.
Enter fullscreen mode Exit fullscreen mode

Recommended report columns 

            Field
            What it reveals




            Accepted requests
            Protocol and schema compatibility


            Selection accuracy
            Whether the correct tool or abstention was chosen


            Schema-valid calls
            Structural compliance before repair


            Semantically exact calls
            Whether arguments match the user’s request


            Clarification accuracy
            Whether missing data was handled without guessing


            Recovery success
            Whether a failed call was repaired correctly


            Stability by case
            Whether repeated runs agree


            Provider-error count
            How much of the planned test was actually observed
Enter fullscreen mode Exit fullscreen mode

Test validation and execution errors as conversations 

      Production llm function calling does not end when a call is
      emitted. The application may reject the arguments, the target system may reject
      the operation, or the result may reveal that the model needs another tool.




      Add a second suite that executes deterministic fake tools. A
      mock tool should return predictable results
      and errors without contacting an external service.
Enter fullscreen mode Exit fullscreen mode 
export function fakeLookupOrder(args: unknown) {
  const validation = validateCall(lookupOrderSchema, args);

  if (!validation.valid) {
    return {
      ok: false,
      error: {
        code: "INVALID_ARGUMENTS",
        message: "The tool arguments did not match the schema.",
        details: validation.errors
      }
    };
  }

  const { order_id } = args as { order_id: string };

  if (order_id === "ORD-000000") {
    return {
      ok: false,
      error: {
        code: "ORDER_NOT_FOUND",
        message: "No order matched that order ID.",
        retryable: false
      }
    };
  }

  return {
    ok: true,
    order: {
      order_id,
      status: "shipped"
    }
  };
}
Enter fullscreen mode Exit fullscreen mode

Use machine-readable error envelopes 

{
  "ok": false,
  "error": {
    "code": "INVALID_ARGUMENTS",
    "message": "weight_grams must be an integer between 1 and 50000.",
    "field": "weight_grams",
    "retryable": true
  }
}
Enter fullscreen mode Exit fullscreen mode 
      The model should not have to infer the error type from a stack trace. At the same
      time, do not expose internal exceptions, database details, or secrets in tool
      messages.
Enter fullscreen mode Exit fullscreen mode

Recovery scenarios 

  •     Repairable type error: return that
    weight_grams must be an integer. Expect one corrected call.

  •     Missing required value: return that
    postal_code is missing. Expect a user clarification, not an
    invented value.

  •     Not found: return ORDER_NOT_FOUND. Expect an
    honest explanation or a request for another identifier, not repeated identical
    calls.

  •     Transient failure: return a retryable error. Permit a bounded
    retry according to application policy.

  •     Non-retryable failure: return a permission or policy error.
    Expect the model to stop.

  Bound the conversation with maxToolRounds. A model that alternates
  between two invalid calls should not consume the entire request budget.

const MAX_TOOL_ROUNDS = 3;

for (let round = 0; round < MAX_TOOL_ROUNDS; round++) {
  const response = await adapter.continue(messages, tools);
  const calls = extractCalls(response);

  if (calls.length === 0) return finalize(response);

  const results = calls.map((call) => fakeToolRouter(call));
  messages.push(response.message);
  messages.push(...toProviderToolResults(calls, results));
}

return {
  kind: "stopped",
  reason: "max_tool_rounds_exceeded"
};
Enter fullscreen mode Exit fullscreen mode

Score recovery precisely 

      A recovery pass should require the expected terminal behavior, not merely a
      second response. Record:
Enter fullscreen mode Exit fullscreen mode

  • number of tool rounds;

  • whether the invalid call was repeated unchanged;

  • whether the corrected arguments validate;

  • whether a missing value was invented;

  • whether the model stopped on a non-retryable error;

  • whether the final user-facing statement agrees with the tool result.

Verify that the comparison is reproducible 

      Before interpreting model quality, verify the harness itself. The following
      checklist prevents many false conclusions.
Enter fullscreen mode Exit fullscreen mode

1. Save a run manifest 

{
  "runId": "2026-08-01T09-30-00Z",
  "datasetVersion": "1.0.0",
  "systemPromptSha256": "computed-at-runtime",
  "toolsSha256": "computed-at-runtime",
  "runnerCommit": "local-revision-or-archive-id",
  "runtime": {
    "node": "record-at-runtime",
    "platform": "record-at-runtime"
  },
  "settings": {
    "temperature": 0,
    "repeats": 5,
    "toolChoice": "auto",
    "timeoutMs": 30000,
    "concurrency": 1
  },
  "models": [
    {
      "provider": "provider-a",
      "requestedModel": "exact-model-id-used",
      "reportedModel": "capture-from-response-if-available"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode 
      Compute hashes from the exact bytes sent by the runner. This catches accidental
      fixture edits and adapter transformations.
Enter fullscreen mode Exit fullscreen mode

2. Snapshot adapter translations 

      For one representative case per provider, save a sanitized request fixture and
      test it locally. The snapshot should prove that:
Enter fullscreen mode Exit fullscreen mode

  • all three tools are present;

  • required fields remain required;

  • enums and numeric bounds survive translation;

  • the system instruction is in the intended message field;

  • the tool-choice mode is correct;

  • no adapter has injected examples or provider-only hints.

3. Unit-test the scorer with deliberate mistakes 

const scorerCases = [
  {
    name: "right tool and exact arguments",
    actual: call("lookup_order", { order_id: "ORD-104821" }),
    expectedScore: "pass"
  },
  {
    name: "right tool but wrong identifier",
    actual: call("lookup_order", { order_id: "ORD-104822" }),
    expectedScore: "semantic_failure"
  },
  {
    name: "wrong tool with plausible arguments",
    actual: call("search_orders", { email: "sam@example.test" }),
    expectedScore: "selection_failure"
  },
  {
    name: "unexpected extra property",
    actual: call("lookup_order", {
      order_id: "ORD-104821",
      email: "sam@example.test"
    }),
    expectedScore: "schema_failure"
  }
];
Enter fullscreen mode Exit fullscreen mode 
      If the scorer cannot distinguish these cases, model comparisons will not be
      trustworthy.
Enter fullscreen mode Exit fullscreen mode

4. Replay stored responses offline 

      Add a replay mode that reads sanitized raw responses, runs normalization again,
      and recomputes scores without contacting a provider. The replay should reproduce
      the original normalized records and summary.
Enter fullscreen mode Exit fullscreen mode 
npx tsx src/run.ts --config benchmark.config.json
npx tsx src/report.ts results/2026-08-01T09-30-00Z.jsonl
npx tsx src/replay.ts results/2026-08-01T09-30-00Z.jsonl
npx tsx src/report.ts results/replayed.jsonl
Enter fullscreen mode Exit fullscreen mode

5. Compare hashes, not formatted output 

      Canonicalize object key order before hashing normalized outcomes. Do not reorder
      arrays unless their order is semantically irrelevant. Tool-call order may matter
      when one call depends on another.
Enter fullscreen mode Exit fullscreen mode

6. Inspect failures manually 

      Metrics identify patterns; raw records explain them. Review at least one example
      from every failure category and every model. Check whether the problem belongs
      to the model, the provider protocol, your adapter, the validator, or an
      incorrectly specified fixture.



      A benchmark is reproducible when another authorized operator can use the same
      fixtures, adapter versions, settings, and scoring code to regenerate a comparable
      report. Reproducible does not mean that a hosted model must return byte-identical
      output forever.
Enter fullscreen mode Exit fullscreen mode

Failure cases worth testing explicitly

Wrong tool chosen from overlapping descriptions 

      Symptoms include using search_orders despite an exact order ID, or
      using lookup_order with an email address. Improve descriptions by
      stating both the positive condition and the boundary:
      “Use only when an exact public order ID is present.”




      Do not immediately add the same rule to the system prompt, every tool
      description, and the user message. First change one variable and rerun the same
      cases. Otherwise you will not know which intervention helped.
Enter fullscreen mode Exit fullscreen mode

Required arguments are guessed 

      A model may invent a postal code because the schema requires it. Include
      missing-information cases and score clarification explicitly. If tool use is
      forced, this test cannot measure natural clarification behavior.
Enter fullscreen mode Exit fullscreen mode

Numbers and identifiers change type 

      Postal codes, account numbers, and telephone numbers should generally be strings.
      Quantities used in calculations should be numbers or integers. The
      02139 case detects pipelines that accidentally turn a code into
      2139.
Enter fullscreen mode Exit fullscreen mode

Units are copied without conversion 

      If the schema expects grams and the prompt says kilograms, semantic validation
      must check the conversion. A structurally valid value of 1 is still
      wrong.
Enter fullscreen mode Exit fullscreen mode

Unsupported enum synonyms 

      A model may emit "fast" when the schema allows only
      "express". Decide whether the model must map synonyms, the
      application may repair them, or the user must clarify. Measure the raw and
      repaired outcomes separately.
Enter fullscreen mode Exit fullscreen mode

Extra arguments leak from the conversation 

      The model may carry an email address into lookup_order even though
      that tool does not accept it. Keep additionalProperties: false in
      the canonical schema and ensure the validator does not remove extras before
      scoring.
Enter fullscreen mode Exit fullscreen mode

Malformed serialized arguments 

      Some protocols represent arguments as serialized JSON. Truncated output, invalid
      escaping, or surrounding prose must become a parse failure, not an empty object.
      Preserve the raw string for diagnosis.
Enter fullscreen mode Exit fullscreen mode

Parallel calls appear unexpectedly 

      A model may request several functions in one response. Your adapter must retain
      every call and its identifier. For cases expecting one call, score additional
      calls as a policy violation even if the first call is correct.
Enter fullscreen mode Exit fullscreen mode

Identical invalid calls loop 

      Hash the tool name and canonical arguments at every round. If the same failed
      call repeats after the model receives a non-transient error, stop early and
      classify the outcome as an unrecovered loop.
Enter fullscreen mode Exit fullscreen mode

Text claims success before execution 

      A proposed call is not a completed action. Reject responses that tell the user
      an order was found, changed, or shipped before the application returns a
      successful tool result.
Enter fullscreen mode Exit fullscreen mode

Provider schema rejection is mistaken for model failure 

      If one llm api rejects a keyword, the model never saw the
      request. Track schema compatibility independently and report the transformed
      schema used for any fallback run.
Enter fullscreen mode Exit fullscreen mode

Retries bias the results 

      An SDK may automatically retry timeouts or rate limits. Record attempt counts
      when possible and apply the same retry policy across adapters. Model-level
      recovery after a tool error is different from transport-level retry.
Enter fullscreen mode Exit fullscreen mode

Turn benchmark findings into production policy 

      The most accurate model in a small fixture is not automatically the best
      production choice. Use the report to build explicit runtime controls.
Enter fullscreen mode Exit fullscreen mode

  • Validate every call on the server, regardless of benchmark performance.

  • Allowlist tool names instead of dynamically resolving arbitrary model text.

  • Separate read-only tools from tools with side effects.

  • Require confirmation or authorization for consequential operations.

  • Set timeouts, retry limits, and maximum tool rounds.

  • Use idempotency keys for retryable write operations.

  • Log sanitized call decisions and validation outcomes.

  • Keep a text fallback for provider outages or unsupported requests.

      For high-risk tools, use a two-stage design: the model proposes a typed intent,
  then deterministic application logic checks permissions, state, and business
  rules before execution. Schema validity is not authorization.

A practical release gate 

      Define thresholds from your own risk tolerance and baseline data rather than
      copying universal numbers. A release gate can require:
Enter fullscreen mode Exit fullscreen mode

  • no regression on high-risk abstention cases;

  • no invented values in missing-required tests;

  • no unresolved validation errors in accepted calls;

  • bounded recovery without repeated-call loops;

  • a documented explanation for every schema transformation;

  • successful replay of the stored evaluation records.

      Keep raw counts beside percentages. “Nine of ten” and “nine hundred of one
  thousand” may display the same percentage while providing very different
  evidence.

Limitations of this benchmark 

      This method measures behavior on a controlled fixture, not universal model
      quality. Results are conditional on the exact prompts, schemas, provider
      adapters, settings, model versions, language, and date of execution.
Enter fullscreen mode Exit fullscreen mode 
  •     Hosted models and provider routing can change even when your application code
    does not.

  •     Temperature zero does not guarantee identical responses.

  •     Synthetic cases do not reproduce the full distribution of production
    conversations.

  •     The portable schema subset may underuse provider-specific capabilities.

  •     Exact-match scoring can penalize valid alternatives unless fixtures define
    acceptable variation.

  •     Latency and token usage can be recorded, but fair cost comparison requires the
    current terms and accounting rules of each provider.

  •     English-only fixtures do not establish multilingual reliability.

  •     Fake tools evaluate recovery logic but cannot reproduce every external-system
    failure.

  Run two complementary suites: a portable suite for cross-provider comparison and
  a provider-optimized suite for the best production configuration on each
  platform. Label them clearly and never merge their scores.

Repeatable workflow

  • Write a canonical tool contract and freeze its version.

  • Create positive, confusable, abstention, argument, and clarification cases.

  • Implement thin adapters around each provider’s llm api.

  • Save sanitized raw requests and responses.

  • Normalize protocol syntax without repairing semantic errors.

  • Score request acceptance, tool choice, schema validity, and semantic accuracy separately.

  • Use deterministic fake tools to test recovery from structured errors.

  • Repeat cases and report stability with raw denominators.

  • Replay stored responses to verify the scorer and normalizer.

  • Promote findings into validation, authorization, retry, and release policies.

      The important outcome is not a permanent leaderboard. It is a versioned,
  inspectable process that shows exactly where llm function calling
  succeeds or breaks in your application—and lets you rerun the same test after a
  model, prompt, schema, SDK, or provider changes.

  Continue with the Agent Lab Journal guides, or review
  terminology in the AI engineering glossary.

© 2026 Agent Lab Journal

Original article: https://agentlabjournal.online/en/llm-function-calling-provider-comparison.html?utm_source=devto&utm_medium=referral&utm_campaign=agentlabjournal-en-global-all&utm_content=article&utm_term=llm-function-calling-provider-comparison

Top comments (0)