Contract Tests for Function-Calling Schema Compliance

#ai #machinelearning #tutorial #llm

Whether the model calls your tool is a model-behaviour question and it will fail your suite on a good day. Whether the tool call it returns has the documented shape is a contract question, and that one is worth a hard assertion.

Two questions, one of which is not a contract

Almost every tool-calling test people write begins “given the prompt ‘what is the weather in Paris’, assert the model calls get_weather”. That test is measuring model behaviour. It is sensitive to the tool description, the system prompt, the temperature, and to any model update the provider ships; it is worth having, but it belongs with your evaluations, alongside diagnosing a tool that does not fire, not in a suite whose job is to be green unless something is broken.

The contract question is different and it is conditional: if the model returns a tool call, is it shaped the way my parser assumes? That holds regardless of temperature, prompt or model version. You can force the condition rather than hoping for it — set tool_choice to require the specific function — which turns a probabilistic test into a deterministic one and, as a bonus, tests that tool_choice is honoured at all. A provider that accepts a forced tool choice and returns prose has a compatibility bug worth a red build.

Everything else here follows from that split. Assert on names, types, parse-ability and validity; never on the argument values the model chose, which are as non-deterministic as prose.

The buffered shape

On a non-streamed completion, a tool call arrives inside the message rather than the content. The parts, and what to assert about each:

choices[0].finish_reason is "tool_calls". Assert this before anything else — it is the branch signal, and a provider that returns tool calls with finish_reason of stop will route your code down the prose path.
message.content is null, or a string if the model emitted commentary alongside. Assert your parser handles null, because that is the normal case and the one an over-tight type crashes on.
message.tool_calls is a non-empty array.
tool_calls[].id is a non-empty string. It is opaque and you must echo it back on the corresponding tool result message; a parser that generates its own id instead breaks the conversation on the next turn.
tool_calls[].type is "function".
tool_calls[].function.name is a member of the set you declared. Assert set membership, not a specific name — that keeps the test about the contract rather than about the choice.
tool_calls[].function.arguments is a string. See below.

The set-membership assertion is quietly the most valuable one in the list, because a hallucinated function name is a real failure mode and it produces a confusing error deep in your dispatcher. Asserting that the returned name exists in the declared tool array turns that into one clear line.

Arguments are a string, and that matters

function.arguments is a JSON-encoded string, not a JSON object. This surprises people every time and it is the detail that OpenAI-compatible reimplementations most often get wrong, usually by being helpful: they parse it for you and return an object, which breaks every client that calls JSON.parse on it.

So the assertion is in two parts and both are necessary. First, the field is of type string — a bare type check, and the one that catches a helpful provider. Second, parsing it yields an object that validates against the same JSON Schema you sent in the tool definition. That second half is the one that catches the more interesting failure: the model producing arguments that do not satisfy the schema it was given, including a missing required property or a string where the schema asked for a number.

Validate against the schema object itself rather than a hand-copied duplicate. If your tool definitions are generated from types — a Zod schema converted to JSON Schema, a pydantic model — then the test can import the same source and you have one description rather than two. Two descriptions of one schema is exactly the drift that produces a green test and a broken dispatcher.

Some providers offer a strict mode that constrains generation to the supplied schema. Where it exists it makes the second assertion nearly always pass, which is a reason to keep the test rather than to drop it — the day it starts failing is the day strict mode stopped being applied. The general treatment is in JSON mode versus structured outputs.

The streamed form

Streaming a tool call is where the shape genuinely gets harder, because the call arrives in fragments across many chunks and reassembly is the caller’s job. The rules that make reassembly possible are the contract, and each one is an assertion:

delta.tool_calls[].index is the key. Not the array position, not the id. With parallel tool calls, fragments for different calls interleave, and index is the only field that tells you which accumulator a fragment belongs to. Code that appends to tool_calls[0] because that is where the fragment appeared in the array silently merges two calls into one.
id and function.name arrive once. Conventionally on the first fragment for that index, and are absent from later ones. A parser that reads the name from the last fragment gets undefined.
function.arguments arrives in pieces. Each fragment is a substring, and the substrings are not individually valid JSON — you will see a lone "{", a partial key, a dangling comma. Assert that concatenating every fragment for one index yields parseable JSON, and assert nothing whatsoever about any single fragment.
The terminal finish_reason is "tool_calls". Same signal as the buffered case, on the last chunk for the choice.

The reassembly logic that follows from those rules is the piece worth testing hardest, because it is yours and it is fiddly. Once you have a recorded fragment sequence you can test it without the network at all — which is the separation described in testing without the model. The contract test proves the fragments still arrive in that shape; the unit test proves your accumulator handles them.

The suite

import { describe, it, expect } from "vitest";
import OpenAI from "openai";
import Ajv from "ajv";

const client = new OpenAI({ baseURL: process.env.TARGET_URL, apiKey: process.env.TARGET_KEY! });
const MODEL = process.env.TARGET_MODEL!;

const parametersSchema = {
  type: "object",
  properties: {
    city: { type: "string" },
    unit: { type: "string", enum: ["c", "f"] },
  },
  required: ["city"],
  additionalProperties: false,
} as const;

const tools = [{
  type: "function" as const,
  function: { name: "get_weather", description: "Current weather for a city", parameters: parametersSchema },
}];

const validate = new Ajv().compile(parametersSchema);
const declared = new Set(tools.map((t) => t.function.name));

describe("tool call contract", () => {
  it("returns a documented tool_call when one is forced", async () => {
    const res = await client.chat.completions.create({
      model: MODEL,
      messages: [{ role: "user", content: "Weather in Paris, metric." }],
      tools,
      tool_choice: { type: "function", function: { name: "get_weather" } },
      max_tokens: 128,
    });

    const choice = res.choices[0];
    expect(choice.finish_reason).toBe("tool_calls");

    const calls = choice.message.tool_calls!;
    expect(calls.length).toBeGreaterThan(0);

    for (const call of calls) {
      expect(typeof call.id).toBe("string");
      expect(call.id.length).toBeGreaterThan(0);
      expect(call.type).toBe("function");
      expect(declared.has(call.function.name)).toBe(true);
      // the field is a STRING, not an object — this line is the compatibility check
      expect(typeof call.function.arguments).toBe("string");
      const args = JSON.parse(call.function.arguments);
      expect(validate(args), JSON.stringify(validate.errors)).toBe(true);
    }
  });

  it("reassembles streamed fragments by index", async () => {
    const stream = await client.chat.completions.create({
      model: MODEL,
      messages: [{ role: "user", content: "Weather in Paris and in Oslo." }],
      tools,
      tool_choice: "required",
      stream: true,
      max_tokens: 256,
    });

    const acc = new Map<number, { name?: string; id?: string; args: string }>();
    let finish: string | null = null;

    for await (const chunk of stream) {
      const choice = chunk.choices[0];
      if (!choice) continue;                       // usage / filter chunks carry none
      if (choice.finish_reason) finish = choice.finish_reason;
      for (const frag of choice.delta.tool_calls ?? []) {
        const slot = acc.get(frag.index) ?? { args: "" };
        if (frag.id) slot.id = frag.id;
        if (frag.function?.name) slot.name = frag.function.name;
        slot.args += frag.function?.arguments ?? "";
        acc.set(frag.index, slot);
      }
    }

    expect(finish).toBe("tool_calls");
    for (const [, slot] of acc) {
      expect(declared.has(slot.name!)).toBe(true);
      expect(typeof slot.id).toBe("string");
      expect(validate(JSON.parse(slot.args))).toBe(true);   // whole, not fragments
    }
  });
});

Neither test asserts that the model chose Paris, or that it picked celsius, or that it made two calls rather than one. Those are the model’s decisions. What is asserted is that whatever it decided arrived in a shape your dispatcher can execute.