Contract Testing an OpenAI-Compatible API Before You Switch Providers

#ai #machinelearning #tutorial #llm

“OpenAI-compatible” is a claim about a URL path and a request body. It is very rarely a claim about the response, and the response is the part your code parses. The gap between those two things is what this suite is for.

What the suite proves

The migration you are contemplating looks cheap. You change a base URL and an API key, the SDK is the same, the request body is the same, and a smoke test comes back with a plausible sentence in it. That smoke test proved one path through one endpoint with one set of parameters. Your application uses more than that: it streams, it asks for tool calls, it counts tokens for billing, it retries on a specific status code, and it has a parser that indexes into fields it has never had to think about because they have always been there.

A contract test is the artefact that turns “it seemed to work” into a list. It exercises every request shape your code actually sends, and asserts on the structure of what comes back — never on its content. That distinction is the whole discipline and it is worth being explicit about, because it is the one people get wrong first: an assertion that the reply contains the word “Paris” is a model test with a contract test’s name on it. It will fail on a temperature change, on a system prompt edit, and on a silent model update, and each failure teaches the team to trust the suite less. An assertion that choices[0].message.role === "assistant" fails only when something is genuinely broken.

The suite also produces something a smoke test cannot: a written record of what you assumed. Six months later, when a parser throws on a response from a provider nobody has touched, the suite is the only place the assumption was ever written down.

The assertions worth running

Order them by what breaks loudest. A missing field that your code dereferences throws immediately and you find out in minutes. A field whose type changed, or whose meaning changed, is the dangerous class — it flows through your system and shows up as a billing discrepancy or a corrupted vector index weeks later.

The envelope. id, object, created, model, choices, and choices[].index. Assert types, not values — except object, which is a documented constant and worth pinning exactly. See the field-by-field treatment.
Token accounting. That usage exists at all, that its three integers are integers, and that prompt_tokens + completion_tokens === total_tokens. If you bill from this object, that invariant is not pedantry.
Finish reasons. That the value you get is one your switch statement handles, and that a truncated response is labelled length rather than stop. A provider that reports stop on truncation will silently hand your parser half a JSON document forever.
The streaming path, separately. Buffered and streamed responses are frequently produced by different code inside the same service, so a passing buffered test says nothing about the stream.
Tool calls, if you use them. Specifically that function.arguments is a JSON string and not an object, which is the compatibility detail reimplementations most often get wrong in the direction of being helpful.
Errors. A deliberately bad request, an invalid key, and an oversized prompt. You want the HTTP status and the error body shape, because your retry logic branches on them.

Where compatible endpoints actually diverge

These are the specific places implementations that all advertise compatibility stop agreeing. Each one is a real assertion you can write today.

Usage during streaming. OpenAI returns token counts in a streamed response only when the request sets stream_options with include_usage true, and delivers them in a final chunk whose choices array is empty. Other implementations attach usage to the last content chunk, to every chunk, or omit it entirely. Code that reads usage from the last chunk it saw will read zero on one of them.
Chunks with no choices. Azure OpenAI’s content filtering emits a first chunk carrying prompt_filter_results and an empty choices array, so chunk.choices[0] throws on the very first event. Microsoft documents the filtering behaviour in its content streaming guidance; the practical rule is that an empty choices array is legal and your loop must skip it rather than index into it.
Embedding encoding. The OpenAI Python SDK sends encoding_format as base64 by default even though the API documents float as the default, and an implementation that only handles float will hand back something your client decodes into nothing. This is documented as openai-python issue 1490.
Stream framing. Some endpoints send SSE comment lines to hold the connection open — OpenRouter documents sending : OPENROUTER PROCESSING for exactly that reason. A hand-written parser that splits on newlines and assumes every line begins data: will crash on a comment.
The terminating sentinel. Most send data: [DONE]. Some proxies close the connection without it. If your loop waits for the sentinel before resolving, one of those hangs until a timeout.
Parameter silence. An unsupported parameter can be rejected with a 400 or accepted and ignored. The second is far worse — a seed or a dimensions that is quietly dropped leaves you believing in a determinism or a vector size you do not have.

Each of these is the documented or reported behaviour at the time of writing. Treat the list as the set of questions to ask a candidate, not as a fixed table — the point of running the suite is that you find out rather than trust this page.

One suite, two base URLs

Write the tests once and parameterise the endpoint. The value of the exercise comes from running identical assertions against your incumbent and your candidate, because a failure on the candidate alone is a migration blocker while a failure on both is a bug in your test.

// contract/openai-compatible.test.ts  —  vitest
import { describe, it, expect } from "vitest";
import OpenAI from "openai";
import { z } from "zod";

const TARGETS = [
  { name: "incumbent", baseURL: process.env.INCUMBENT_URL!, key: process.env.INCUMBENT_KEY!, model: "gpt-4o-mini" },
  { name: "candidate", baseURL: process.env.CANDIDATE_URL!, key: process.env.CANDIDATE_KEY!, model: "llama-3.3-70b" },
];

const Completion = z.object({
  id: z.string().min(1),
  object: z.literal("chat.completion"),
  created: z.number().int().positive(),
  model: z.string().min(1),
  choices: z.array(z.object({
    index: z.number().int(),
    message: z.object({ role: z.literal("assistant"), content: z.string().nullable() }),
    finish_reason: z.enum(["stop", "length", "tool_calls", "content_filter", "function_call"]),
  })).min(1),
  usage: z.object({
    prompt_tokens: z.number().int().nonnegative(),
    completion_tokens: z.number().int().nonnegative(),
    total_tokens: z.number().int().nonnegative(),
  }),
});

describe.each(TARGETS)("$name", ({ baseURL, key, model }) => {
  const client = new OpenAI({ baseURL, apiKey: key });

  it("returns the documented completion envelope", async () => {
    const raw = await client.chat.completions.create({
      model,
      messages: [{ role: "user", content: "Reply with the single word: ok" }],
      max_tokens: 16,
      temperature: 0,
    });
    const parsed = Completion.parse(raw);
    expect(parsed.usage.prompt_tokens + parsed.usage.completion_tokens)
      .toBe(parsed.usage.total_tokens);
  });

  it("reports truncation as length, not stop", async () => {
    const res = await client.chat.completions.create({
      model,
      messages: [{ role: "user", content: "Count from 1 to 200, one number per line." }],
      max_tokens: 8,
    });
    expect(res.choices[0].finish_reason).toBe("length");
  });
});

The schema is deliberately not strict about unknown keys. Providers add fields — system_fingerprint, provider-specific reasoning blocks, native finish reasons — and an addition breaks nothing. A removal or a type change breaks everything, and that is what a non-strict schema still catches.

Turning the run into a decision

Enumerate the request shapes your production code actually sends. Grep for the SDK call sites rather than working from memory; the parameter somebody added for one feature eighteen months ago is the one that will not be supported.
Write the envelope and usage assertions first and get them green against your incumbent. Anything red here is a bug in your test, not in the candidate.
Add the streaming suite as a separate file that reads the raw event stream rather than the SDK’s assembled object, so that framing differences are visible rather than smoothed over.
Run the whole thing against the candidate and record every failure as one of three things: a blocker, a change you will make to your code, or a documented difference you accept.
Keep the suite. It stops being a migration tool and becomes the regression gate described in running it on a schedule against the live API.

Two endpoints that disagree about streaming framing and usage placement mean two code paths in your application, and the second one is the one nobody maintains. A gateway is the other way to resolve that: Multigrid presents one response shape across providers, which moves the divergence into a layer you can contract-test once instead of a branch you carry in every call site.