Contract Tests for the /v1/chat/completions Response Shape

#ai #machinelearning #tutorial #llm

The reply is non-deterministic. The envelope around it is not, and almost everything your code touches lives in the envelope. That is what makes this the one part of an LLM response you can assert on hard.

The envelope, field by field

A buffered chat completion is a small object with a fixed skeleton. Walking it in order, with what each field is actually for:

id — an opaque request identifier, conventionally prefixed chatcmpl- by OpenAI but not by everyone. Assert it is a non-empty string; do not assert the prefix unless you have a reason to depend on it, because you will be pinning a cosmetic detail of one vendor.
object — the literal string "chat.completion". This is a discriminator, it is documented as a constant, and it is the one string field worth pinning exactly. If it ever becomes something else, you want a failure.
created — a Unix timestamp in seconds. Assert it is a positive integer and, if you want a real check, that it is within a few minutes of now. A provider returning milliseconds here is a genuine compatibility bug and this assertion is how you find it.
model — the model that actually served the request, which is frequently not the string you sent. Alias resolution and version pinning both surface here. Assert it is a non-empty string; if you need a specific version, assert a prefix rather than equality.
choices — an array. It has one element unless you sent n greater than one, and each element carries index, message and finish_reason. Assert choices[i].index === i; array order is not guaranteed to be index order by anything except habit.
message.role — always "assistant" on a completion.
message.content — a string or null. Null is correct and expected when the model returned tool calls instead of prose, and a type that declares this non-nullable is a runtime crash waiting for the first tool call.
usage — three integers, covered below.

How strict to be about each field

The instinct with a schema is to make it strict and reject anything unexpected. That instinct is wrong here, and understanding why is what keeps the suite alive past its first month.

Providers add fields. system_fingerprint appeared as an addition, reasoning content blocks appeared as additions, cache-hit counters and native finish reasons are additions today. Every one of those arrives without warning and none of them breaks a consumer that ignores unknown keys. If your schema rejects unknown keys, your scheduled contract run goes red for a change that harmed nothing, and after the second false alarm somebody adds an exception, and after the third somebody stops reading the alert.

So the rule is asymmetric: be strict about the absence of what you need and the type of what you read, and permissive about additions. Pin exact values only for documented constants (object) and closed enums you branch on (finish_reason, message.role). Everything else gets a type check and a non-empty check. If you specifically want to know about additions — and it is reasonable to want that, since a new field is often the first sign of a capability you could use — run a second, non-blocking check that diffs the key set against a recorded baseline and reports rather than fails.

finish_reason is the field that bites

Of everything in the envelope, this is the field whose mishandling costs the most, because getting it wrong produces corrupted data rather than an error. The documented values on a buffered completion are stop, length, tool_calls, content_filter, and the deprecated function_call. During streaming it is null on every chunk except the last one for that choice.

The failure that matters is truncation reported as completion. You asked for JSON, the model hit max_tokens halfway through, and the provider labelled it stop. Your parser now receives a half-written object. If you check finish_reason before parsing, this is a clean error; if you do not, it is a partial JSON document that either throws somewhere unhelpful or, worse, parses into something plausible and wrong.

Test it directly by forcing it: send a prompt that cannot be answered within a tiny max_tokens and assert the value is exactly length. Then assert the negative case — a short prompt with a generous cap returns stop. Two assertions, and between them they pin the one behaviour that silently corrupts downstream data.

The enum above is the set OpenAI documents at the time of writing. Providers do add values — OpenRouter documents emitting error as a finish reason when a stream fails mid-response — so treat an unrecognised value as a case your code handles rather than a case that crashes it, and let the contract test tell you when a new one appears.

Usage, and the invariant worth asserting

usage carries prompt_tokens, completion_tokens and total_tokens. If any part of your system bills, budgets or rate-limits from these numbers, they are not metadata, they are financial records, and they deserve an invariant rather than a type check.

The invariant is that the first two sum to the third. It sounds too obvious to test until you consider what breaks it: a provider that counts reasoning tokens in the total but not in either component, a gateway that recomputes one field and forwards the others, a cached prompt whose discounted tokens are reported inconsistently. Each of those is a real shape and each one shows up as an arithmetic failure long before it shows up as a bill you can explain. Assert the sum, and assert that prompt_tokens is greater than zero — a zero there means the provider is not counting your input, which will be reconciled against you later.

The schema in one place

Define it once, import it into every test that touches a completion, and reuse it in production if your language lets you. A schema that lives only in the test suite is a second description of the same thing, and two descriptions drift.

# contract/test_chat_completions.py  —  pytest + pydantic
from typing import Literal, Optional
import os, time, pytest
from pydantic import BaseModel, Field
from openai import OpenAI

class Message(BaseModel):
    role: Literal["assistant"]
    content: Optional[str] = None

class Choice(BaseModel):
    index: int
    message: Message
    finish_reason: Literal["stop", "length", "tool_calls", "content_filter", "function_call"]

class Usage(BaseModel):
    prompt_tokens: int = Field(ge=0)
    completion_tokens: int = Field(ge=0)
    total_tokens: int = Field(ge=0)

class Completion(BaseModel):
    id: str = Field(min_length=1)
    object: Literal["chat.completion"]
    created: int
    model: str = Field(min_length=1)
    choices: list[Choice] = Field(min_length=1)
    usage: Usage
    # no "extra=forbid": additions are not breakages

client = OpenAI(base_url=os.environ["TARGET_URL"], api_key=os.environ["TARGET_KEY"])
MODEL = os.environ["TARGET_MODEL"]

def test_envelope_and_usage_invariant():
    raw = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": "Reply with the single word: ok"}],
        max_tokens=16, temperature=0,
    )
    c = Completion.model_validate(raw.model_dump())
    assert c.choices[0].index == 0
    assert abs(c.created - int(time.time())) < 600
    assert c.usage.prompt_tokens > 0
    assert c.usage.prompt_tokens + c.usage.completion_tokens == c.usage.total_tokens

def test_truncation_is_length():
    raw = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": "Count from 1 to 200, one per line."}],
        max_tokens=8,
    )
    assert raw.choices[0].finish_reason == "length"

Nothing in that file asserts on what the model said, which is why it will still pass after a model upgrade and still fail when the contract actually breaks.