Contract Tests for Streaming Chunk Format

#ai #machinelearning #tutorial #llm

The client library is very good at making two differently-shaped event streams look identical. That is what it is for, and it is exactly why you cannot use it to test whether they are.

Why the SDK hides what you need to test

Iterate an OpenAI SDK stream and you get parsed chunk objects. The SDK has already discarded the comment lines, absorbed the terminating sentinel, skipped whatever it decided was noise, and in some clients assembled tool-call fragments for you. Every one of those is a place two providers can disagree, and every one of them has been successfully hidden before your test sees anything.

That is the right behaviour for application code. It is the wrong layer for a contract test, because the failure you are trying to catch is precisely the one that a newer SDK will absorb and an older one will not — or that your own hand-rolled parser in some other service does not absorb at all. So this suite talks to the endpoint with fetch and reads bytes.

Keep both. The SDK-level streaming test proves your application code works; this one proves the wire is what your application code assumes. They fail for different reasons and the difference is diagnostic: if the raw suite is red and the SDK suite is green, the client library is absorbing a divergence for you, and you have just learned that an SDK upgrade is a change to your provider compatibility. If both are red, the provider moved.

There is a second reason to read bytes, which applies even when you only ever use one SDK. Streaming code is frequently duplicated across services in a way buffered code is not — a browser reading EventSource, a worker reading with fetch, a mobile client with its own parser. Those consumers do not share the SDK, so the only description of the format they all depend on is the wire itself. A raw-level suite is the one place that description can be written down and checked.

The wire format

Streamed chat completions are Server-Sent Events. The properties worth asserting at this level, before any JSON is parsed:

Content type. The response header should be text/event-stream. A provider that returns application/json to a request with stream: true has ignored the parameter and buffered the whole thing, and this single header assertion catches it before you wonder why latency got worse.
Event framing. Events are separated by a blank line, and data lines begin with data: . A parser that splits on single newlines works until a payload arrives that it should have treated as one event.
Comment lines. A line beginning with a colon is an SSE comment and must be ignored. This is not hypothetical: OpenRouter documents sending : OPENROUTER PROCESSING to keep connections alive during long waits. A hand-written parser that assumes every non-blank line is data throws on the first one.
The sentinel. data: [DONE] conventionally terminates the stream. It is a literal, not JSON, so a parser that calls JSON.parse on every data payload fails on the last one. Assert both that it arrives and that it is the final event — and note that some proxies simply close the connection instead, so code that waits for it should also handle end-of-stream.

The chunk envelope

Each data payload other than the sentinel is a JSON object that mirrors the buffered completion with two differences: object is "chat.completion.chunk" rather than "chat.completion", and each choice carries delta instead of message.

The delta is where the divergence lives. The reference behaviour is that the first chunk for a choice carries delta.role set to "assistant" with empty or absent content, subsequent chunks carry only delta.contentfragments, and the final chunk for that choice carries an empty delta with a non-null finish_reason. Plenty of implementations send content in the very first chunk, or repeat the role on every chunk, or put finish_reason on the last content chunk rather than on a chunk of its own. None of those is fatal to a well-written consumer, and all of them break a consumer that assumed the reference shape — which is why you assert what you rely on rather than what the documentation shows.

Two envelope facts deserve their own assertions because they contradict what a naive parser assumes. First, choices can be an empty array on a legitimate chunk: Azure OpenAI’s content filtering emits a leading chunk carrying prompt_filter_results with no choices, and the final usage chunk on OpenAI does the same. Anything that writes chunk.choices[0] without checking length will throw on the first event against one provider and on the last against another. Second, usage in a stream is opt-in: OpenAI returns token counts only when the request sets stream_options with include_usage true, and delivers them in that final choice-less chunk. Assert both branches — that usage is absent without the flag and present with it — because a provider that ignores the flag silently leaves you with no token counts on your entire streaming traffic.

Sequence properties, not chunk properties

Some of what you need to know is not a property of any single chunk. Collect the whole sequence first, then assert on it as a list. These are the properties that matter and none of them touches the text.

Exactly one terminal chunk per choice index. Count the chunks with a non-null finish_reason for each index and assert it is one. Two means a provider is repeating the terminal state, which breaks any consumer that finalises on it.
Nothing after the terminal. No content arrives for a choice after its finish_reason, other than the usage chunk.
Stable identity. The id and model are the same across every chunk of one response. A gateway that re-generates an id per chunk breaks correlation in your logs.
Concatenation equals the buffered answer. Send the same request twice at temperature zero, once streamed and once not, and assert that joining the content deltas produces the same string as message.content. This is a metamorphic relation rather than an assertion on prose, so it survives model changes — and it is the single check that catches a provider dropping or duplicating a fragment. Determinism at temperature zero is approximate on most hosted models, so treat a failure as a prompt to look rather than a certainty, and see token reassembly for the stronger version of the same idea.

A raw stream harness

import { describe, it, expect } from "vitest";

type Event = { comment: string } | { data: string };

async function readEvents(res: Response): Promise<Event[]> {
  const reader = res.body!.getReader();
  const decoder = new TextDecoder();
  const events: Event[] = [];
  let buffer = "";

  for (;;) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });

    let sep: number;
    while ((sep = buffer.indexOf("\n\n")) !== -1) {
      const block = buffer.slice(0, sep);
      buffer = buffer.slice(sep + 2);
      for (const line of block.split("\n")) {
        if (line.startsWith(":")) events.push({ comment: line.slice(1).trim() });
        else if (line.startsWith("data:")) events.push({ data: line.slice(5).trim() });
      }
    }
  }
  return events;
}

describe("streaming chunk contract", () => {
  it("frames events correctly and terminates with the sentinel", async () => {
    const res = await fetch(process.env.TARGET_URL + "/v1/chat/completions", {
      method: "POST",
      headers: {
        "content-type": "application/json",
        authorization: "Bearer " + process.env.TARGET_KEY,
      },
      body: JSON.stringify({
        model: process.env.TARGET_MODEL,
        messages: [{ role: "user", content: "Count slowly from one to ten." }],
        max_tokens: 64,
        temperature: 0,
        stream: true,
        stream_options: { include_usage: true },
      }),
    });

    expect(res.headers.get("content-type")).toContain("text/event-stream");

    const events = await readEvents(res);
    const data = events.filter((e): e is { data: string } => "data" in e);
    expect(data.at(-1)!.data).toBe("[DONE]");

    const chunks = data.slice(0, -1).map((e) => JSON.parse(e.data));
    expect(chunks.length).toBeGreaterThan(1);

    // envelope
    for (const c of chunks) {
      expect(c.object).toBe("chat.completion.chunk");
      expect(c.id).toBe(chunks[0].id);
    }

    // exactly one terminal chunk for choice 0
    const terminal = chunks.filter(
      (c) => c.choices?.length && c.choices[0].index === 0 && c.choices[0].finish_reason !== null,
    );
    expect(terminal).toHaveLength(1);
    expect(["stop", "length"]).toContain(terminal[0].choices[0].finish_reason);

    // usage arrives, on a chunk with no choices
    const usageChunk = chunks.find((c) => c.usage);
    expect(usageChunk).toBeDefined();
    expect(usageChunk.choices).toHaveLength(0);
  });
});

The harness returns comments as events rather than discarding them, so a second test can assert that your production parser tolerates one — feed it a recorded stream with a comment line spliced in and check it still produces the right text. That is the assertion that would have caught the failure before it reached anybody.