Testing That Partial JSON Mid-Stream Doesn't Crash the Parser

#ai #machinelearning #llm #tutorial

Two separate things are called “partial JSON in a stream” and only one of them needs a special parser. Confusing them produces either a crash on every second chunk or an elaborate incremental parser solving a problem that a five-line buffer already solved.

Two different parsers, two different bugs

The framing layer. An SSE event is delivered as bytes, and a read can end anywhere. The JSON object on a data: line is complete, but the line itself may be split. The answer here is not a partial JSON parser: it is a buffer that only hands a frame onward once it has seen the blank line. Calling JSON.parse on a raw socket chunk is the bug, and it is fixed by framing, which has its own page — testing reassembly of streamed tokens. Everything below assumes framing already works.

The payload layer. Some values genuinely arrive in fragments across many complete frames. A tool call’s arguments do this: Anthropic documents that input_json_delta events carry partial JSON strings, whereas the final tool_use.input is always an object; the OpenAI-shaped equivalent accumulates into choices[0].delta.tool_calls[i].function.arguments. A structured response requested through a JSON schema arrives the same way, as text that is only valid JSON once complete.

For the payload layer you have two legitimate strategies. Buffer and parse once at the end, on content_block_stop or on the chunk carrying finish_reason: "tool_calls" — simple, correct, and the right default. Or parse each prefix with a partial-tolerant parser so a UI can render fields as they land. The second is what needs testing, and it is what this page is about.

How the fragments actually arrive

Read a real trace before designing the test, because the fragment boundaries are not where intuition puts them. A documented example of a tool call streaming its input shows deltas such as {"location":, then "San, then Francisc, then o,, then CA"}. Note what that means: a prefix can end in the middle of a string value, in the middle of a word, between a key and its colon, or after an opening brace with nothing else. Any of those is a valid input to your parser and none of them is valid JSON.

Note also that the fragments are not tokens and not fixed-size. Do not write a test that splits on a fixed length and call it representative; split at every offset, which is both stronger and easier.

The property to assert

Asserting the exact object at each prefix is unmaintainable — it hard-codes dozens of intermediate states that carry no meaning. Assert a relation between consecutive states instead. A correct partial parser satisfies three things:

Totality. Every prefix either parses or is reported as not-yet-parseable. It never throws.
Monotonic growth. Each parse extends the previous one. Keys never disappear; a completed scalar never changes; the one legitimate change is a string value growing by appended characters, because it is still arriving.
Convergence. The parse of the full document equals what JSON.parse gives you.

Monotonicity is the assertion that matters, because it is exactly the property a UI depends on. If a field can change from San Franc to null and back, every consumer downstream has to defend against it.

function extendsPrevious(next: unknown, prev: unknown): boolean {
  if (typeof prev === "string") {
    return typeof next === "string" && next.startsWith(prev);
  }
  if (Array.isArray(prev)) {
    if (!Array.isArray(next) || next.length < prev.length) return false;
    return prev.every((v, i) => extendsPrevious(next[i], v));
  }
  if (prev && typeof prev === "object") {
    if (!next || typeof next !== "object" || Array.isArray(next)) return false;
    return Object.entries(prev).every(([k, v]) =>
      k in (next as object) && extendsPrevious((next as any)[k], v),
    );
  }
  return Object.is(next, prev);          // numbers, booleans, null
}

Numbers are the exception worth knowing about. A prefix ending after 1 in the value 12.5 can parse as the number 1, and a later prefix will report 12.5 — which violates the rule above. Well-behaved partial parsers withhold an incomplete number rather than emitting a truncated one; if yours does not, exclude numeric leaves from the monotonicity check and say so in a comment, rather than weakening the whole relation.

Prefixes that end inside an escape

These are the cases that break hand-rolled parsers, and they are why you should reach for an existing one. Include all of them in the fixture:

A prefix ending on a lone backslash inside a string. Emitting the backslash as a literal character is wrong; it is the start of an escape.
A prefix ending part-way through a \u00e9 escape. Four hex digits arrive one at a time and none of the intermediate states is a character.
A prefix ending between the two halves of a surrogate pair, which is how an emoji is written in escaped JSON.
A prefix ending immediately after a comma or a colon, where the next value has not started.
Deeply nested closes: a prefix that has opened three objects and closed none.

Reputable implementations exist for this and are a better starting point than your own. In JavaScript there are dedicated partial-JSON packages on npm; in Python, Pydantic documents partial JSON parsing directly, and Anthropic’s streaming documentation points at Pydantic’s partial JSON parsing as the recommended approach for accumulating input_json_delta fragments. Whichever you pick, the test below is what tells you it behaves the way your UI assumes.

One design decision the test forces you to make explicit: what a partial parser should do with a key whose value has not started. Some implementations omit the key entirely until the value exists; others emit it with a null or an empty placeholder. Both are defensible and they are not interchangeable, because the second violates the monotonicity relation the moment the real value arrives. Decide which one you want, encode it in the test, and your rendering code can rely on it — which is the actual deliverable here, rather than the parser itself.

The test

import { expect, test } from "vitest";

const TARGET = {
  location: "San Francisco, CA",
  units: "celsius",
  days: 3,
  flags: { hourly: true, alerts: ["wind\u00a0advisory", "\ud83d\udd25 heat"] },
};
const DOCUMENT = JSON.stringify(TARGET);

test("no prefix throws, and each parse extends the last", () => {
  let previous: unknown;
  for (let i = 1; i <= DOCUMENT.length; i++) {
    const prefix = DOCUMENT.slice(0, i);
    let value: unknown;
    expect(() => { value = parsePartial(prefix); }, "prefix of length " + i).not.toThrow();
    if (value === undefined) continue;              // not yet parseable is fine
    if (previous !== undefined) {
      expect(extendsPrevious(value, previous), "at length " + i + ": " + prefix).toBe(true);
    }
    previous = value;
  }
  expect(previous).toEqual(TARGET);
});

test("the accumulator handles fragments as the provider sends them", () => {
  const fragments = ['{"location":', ' "San', ' Francisc', 'o,', ' CA"}'];
  let buffer = "";
  const states: unknown[] = [];
  for (const fragment of fragments) {
    buffer += fragment;
    const value = parsePartial(buffer);
    if (value !== undefined) states.push(value);
  }
  expect(states.at(-1)).toEqual({ location: "San Francisco, CA" });
  expect(JSON.parse(buffer)).toEqual({ location: "San Francisco, CA" });
});

The second test exists because the first, iterating every prefix, is a superset that can obscure what the real fragment boundaries look like. Keeping a case with literal provider-shaped fragments documents the format for the next person to read the file, and it fails in a way that points straight at the accumulator rather than at the parser.

One last assertion worth adding: the accumulated buffer, once complete, must equal the document the provider intended — so also assert JSON.parse(buffer) succeeds. A partial parser that is generous enough to accept anything can mask an accumulator that dropped a fragment, and comparing against the strict parser at the end catches exactly that.

DEV Community