I Streamed Structured Output Into a Live Panel. A Field Rendered Before I Knew It Was Wrong

The panel was for our support team. An agent pastes a customer's message in, and four fields fill in next to it: category, priority, a one-line summary, and which team it should go to. All four come from one model call against one schema. Nothing exotic - a single generate() call would have done it in about three lines.

The problem was the three seconds that call took. Agents kept re-clicking the button mid-request, assuming it had hung, because the panel just sat there blank until the whole object came back at once. So the ask became: show each field the moment it's ready, not all four at once at the end.

First pass: parse whatever text has arrived so far

The obvious move was to switch to the streaming call, take the raw text deltas as they arrived, and after every delta try JSON.parse() on whatever had accumulated so far, wrapped in a try/catch. If it parsed, grab whatever fields existed and render them.

It almost never parsed. An object with unclosed braces throws every time, right up until the very last delta closes the outer }. So in practice this did nothing for the entire three seconds and then dumped all four fields on screen at once anyway - the same experience as not streaming, with extra code.

Second pass: track bracket depth by hand

Next I tried tracking whether a field's value had structurally closed by counting {/} and quote state as deltas came in, and slicing out that one field's substring the moment its depth returned to zero. That got fields appearing one at a time, which was the actual goal.

It also broke in two ways I didn't expect. The assignee field wasn't a plain string, it was { team, reason }, and nested braces threw my depth counter off by exactly the amount you'd guess. And separately - this is the one that actually worried me - "structurally closed" isn't the same as "correct." The model once returned "priority": "urgent-ish" instead of one of the four enum values. My counter saw the closing quote, decided the field was done, and rendered an orange "urgent-ish" badge on screen. A second later, when the full object finally failed validation and the whole thing retried, the badge silently flipped to something else. From the agent's side, a wrong priority had flashed on their ticket for a second before quietly changing. That's a bad thing to ship to people making triage calls under time pressure.

What was already there

I went looking for how generateStream() itself decides when a field is "done," assuming I'd have to patch around it, and found it already draws the line where I actually needed it drawn: a field only becomes visible once it's both structurally closed and has passed its own piece of the schema.

import { generateStream, openai } from "@aviasole/shapecraft";
import { z } from "zod";

const TicketSchema = z.object({
  category: z.enum(["billing", "bug", "how-to", "account"]),
  priority: z.enum(["low", "normal", "urgent"]),
  summary: z.string(),
  assignee: z.object({ team: z.string(), reason: z.string() }),
});

const stream = generateStream(openai({ model: "gpt-4o-mini" }), TicketSchema, rawMessage);

for await (const event of stream.events) {
  if (event.type === "partial") {
    renderFields(event.value); // only ever the fields that have already passed validation
  }
  if (event.type === "attempt-failed") {
    resetPanel(); // a field failed its own check mid-stream; the next attempt starts clean
  }
}

const { data } = await stream.result;

The partial event only fires after a field is checked against its own sub-schema, not just parsed. So a priority of "urgent-ish" never reaches renderFields() at all - the moment that field fails, the attempt is marked failed and abandoned mid-stream, without waiting to finish consuming the rest of that doomed response, and a fresh attempt starts. The panel never had a badge to flip back from, because it never rendered the wrong one in the first place.

The nested assignee object turned out to be handled too, just at a coarser grain than I expected: it's validated and emitted as one field once its own { team, reason } closes, not field-by-field inside it. That was finer-grained than my hand-rolled version actually needed to be.

Where it landed

Fields now start appearing within a few hundred milliseconds of the first one closing instead of all landing at once after three seconds, so the panel stopped looking hung. And the flash-then-correct badge, which agents had started asking about roughly once a shift under my bracket-counting version, hasn't come back - there's nothing left to flash, because nothing renders until it's already right.