Rendering an LLM's structured output while it streams, in Dart

#dart #ai #llm #flutter

Ask a model for JSON and it comes back one token at a time. If you want to show
the result as it arrives, and not stare at a spinner until the last token lands,
you have to parse a buffer that is almost never valid JSON yet: a string is
half-written, an object is still open, a comma dangles. jsonDecode throws on
all of those until the very end.

I wrote stream_struct because I kept writing that tolerant parser by hand, and
getting the edge cases wrong. It is the parser, done once and tested, plus the
glue for the common providers.

dart pub add stream_struct

Parse the buffer you have so far

parsePartialJson decodes a truncated buffer into the value it represents right
now. It closes an open string value and any open array or object, and drops a
dangling key, colon, or comma.

import 'package:stream_struct/stream_struct.dart';

parsePartialJson('{"title": "The quick bro');   // {title: The quick bro}
parsePartialJson('{"a": 1, "tags": ["x"');       // {a: 1, tags: [x]}
parsePartialJson('{"a": 1, "colo');              // {a: 1}   (partial key dropped)

It returns null while nothing is decodable yet. A caller treats null as "no
update this frame" and keeps the previous value; the next token resolves it, so a
half-written value never flashes a wrong result on screen.

Stream the object as it grows

streamPartialJson accumulates a delta stream and emits the value after each
token, skipping frames that do not parse or did not change:

await for (final partial in streamPartialJson(modelDeltas)) {
  setState(() => _draft = partial as Map<String, dynamic>);
}

How much does that actually buy you? I replayed a recorded Anthropic stream
carrying a 233-character JSON review object, split across the wire the way a
model emits it. The recording holds 22 content deltas, and streamPartialJson
turns them into 19 complete, decodable objects along the way.

So a UI paints 19 times instead of waiting and painting once, and every one of
those 19 is a real map rather than a half-parsed fragment you have to guard
against. That is the whole difference between a screen that sits empty for the
length of a generation and one that fills in.

The providers stream different shapes, so the adapters are built in. Decode each
Server-Sent Event into a Map, then pick the extractor:

streamPartialJsonFrom(chunks, openAiDelta);     // choices[0].delta.content
streamPartialJsonFrom(chunks, anthropicDelta);  // delta.text / delta.partial_json
streamPartialJsonFrom(chunks, geminiDelta);     // candidates[0].content.parts[0].text

Type it

streamPartial<T> maps each growing object through a builder you write to
tolerate a half-filled map, so you get a typed value on every step:

final titles = streamPartial<String>(
  modelDeltas,
  (m) => (m['title'] as String?) ?? '',
);

What it handles

The tolerant parse is the part that is easy to get wrong, so it is the part that
is tested:

open string values are kept and closed, so partial text shows as it types
open objects and arrays are closed to any depth
dangling keys, colons, and commas are dropped
braces and quotes inside strings, and escaped quotes, do not confuse it
a valid partial number is kept; an unresolved literal like tr skips that frame rather than guessing true

A test streams a full object one character at a time and asserts every prefix
lands on a prefix-consistent value, ending exactly on the final object.

What it is not

It does the parsing, not the model. It never makes a request; you bring the
stream. And a frame it cannot parse yet returns null instead of a guess, which
keeps a wrong intermediate value off the screen.