Last quarter we shipped a contract extraction feature that streamed its output field by field into a review UI, so a paralegal could start reading the extracted renewal date and counterparty name before the model finished the whole document. Nice idea. Faster perceived latency, better demo, the kind of thing that gets a thumbs up in a design review.
Then a reviewer flagged a contract where the counterparty name in the UI read "Meridian Hold" and the final, settled value was "Meridian Holdings Group LLC." Nobody had touched anything. The UI had simply displayed the string as it existed at a moment mid-stream, before the model had finished writing it, and the user had already glanced at it, nodded, and moved to the next tab. By the time the full value arrived, the case for that field was already closed in the reviewer's head.
That's the part that stings about streaming structured output. It doesn't usually fail loud. It fails by being plausible at exactly the wrong instant.
The naive approach is to accumulate the streamed text and try json.loads on the buffer every time a new chunk lands, catching the exception when it's not valid yet. That works, technically, right up until it doesn't, and the failure mode is worse than a crash: it's a false negative that becomes a false positive.
Concretely, here's what breaks:
1. Strings that parse as valid but aren't finished. json.loads doesn't fail on {"counterparty": "Meridian Hold"}. That's completely valid JSON. It's also not the value you want. A string field only becomes trustworthy at the character where its closing quote lands, and nothing before that tells you where that is, because the model doesn't announce "I'm two tokens from done."
2. Arrays that look closed but aren't. With tool-calling providers that stream token-by-token, we've seen an array of extracted clauses close its bracket, then reopen because the provider's underlying generation retried a truncated chunk server-side (this is provider-side behavior we can't fully control or always observe, and it showed up more under load, though I'd want a longer sample before I called that a hard rule rather than a pattern we noticed on high-traffic days). If your parser saw the first closing bracket and moved on, you've committed to a value that got silently superseded.
3. Parallel tool calls interleaving. When a model issues more than one tool call in the same turn, providers don't always guarantee the chunks for call A finish before chunks for call B start arriving. We had a case where two extract_clause calls were in flight and a naive buffer-per-response (rather than buffer-per-call) approach spliced fragments from both into one JSON blob that parsed successfully and meant nothing.
None of these are edge cases in the sense of being rare. In our sample of roughly 400 flagged extraction sessions over about six weeks (anecdotal, drawn from our own error queue, not an industry number), something in this family, a field displayed before it was complete, showed up in just under 3% of streamed sessions. Low frequency, high cost, because the ones that go wrong are the ones a human trusted.
What "schema-aware" actually means here
The fix isn't a smarter JSON parser. Full-document partial-JSON parsers exist and are useful for a different problem (rendering a tree view of an in-progress object), but they answer the wrong question. The question isn't "can I parse this yet." It's "is this specific field's value done being written."
That reframes the problem as a cursor per field, not a parser over the whole buffer. You track, for each key you care about, whether you've seen its terminating character. For strings, that's an unescaped closing quote. You do not surface the field to any downstream consumer, UI or otherwise, until that condition is met. Everything before that point stays internal state, not output.
Here's a minimal version of that, enough to show the shape (not a production parser, we handle nested objects and arrays with a similar but longer state machine):
from typing import Optional
class FieldExtractor:
"""Extracts only fields whose values are structurally complete
from a growing buffer of streamed JSON text. Never calls
json.loads on the partial buffer.
A field counts as 'safe to surface' only once we've seen the
terminating character for its type: here, a closing quote for
strings (not preceded by an odd number of backslashes).
"""
def __init__(self, expected_keys: set):
self.expected_keys = expected_keys
self.raw = ""
self.emitted = {}
def feed(self, chunk: str) -> dict:
self.raw += chunk
newly_closed = {}
for key in self.expected_keys:
if key in self.emitted:
continue
value = self._extract_closed_string_value(key)
if value is not None:
self.emitted[key] = value
newly_closed[key] = value
return newly_closed
def _extract_closed_string_value(self, key: str) -> Optional[str]:
marker = f'"{key}"'
start = self.raw.find(marker)
if start == -1:
return None
after_colon = self.raw.find(":", start + len(marker))
if after_colon == -1:
return None
i = after_colon + 1
while i < len(self.raw) and self.raw[i] in " \t\n":
i += 1
if i >= len(self.raw) or self.raw[i] != '"':
return None # not a string field, or value hasn't started
j = i + 1
while j < len(self.raw):
if self.raw[j] == '"' and self.raw[j - 1] != "\\":
return self.raw[i + 1:j] # closing quote found, value is safe
j += 1
return None # still being written
if __name__ == "__main__":
extractor = FieldExtractor(expected_keys={"vendor_name", "renewal_date"})
stream_chunks = [
'{"vendor_name": "Acme Ind',
'ustries, Inc.", "renewal_date": "2027-0',
'3-01"}',
]
for chunk in stream_chunks:
closed = extractor.feed(chunk)
for k, v in closed.items():
print(f"safe to show: {k} = {v!r}")
Run it and the first chunk produces nothing (the vendor name string hasn't closed), the second chunk produces vendor_name, and the third produces renewal_date. Nothing gets displayed until its own value is structurally complete, independent of whatever else is still being written elsewhere in the object.
The version we actually run in production extends this with a stack for nested objects and arrays (so a clause list only emits an item once that item's closing brace is seen, not when the array itself closes), and a separate buffer keyed by tool-call ID so parallel calls can't splice into each other. Pydantic still validates the finished object at the end, same as before. This layer sits earlier and answers a narrower question: not "is this valid," but "is this done."
The part that's easy to miss
The instinct once you've built something like this is to treat it as purely a UI nicety, debounce the flicker, smooth the experience. But the actual failure we hit wasn't a UI glitch. It was an epistemic one: a human formed a belief about a fact ("counterparty is Meridian Hold, whatever that is") from data that hadn't finished existing yet. The fix isn't cosmetic. It's a correctness boundary between "data that exists" and "data that is still in the process of becoming data," and once you see it that way, treating it as a parsing convenience undersells what's actually broken.
Where I'd push back on this
The strongest objection to all of the above is that schema-aware incremental parsing adds real complexity, a stateful cursor per field, careful handling of escape sequences, tool-call ID bookkeeping, for a problem that a simpler mitigation might solve just as well: don't stream field values into a UI at all, stream only a coarse progress indicator ("extracting... 60%"), and reveal the full object once json.loads succeeds on the complete response. That removes the entire failure class in one move, and for a lot of products that's the right call, especially early on, when the team building the UI doesn't want to own a state machine.
I'd accept that objection for a lot of use cases. Where I wouldn't accept it is anywhere the whole point of streaming was the perceived-latency win, which was our actual reason for building this. If the product requirement is "show the paralegal something before the model finishes," you've already decided you need partial data, and the choice is really between partial data that's honest about its own completeness and partial data that isn't. Once you frame it that way, the extra state tracking looks less like gold-plating and more like the minimum bar for showing someone a fact you're asking them to trust.
The other pushback worth taking seriously: none of this catches a value that's structurally complete but semantically wrong, a well-formed string that just happens to be the wrong string. That's a different failure class with a different fix, and conflating the two is its own mistake.
Top comments (0)