A Streaming Response Is Not a Payload: Buffer JSON Before You Validate

A streaming model response is not a payload until you decide where one ends. Naive JSON parsing fails at chunk boundaries, not because the model output is malformed.

The bug

Most model APIs send responses as a stream. The client sees bytes in chunks, and chunk boundaries are not promises about JSON objects.

• A chunk can end inside the middle of a key name.
• A chunk can start with a comma from the previous object.
• The same response can pass or fail depending on network timing.

You cannot fix this by making the model output smaller. You fix it by separating transport from validation.

What a stream actually looks like

A JSON array of objects may leave the server as one byte sequence and arrive as:

[{"role":"assistant","content":"ok","status":"ready"},{"role":"assistant","content":"ok","status":"ready"},...]

A client that calls json.loads on every chunk will see incomplete fragments like:

[{"role":"assistant","content":"ok","st

That is not bad model output. That is a bad parser.

Reproduce the failure before production

You can run this with any HTTP endpoint that emits Transfer-Encoding: chunked. I use the free server option through MonkeyCode to host the fixture because it removes a local network dependency and lets me replay the same split conditions. The free model access supplies realistic JSON output without a paid key.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Artifact

The first module is a chunked JSON server. It does not model the language model. It models the transport: small, irregular chunks with variable pauses.

# chunk_server.py
import http.server
import json
import random
import time

ROWS = [
    {'role': 'assistant', 'content': 'ok', 'status': 'ready'},
    {'role': 'assistant', 'content': 'ok', 'status': 'ready', 'detail': {'rows': 2}},
]

class Handler(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get('Content-Length', 0))
        self.rfile.read(length)
        body = json.dumps(ROWS * 5).encode()
        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Transfer-Encoding', 'chunked')
        self.end_headers()
        i = 0
        while i < len(body):
            step = random.choice([7, 13, 29, 61, 97])
            chunk = body[i:i+step]
            self.wfile.write(f'{len(chunk):X}\\r\\n'.encode())
            self.wfile.write(chunk + b'\\r\\n')
            self.wfile.flush()
            i += step
            time.sleep(0.01)
        self.wfile.write(b'0\\r\\n\\r\\n')

if __name__ == '__main__':
    http.server.ThreadingHTTPServer(('127.0.0.1', 8001), Handler).serve_forever()

This server writes valid HTTP chunked framing, but the logical JSON object boundary is ignored by the chunker. That is the exact property you need to test.

The second module is a small validator with two modes.

# validate_stream.py
import json
import urllib.request

URL = 'http://127.0.0.1:8001'

def read_chunks():
    chunks = []
    with urllib.request.urlopen(URL, data=b'') as resp:
        for chunk in resp:
            chunks.append(chunk)
    return chunks

def naive_parse():
    data = b''
    for chunk in read_chunks():
        data += chunk
        try:
            json.loads(data)
            return 'unexpected early parse'
        except json.JSONDecodeError:
            continue
    return 'failed until complete'

def buffered_parse():
    chunks = read_chunks()
    record = json.loads(b''.join(chunks))
    assert isinstance(record, list), 'root must be an array'
    for item in record:
        assert set(['role', 'content', 'status']).issubset(item), item
        assert item.get('status') == 'ready', item
    return f'accepted {len(record)} records'

print('naive:', naive_parse())
print('buffered:', buffered_parse())

Run the server first. In a second shell:

python3 chunk_server.py
python3 validate_stream.py

The naive path will sometimes raise unexpected early parse if a random chunk ends exactly on a complete object. More often it will report failed until complete because it only got a valid document after reading every chunk. Either way, the result depends on timing. That is the point.

Test matrix

Condition	Naive parser	Buffered parser
Random chunk sizes 7–97 bytes	nondeterministic pass or fail	always validates whole array
Replayed with same chunk sizes	may change if early object is complete	stable outcome
Required field missing in one record	may parse but skip validation	fails assertion
Server pauses mid-stream	can timeout or hold a partial parse	waits for 0\\r\\n\\r\\n

The buffered parser performs one validation after the terminating chunk. It does not claim the stream is complete early.

What the buffering fixes

It fixes one thing precisely: boundary confusion.

• Transport bytes stay separate from semantic records.
• Validation runs once against a complete document.
• Bad network timing cannot change the validation result.
• Partial objects are not processed as if they were full objects.

It does not fix schema drift, prompt injection, or the model returning the wrong object shape. The assertions above are deliberately tiny.

Limits

• Buffering the entire stream uses more memory. For very large responses, switch to an incremental parser such as ijson or json-stream.
• A timeout must still be enforced. Waiting for the terminating chunk should not be infinite.
• Arbitrary chunk boundaries can also appear in SSE data lines, not only raw chunked HTTP. The same rule applies: accumulate the event first, then parse the data field.
• This fixture uses a local server with random delays. It does not prove any specific vendor's streaming behavior.

Who should skip this

• If you always call the model API through a well-tested client with response.json(), the client already buffers for you.
• If your payloads are small enough to fit in a single response body, a simple read() is enough.
• If you do not return model JSON directly to customers, this may be an internal issue with lower priority.

Closing

Validate the complete document, not the fragments that arrive during a slow afternoon network path. If you maintain a JSON API that consumes model output, run this fixture against your real HTTP client once, then keep the buffered validator in the integration test that every model-backed endpoint must pass before release.

Comments

[Alex Shev]

Streaming JSON is a contract problem before it is a parser problem. Validation needs a complete payload, or the system ends up judging fragments with too much confidence.