48-Hour Field Notes: My Stream Died Mid-Object and I Re-Asked the Model — That Was the Bug

At 02:10 on day two of a 48-hour debugging window, a generation stopped at event 37 of 60 and my consumer printed JSONDecodeError: Expecting value: line 1 column 1 (char 0). The model had done nothing wrong, the payload was simply cut in half somewhere between the socket and my parser. Why does a network failure always arrive dressed as a syntax error, and why was I so quick to believe the error message?

I spent the first several hours treating this as a parser problem, then as a prompt problem, then as a provider problem. All three were wrong. The interesting part is not that a stream died — streams die all the time — but that my recovery path made the failure permanent. Here are the field notes, the artifact that fixed it, and the parts I would do differently next time.

What actually failed: three failures behind one message

The first failure was conceptual. Truncation and corruption are indistinguishable to json.loads, so the exception told me nothing about where the damage happened. The second failure was structural: server-sent events are not transactional, and a cut can land inside a JSON string right after an escaped quote, which is precisely the spot a human eye cannot check. The third failure was mine — my resume logic re-asked the model for the rest of the answer.

That last one deserves its own sentence, because it cost me the most. A resumed request produced a different generation, and my client stitched the tail of B onto the head of A. The output parsed cleanly, which is the worst outcome: a syntactically valid object that describes two different answers.

So my mental model changed from "retry the call" to "make every event self-checking, and never ask the model twice for the same generation."

The artifact: self-checking event frames

The fix is boring and that is the point. Every event becomes one NDJSON line with a sequence number, the payload length, and a truncated hash of the payload. The producer looks like this:

import hashlib, json

def frame(seq: int, obj) -> str:
    payload = json.dumps(obj, separators=(",", ":"), sort_keys=True)
    digest = hashlib.sha256(payload.encode()).hexdigest()[:16]
    envelope = {"seq": seq, "len": len(payload), "sum": digest, "payload": payload}
    return json.dumps(envelope) + "\n"

The consumer accepts an event only if it is contiguous, correctly sized, and correctly hashed. Everything else goes into a quarantine list so you keep the evidence instead of losing it:

import hashlib, json

def consume(lines, last_seq: int = 0):
    good, quarantine = [], []
    for line in lines:
        try:
            env = json.loads(line)
        except json.JSONDecodeError:
            quarantine.append(("unframed", line[:40]))
            continue
        payload = env.get("payload", "")
        sum_ok = hashlib.sha256(payload.encode()).hexdigest()[:16] == env.get("sum")
        if sum_ok and env.get("len") == len(payload) and env.get("seq") == last_seq + 1:
            good.append(json.loads(payload))
            last_seq = env["seq"]
        else:
            quarantine.append((env.get("seq"), "length, hash, or sequence mismatch"))
    return good, quarantine, last_seq

The buffer is the source of truth, not the model

My first resume implementation asked the model to continue, which is exactly how you get two half-answers fused together. The correct design keeps frames server-side, keyed by a hash of the normalized prompt plus parameters, and replays from a client-supplied sequence number. The model is asked once per generation, and the replay never touches it. If you cannot buffer events, you cannot resume — you can only restart, and restarting means throwing away whatever your client already rendered.

A drop-injection test you can run in a minute

Manual testing will never find the cut that lands inside a nested string, so I made the cut a parameter. The harness serializes the full event list, truncates the byte stream at every offset, and asserts two properties: no partial payload is ever accepted, and replay from the last good sequence reconstructs the complete list exactly.

def cut(stream: bytes, at: int) -> bytes:
    return stream[:at]

for at in range(len(stream_bytes)):
    good, quarantine, last_seq = consume(cut(stream_bytes, at).decode(errors="ignore").splitlines())
    assert all(isinstance(e, dict) for e in good)
    assert len(good) == last_seq
# exit non-zero if any partial payload slipped through

Run it with python test_stream.py. It takes seconds, it fails loudly on real regressions, and it is the test I should have written before the retry policy, not after.

Decision table: when framing beats plain retrying

| Situation | Better default | Why |
| --- | --- |
| Long generation, renderer shows partial output | Self-checking frames plus replay | Users see progress, resume is cheap, no fused output |
| Single small JSON response | Plain retry with an idempotency key | Framing overhead buys nothing |
| Provider supports verified resumption | Use theirs, then frame the remainder | Do not rebuild what you can contract for |
| Side effects already executed | Idempotency key first, framing second | Framing protects the read path, not the write path |

Limitations and who should skip this

Framing adds roughly sixty bytes per event and one small parse per line, which is fine for dozens of events and wasteful for a single response. It only helps when you control the producer or the relay; if the provider's stream is the only source, you can still validate on the client but you cannot replay from a server buffer. It also does nothing for side effects — a payment written twice is still written twice, and only idempotency keys fix that. Skip this entirely if your outputs are short, your network path is trusted, or your provider already gives you resumption you have actually tested.

Where the free tier genuinely mattered

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The cut harness multiplies quickly: every event count times every truncation offset, on every change to the framing code. MonkeyCode's free model access kept that loop affordable while I iterated, and the free server option hosted the relay that buffered and replayed frames during the same window. That is the whole role it played — cheap iterations and a place to run the relay, not a claim about speed or scale. I did not benchmark throughput or quota headroom, and availability claims come from the operator, so read the current terms yourself before you depend on either one.

What I would repeat and what I would change

I would repeat the quarantine list, because reading actual broken frames taught me more in ten minutes than a day of guessing. I would repeat making the cut offset a parameter, since that single change turned a heisenbug into a regression test. I would change the order of work: checksum on day one, join test before retry policy, and no resume path that touches the model.

If you are wiring a streaming relay this week, write the cut test before you write the retry logic — and if you need somewhere to run it, the free server option is a reasonable place to start.