The Half-Sentence Workshop: Making Streamed Model Output Safe to Stop

A support ticket arrived with a screenshot of a form that said status: complete. Underneath it sat a JSON blob that ended in "steps": [1, 2,. The retry counter read three. The database held half a record, and nothing in the pipeline had complained.

That is the failure this workshop teaches people to prevent. Streaming turns a document into a sequence of arrivals, and a sequence can be interrupted in more places than most code is written to survive. Ninety minutes, one file, no network required until the last exercise.

Why the happy path lies

Non-streaming calls give you one of two outcomes: a body or an error. Streaming gives you a body that may become an error halfway through, and the intermediate states look exactly like success to a parser that runs on every chunk. The instinct is to call json.loads inside the loop and catch the exception. That works until the prefix happens to be valid JSON on its own, at which point the loop commits garbage and moves on.

The fix is structural rather than clever. Separate the buffer from the commit, and make the commit conditional on a proof that the document is finished. The rest of the workshop is spent making that proof honest.

The 90-minute plan

Minutes	Block
0–10	Reproduce a cut with a scripted stream
10–30	Build the two-phase gate
30–50	Break the gate with prefixes that parse
50–70	Add abort and commit-once semantics
70–90	Swap in a live endpoint and re-run the suite

Exercise 1: cut the stream at every byte

The first artifact is a harness, not a client. It takes a finished payload and replays it in slices, stopping at a chosen offset to simulate a dropped connection or a user pressing stop.

# harness.py — no network, runs in under a second
import json

PAYLOAD = json.dumps({"title": "half", "steps": [1, 2, 3], "done": True})

def replay(payload: str, cut_at: int, size: int = 5):
    """Yield the payload in slices, then stop dead at cut_at."""
    for i in range(0, cut_at, size):
        yield payload[i:i + size]

Students run replay(PAYLOAD, 12) and print what a naive json.loads does with it. Some offsets raise. Others return a dictionary with missing keys and no error at all. The room usually goes quiet at the second case.

Exercise 2: buffer, then commit

Now the gate. Two methods matter: feed accumulates, finish decides. Until finish says yes, nothing downstream can see the document.

class StreamGate:
    def __init__(self, required):
        self._buf = []
        self._required = set(required)
        self._committed = None

    def feed(self, piece: str) -> None:
        if self._committed is not None:
            raise RuntimeError("feed after commit")
        self._buf.append(piece)

    def abort(self) -> tuple:
        self._buf.clear()
        return ("aborted", 0)

    def finish(self) -> tuple:
        raw = "".join(self._buf)
        try:
            doc, end = json.JSONDecoder().raw_decode(raw)
        except json.JSONDecodeError:
            return ("partial", len(raw))
        if end != len(raw):
            return ("partial", len(raw))
        if not self._required.issubset(doc):
            return ("partial", len(raw))
        self._committed = doc
        return ("ok", doc)

raw_decode returns the index where parsing stopped, which is the piece the naive version is missing. Requiring end == len(raw) rejects a valid prefix followed by anything else. Requiring the declared keys rejects the smaller class of prefixes that are both valid and complete-looking.

Exercise 3: try to fool it

Here is the assertion that carries the whole workshop. Walk every cut point, and the gate must report ok exactly once — at the full length.

def sweep(payload: str, required, size: int = 5) -> list:
    ok_at = []
    for cut in range(len(payload) + 1):
        gate = StreamGate(required)
        for piece in replay(payload, cut, size):
            gate.feed(piece)
        if gate.finish()[0] == "ok":
            ok_at.append(cut)
    return ok_at

assert sweep(PAYLOAD, ["title", "steps", "done"]) == [len(PAYLOAD)]

Then hand out a payload of a bare integer, "12345", and watch the assertion fail. Every prefix is valid JSON, and raw_decode will happily consume 123 and stop. This is the teaching moment: no parser can prove completeness from syntax alone, so the required-key check and an explicit terminator in the protocol are doing real work, not decoration.

Exercise 4: aborts are not retries

The gate needs to distinguish three endings, and conflating them causes duplicate writes. A partial buffer should be discarded and retried from scratch. An abort requested by the user should not be retried at all, because retrying a cancelled request burns tokens on work nobody wants.

The commit-once guard in feed is deliberately noisy — it raises rather than silently ignoring the second stream. During the exercise, students remove the guard and watch two concurrent streams interleave into a buffer that validates against nothing.

Exercise 5: real chunk boundaries

Everything above runs against a fake stream, which is the point: the suite is deterministic and each student can rerun it after the workshop. The final block swaps the fake for a live endpoint so people see how their provider actually slices tokens, which is usually neither per character nor per line.

# Pseudocode for the shape only. Adapt to the SDK you actually use.
def live_pieces(client, prompt):
    with client.stream(prompt) as events:
        for event in events:
            if event.text:
                yield event.text

The adapter is ten lines; the gate underneath does not change. That separation is what makes the workshop portable across providers, and it is the reason the last exercise is a swap rather than a rewrite.

This is where free model access earns its keep. MonkeyCode, as an open-source project, offers a free model tier and a free server option, which means a room of students can run the live exercise without each one provisioning a card or a personal key. The operator also advertises a large free token allowance; treat that figure as perishable, and check the project page on the morning you print the syllabus rather than copying a number out of a blog post.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The honest limitation is that a free tier is a shared resource with terms that can change. If your workshop depends on a specific model or a specific latency, pin it in advance and bring a recorded fallback. The suite runs offline either way, so a bad morning on the network costs you one exercise, not the session.

What this does not fix

Buffering the whole document defeats the purpose if your UI is supposed to paint tokens as they arrive. Everything here applies to the commit boundary, not to the rendering path; a chat surface can stream freely and still refuse to write to the database until finish agrees. Likewise, tool-call streaming usually carries its own completion signal, and you should use that instead of guessing from JSON shape.

Semantic truncation is out of scope too. A payload that parses cleanly but contains two of the four expected steps will pass the gate. Catching that needs business rules, not a parser, and pretending otherwise is how teams end up trusting a green test suite.

Skip this approach if your output is binary, if the provider already gives you a reliable terminal event you can trust, or if you need sub-second first paint and full buffering cannot meet it. For everyone else shipping a form, a job queue, or a database write behind a model call, the two-phase gate is cheap insurance.

Going home with something runnable

Keep the harness and the sweep assertion. The model is replaceable, the chunk sizes are arbitrary, and the payload is a toy — but the property under test is exactly the one that failed in that support ticket. A stream is finished when you can prove it is finished, and nothing else counts.

If you want to run the last exercise against real chunk boundaries instead of mine, the free tier is enough to try it with a handful of prompts.