Half an Answer Is Worse Than No Answer: Streaming Termination as a Protocol

Last month our customer-support agent started answering in fragments. A user would ask about a refund policy and receive "Your refund will be processed within 5–7 business days and you will receive a confirmation email at" — and then nothing. The UI showed a clean end state, the server logs recorded a 200 response, and the observability dashboard reported zero errors. From every metric that mattered, the system was healthy. The user, meanwhile, had just read half a sentence and had no idea whether the rest was coming.

The root cause was not the model. It was my streaming client's assumption that a closed connection means a complete response. I spent a week reproducing the failure, and the fix required treating streaming termination as a protocol with explicit states, not as a transport detail. This article walks through that design, and it uses MonkeyCode's free server and current free tier of 10 million tokens as a test bed for the experiments.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The anatomy of a half-answer

A streaming response has a lifecycle that most request-response code never thinks about. The client opens a connection, receives a sequence of chunks, and eventually sees the connection close. The question is: what does that close mean?

In a well-behaved streaming protocol, the server sends an explicit termination marker before closing. For OpenAI-compatible endpoints, that marker is the literal string [DONE]. The marker is a contract: it says "the generation finished, and everything after this point is transport noise." My buggy client never checked for the marker. It treated any close event as a successful end-of-stream, which is roughly equivalent to treating a dropped phone call as the other person hanging up on purpose.

# buggy: any close means success
async def consume_stream(resp):
    text = ""
    async for chunk in resp.content:
        text += chunk.decode()
    return text  # no verification that termination was clean

The failure mode is silent. The response is not an error, so there is no exception to catch. The text is simply shorter than it should be, and the only way to detect the problem is to know what the server intended to send.

Termination as a state machine

A streaming consumer should model the connection as a state machine with four states: connecting, streaming, terminated_cleanly, and terminated_abruptly. The transition into terminated_cleanly requires exactly one thing: the [DONE] marker. Everything else — connection close, timeout, empty chunk, malformed chunk — transitions into terminated_abruptly.

The key insight is that the state machine must treat the marker as data, not as metadata. The marker arrives inside the byte stream, so the consumer has to parse it out before deciding what the close means.

TERMINATOR = b"[DONE]"

class StreamState:
    def __init__(self):
        self.buffer = b""
        self.clean = False

    def feed(self, chunk: bytes):
        self.buffer += chunk
        if TERMINATOR in self.buffer:
            self.clean = True
            self.buffer = self.buffer.split(TERMINATOR)[0]

    def finish(self):
        if not self.clean:
            raise IncompleteStreamError(
                "stream closed without [DONE] marker"
            )
        return self.buffer.decode()

This is a small change, but it converts an invisible failure into an explicit exception. The client now distinguishes between "the model finished its thought" and "the network stopped carrying bytes," and those are two very different facts.

The timeout dimension: how long do you wait for the marker?

The state machine handles the case where the connection closes early. A second failure mode is the connection that stays open but goes silent. The server accepted the request, sent a few chunks, and then stopped. No close, no marker, just silence.

This requires a separate timeout budget for the inter-chunk gap, distinct from the total request timeout. A slow model might legitimately take 60 seconds to produce its first token, but once streaming starts, chunks should arrive at a predictable cadence. If the gap between chunks exceeds a threshold — I use 15 seconds in my harness — the stream is dead, and the client should fail fast rather than hang.

async def consume_with_heartbeat(resp, idle_timeout=15.0):
    state = StreamState()
    async with async_timeout.timeout(idle_timeout):
        async for chunk in resp.content:
            state.feed(chunk)
    state.finish()
    return state.buffer

The two mechanisms work together: the marker detects truncation, and the idle timeout detects silence. Neither alone is sufficient.

Reproducing the failure with fault injection

To validate the client, I built a small fault-injection proxy that sits between the client and a real model endpoint. The proxy can truncate the stream, drop the [DONE] marker, inject an idle gap, or reorder chunks. I ran the suite against MonkeyCode's free server with the current free token allocation, which was enough to execute hundreds of streaming requests without worrying about cost.

The test matrix was simple: four fault classes times three payload sizes, with the acceptance rule that every truncated stream must raise IncompleteStreamError and every clean stream must return the full text.

# fault injection: strip the [DONE] marker
async def strip_terminator(reader, writer):
    async for chunk in reader:
        cleaned = chunk.replace(b"[DONE]", b"")
        writer.write(cleaned)
    writer.close()

The buggy client passed zero of the truncation tests. The state-machine client passed all of them, and the idle-timeout test caught the silent-stream case that neither client had handled before.

Tradeoffs and limitations

Approach	Detects truncation	Detects silence	Cost
Close-event only	No	No	Zero
Marker check only	Yes	No	One line
Marker + idle timeout	Yes	Yes	Two timers

The marker-check approach costs almost nothing, and it catches the most common failure. The idle timeout adds complexity because you have to tune the threshold: too low, and slow-but-healthy streams get killed; too high, and you wait minutes for a dead connection.

There are cases where this design does not help. If the model endpoint itself does not emit a [DONE] marker, you cannot verify clean termination, and you need a different contract. If you are consuming a non-streaming response, the entire state machine is irrelevant. And if the model produces a syntactically valid but semantically wrong answer, no termination protocol will catch it — that is an evaluation problem, not a transport problem.

The counterexample question

Here is the scenario I could not solve with the state machine alone: the server sends the full text, then the [DONE] marker, then the connection drops before the client reads the final chunk. The marker is in the kernel buffer, the client never sees it, and the stream is marked terminated_abruptly even though the answer was complete. Should the client retry the entire generation, or should it trust the partial text and reconstruct the missing marker? And how does it know which case it is in? If your streaming client cannot answer that question, you have not finished the protocol design — and a free test endpoint is the cheapest place to find out.

MonkeyCode provides free models that can run this workflow.