TL;DR: Streaming text is evidence of progress, not evidence of success. Treat the terminal event as part of your API contract: normalize it, test it, record it, and never silently present a partial model response as complete. That distinction makes for a much stronger AI-engineering interview answer than saying "I would use streaming for lower latency."
A streamed response can look healthy right up until it is not. Tokens appear in the UI, the socket stays open for a while, and then the connection disappears or the provider reports a non-successful finish state. If the product displays whatever arrived as a completed answer, it has converted an unknown result into a confident one.
That is a small implementation mistake with large consequences. In an AI product, a partial answer can look plausible enough to be acted on. In an interview, it is also a useful way to show that you distinguish transport progress from application completion.
This post builds a tiny Node.js contract test for that boundary. It has no SDK dependencies and is deliberately provider-neutral.
Why chunks are not a success signal
A streamed answer has at least two layers of state:
- The transport layer: bytes or events are still arriving.
- The generation layer: the provider has declared why generation ended.
Those states are independent. A request can emit 90% of a useful answer and then stop because it hit an output limit, a safety policy, a cancelled request, or a broken connection. The first layer says "some work happened." Only the second can say "this result is complete."
Provider event names differ, which is exactly why application code should not spread raw event checks throughout a UI. OpenAI documents lifecycle events for streaming responses, while Anthropic documents a distinct message streaming event sequence. Keep the provider-specific parsing at one adapter boundary and give the rest of the product a small, stable contract.
For this drill, the adapter produces just two relevant events:
| Event | Meaning |
|---|---|
{ type: "text", value } |
A displayable text fragment arrived. |
{ type: "final", reason, requestId } |
The provider completed the stream and supplied an outcome. |
Build the terminal contract
Here is the collector. It accepts text only when it has seen a final event with reason: "stop". It also rejects the surprisingly common bug where a late event is processed after the response has already been finalized.
function collectText(events) {
let text = "";
let terminal = null;
for (const event of events) {
if (terminal) {
throw new Error("received an event after the terminal event");
}
if (event.type === "text") {
text += event.value;
continue;
}
if (event.type === "final") {
terminal = event;
}
}
if (!terminal) {
throw new Error("stream ended without a terminal event");
}
if (terminal.reason !== "stop") {
throw new Error(`stream did not complete normally: ${terminal.reason}`);
}
return { text, requestId: terminal.requestId };
}
The point is not that every provider literally calls the successful state stop. Map the provider's documented terminal state to your own domain vocabulary. The point is that "the iterator ended" is not a substitute for a terminal contract.
In production, I would return a tagged result instead of throwing directly from a request handler. Throwing keeps the drill compact; a real client needs a state it can render and telemetry it can aggregate.
Prove the unhappy paths
A happy-path demo is not a contract test. The valuable cases are the ones that make a partial answer tempting to show.
Save this as stream-contract-drill.mjs and run node stream-contract-drill.mjs.
import assert from "node:assert/strict";
function collectText(events) {
let text = "";
let terminal = null;
for (const event of events) {
if (terminal) {
throw new Error("received an event after the terminal event");
}
if (event.type === "text") {
text += event.value;
continue;
}
if (event.type === "final") {
terminal = event;
}
}
if (!terminal) {
throw new Error("stream ended without a terminal event");
}
if (terminal.reason !== "stop") {
throw new Error(`stream did not complete normally: ${terminal.reason}`);
}
return { text, requestId: terminal.requestId };
}
assert.deepEqual(
collectText([
{ type: "text", value: "Ship " },
{ type: "text", value: "the test." },
{ type: "final", reason: "stop", requestId: "req_42" },
]),
{ text: "Ship the test.", requestId: "req_42" },
);
assert.throws(
() => collectText([{ type: "text", value: "half" }, { type: "final", reason: "length" }]),
/length/,
);
assert.throws(
() => collectText([{ type: "text", value: "half" }]),
/without a terminal event/,
);
assert.throws(
() =>
collectText([
{ type: "final", reason: "stop" },
{ type: "text", value: "late" },
]),
/after the terminal event/,
);
console.log("stream terminal contract assertions passed");
Those four assertions answer four different operational questions:
- Does a normal stream preserve every fragment and the request identifier?
- Does an output-limit termination refuse to masquerade as a full answer?
- Does a dropped connection remain visibly unknown?
- Does the consumer reject an ordering violation rather than corrupting a finalized answer?
A fifth worthwhile test in a real system is cancellation: assert that a client-aborted request is recorded as cancelled, not as a provider failure. That distinction matters when you decide whether the next change is a UI fix, a timeout adjustment, or a model-capacity investigation.
Decide the product behavior before you ship
Once a terminal reason is explicit, the UI and operations choices become much less vague.
| Terminal state | User-facing behavior | Operational action |
|---|---|---|
stop |
Render the answer as complete. | Record latency and token counts. |
length |
Mark it incomplete; offer a deliberate continuation where safe. | Track prompt/output sizing and continuation rate. |
| Safety refusal | Show the refusal, not a clipped substitute. | Preserve the reason for policy and product review. |
| No terminal event | Say the response was interrupted. | Record provider, request ID, elapsed time, and retry eligibility. |
The subtle row is retry. A blind retry can duplicate writes, tool calls, or side effects. For an idempotent text-only request, a retry or continuation may be reasonable. For an agent that can send messages or mutate data, first make the request idempotent and ensure the tool layer can distinguish "not attempted" from "attempted but response lost."
Turn it into an interview answer
When an interviewer asks how you would build a streaming AI feature, start with the user-visible latency benefit, then make the reliability boundary concrete:
"I would stream text to improve time-to-first-token, but I would not treat the arrival of chunks as completion. I would normalize provider events behind an adapter, require a terminal outcome, test successful, length-limited, interrupted, and out-of-order cases, and instrument the terminal reason with the request ID. That lets the UI be honest about partial answers and lets us decide whether a retry is safe."
That is a compact answer, but it carries several senior signals: an interface boundary, failure-state design, observability, and idempotency awareness.
For rehearsal, aceround.app is an AI interview assistant that can help keep this four-part explanation visible while you practice responding to follow-up questions under time pressure. The important part is still being able to defend the trade-off in your own words.
FAQ
Should I automatically continue every length-limited response?
No. A continuation can be useful for a read-only explanation, but it may repeat or contradict the partial answer. Give the continuation a clear boundary, preserve the incomplete state, and never replay a request with side effects unless the downstream action is idempotent.
Is a transport error the same as a refusal?
No. A refusal is a known terminal outcome. A disconnected stream without a terminal event is unknown. Conflating them makes support, monitoring, and user messaging worse because the recovery options differ.
Do I need this abstraction if I use only one model provider?
Yes. Even one provider can evolve event names and finish states. A narrow adapter keeps that churn out of the UI, and the test documents the behavior your product relies on.
Sources
- OpenAI, Streaming responses
- Anthropic, Streaming Messages
Disclosure: This article was drafted with AI assistance and reviewed, tested, and edited by the author.
Top comments (0)