Streaming Interrupted: How to Debug “Successful” LLM Streams (Before Support Tickets Start)

Streaming failures are the worst kind of incidents: your API call can look successful while users still get a broken experience—cut-off answers, truncated JSON, missing tool outputs, or long “hangs” after the first tokens.

The fastest way to stop guessing is to instrument streaming so you can answer one question with confidence:

Did the model stream actually finish, or did it stop “silently”?

In this post, I’ll show you the logging shape I use to detect streaming interruptions, plus a practical checklist to find the root cause quickly.

---

Why streaming “success” is not success

When you stream, you have more failure modes than plain REST calls. A request can return HTTP 200 and still be wrong in practice:

• the stream starts, but ends early
• chunks stop arriving (stall)
• the stream ends with fewer tokens than expected
• streaming completes but the output is malformed (e.g., incomplete JSON)
• the client disconnects mid-stream
• retries and fallbacks hide the original failure from user reports

If your logs only capture status code and latency, you’ll miss the real issue. You need stream lifecycle and what you actually received.

---

The logging schema that makes streaming debuggable

For every logical LLM request, log enough information to answer four questions:

1. Was the stream finished?
2. How much data did we receive (chunks/tokens)?
3. Why did it stop (stop vs interruption / completion reason)?
4. Did retry or fallback hide it?

Below is a minimal event model you can adapt.

1) Successful stream (completed normally)

{
  "event": "llm_stream",
  "request_id": "req_123",
  "provider": "tokenbay",
  "model": "gpt-4.1-mini",
  "operation": "chat_completion",

  "streaming": true,
  "status": "success",
  "stream_started_at": 1719999990000,
  "stream_finished_at": 1719999991842,
  "stream_duration_ms": 1842,

  "retry_count": 0,
  "fallback_from": null,
  "fallback_to": null,

  "chunks_received": 42,
  "tokens_received": 256,

  "completion_reason": "stop",
  "client_disconnected": false,

  "error_type": null,
  "error_message": null
}

2) Interrupted stream (ended early / disconnected / incomplete)

{
  "event": "llm_stream",
  "request_id": "req_124",
  "provider": "tokenbay",
  "model": "gpt-4.1-mini",
  "operation": "chat_completion",

  "streaming": true,
  "status": "interrupted",
  "stream_started_at": 1719999990000,
  "stream_finished_at": 1719999993020,
  "stream_duration_ms": 3020,

  "retry_count": 1,
  "fallback_from": "gpt-4.1-mini",
  "fallback_to": "backup-model",

  "chunks_received": 8,
  "tokens_received": 43,

  "completion_reason": null,
  "client_disconnected": true,

  "error_type": "stream_interrupted",
  "error_message": "Client disconnected"
}

What matters most: the categories.

• lifecycle (started/finished/duration)
• received amount (chunks/tokens)
• termination (completion_reason or null)
• whether retries/fallback were involved (retry_count, fallback_from/to)

---

Normalize stream failures into a small set of error_types

Raw streaming errors are inconsistent across SDKs and providers. Normalize them into stable categories so dashboards and alerts work.

A practical set:

• client_disconnected
• upstream_timeout
• stream_interrupted
• json_incomplete (incomplete structured output)
• max_tokens_reached (if you can detect it)
• unknown_stream_failure

Even if your exact cause varies, the category stays consistent.

---

A practical wrapper: detect “no terminal finish” during streaming

The core idea:

Treat a stream as “completed” only when you receive a terminal signal (or equivalent).

If the connection ends without a terminal finish, you’ve got an interruption.

Here’s a working Node.js pattern using an OpenAI-compatible streaming interface. You will need to adapt two parts to your exact SDK payload:

• how you extract terminal signals (e.g., finish_reason)
• how you count tokens (provider usage fields vs your own estimation)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.LLM_API_KEY,
  baseURL: process.env.LLM_BASE_URL || "https://api.openai.com/v1"
});

function nowMs() {
  return Number(process.hrtime.bigint() / 1000000n);
}

function logEvent(evt) {
  console.log(JSON.stringify(evt));
}

export async function streamLoggedChatCompletion({
  requestId,
  provider = "tokenbay",
  model,
  messages,
  temperature = 0.2,
  maxTokens = 500,
  onToken // optional callback for UI
}) {
  const startedAt = nowMs();

  const base = {
    event: "llm_stream",
    request_id: requestId,
    provider,
    model,
    operation: "chat_completion",

    streaming: true,
    retry_count: 0,
    fallback_from: null,
    fallback_to: null
  };

  let chunksReceived = 0;

  // Token counting:
  // - If your provider returns usage or token counts for streaming, use that.
  // - Otherwise you can estimate, but be explicit.
  // Here we keep it simple: count received text length as a placeholder.
  let tokensReceived = 0;

  let terminal = false;
  let completionReason = null;

  try {
    const stream = await client.chat.completions.create({
      model,
      messages,
      temperature,
      max_tokens: maxTokens,
      stream: true
    });

    for await (const event of stream) {
      chunksReceived += 1;

      const delta = event?.choices?.[0]?.delta || {};
      const content = delta?.content;

      if (typeof content === "string") {
        onToken?.(content);
        tokensReceived += content.length; // placeholder; replace with real token usage if available
      }

      // Terminal detection: adapt this to your SDK/provider payload.
      const fr = event?.choices?.[0]?.finish_reason;
      if (fr) {
        terminal = true;
        completionReason = fr;
      }
    }

    const endedAt = nowMs();
    const duration = endedAt - startedAt;

    if (!terminal) {
      // Connected but no terminal finish signal => interruption/incomplete stream
      logEvent({
        ...base,
        status: "interrupted",
        stream_started_at: startedAt,
        stream_finished_at: endedAt,
        stream_duration_ms: duration,
        chunks_received: chunksReceived,
        tokens_received: tokensReceived,
        completion_reason: null,
        client_disconnected: false,
        error_type: "stream_interrupted",
        error_message: "Stream ended without terminal finish"
      });

      return { ok: false, interrupted: true };
    }

    logEvent({
      ...base,
      status: "success",
      stream_started_at: startedAt,
      stream_finished_at: endedAt,
      stream_duration_ms: duration,
      chunks_received: chunksReceived,
      tokens_received: tokensReceived,
      completion_reason: completionReason,
      client_disconnected: false,
      error_type: null,
      error_message: null
    });

    return { ok: true, interrupted: false };
  } catch (err) {
    const endedAt = nowMs();
    const duration = endedAt - startedAt;

    const msg = String(err?.message || "");
    const clientDisconnected = msg.toLowerCase().includes("disconnect");

    logEvent({
      ...base,
      status: "interrupted",
      stream_started_at: startedAt,
      stream_finished_at: endedAt,
      stream_duration_ms: duration,
      chunks_received: chunksReceived,
      tokens_received: tokensReceived,
      completion_reason: null,
      client_disconnected: clientDisconnected,
      error_type: clientDisconnected ? "client_disconnected" : "stream_interrupted",
      error_message: msg || "Unknown error"
    });

    throw err;
  }
}

Two common gotchas:
1) tokensReceived above is a placeholder. Replace it with real token accounting if you can (usage fields, provider logs, or a token estimator you trust).

2) Terminal detection (finish_reason) is provider/SDK-dependent. The pattern is right; the exact field name must match your runtime.

---

Silent streaming failures you should watch for

Once you have the events, you can catch regressions quickly. Here are patterns that consistently show up in production:

1. Chunks/tokens suddenly drop for a specific model or route
   
   Compare chunks_received / tokens_received distributions by model and time window.

2. Completion reason becomes null more often
   
   If completion_reason is missing, you likely have a terminal detection mismatch, a new provider behavior, or a transport issue.

3. Interrupted rate spikes but HTTP errors don’t
   
   That’s the definition of “silent streaming failure”: it looks healthy in REST metrics, but not in stream lifecycle metrics.

4. Retries increase while user experience seems OK
   
   Users might only see the final attempt. Your logs will show the hidden retry loop.

5. Fallback becomes the default
   
   If fallback_to is frequent after interruptions, the system is masking an upstream streaming stability issue.

---

The alert set that won’t annoy you

If you only add a few alerts, add these:

• interrupted_rate > baseline (per model + feature/route)
• success stream_duration_ms p95 shifts upward
• chunks_received median drops for a feature
• completion_reason_null_rate exceeds threshold

This is enough to catch most streaming breakages without drowning in noise.

---

Closing thought

Streaming is not just “more responsive UI.” It changes the failure model.

A request can be “successful” while the stream is actually incomplete.

If your logs can’t tell you whether the stream finished—and how much you received—you’re debugging uncertainty. You don’t need fancy observability. You need stream lifecycle fields.

tokenbay: https://www.tokenbay.com/?utm_source=devto&utm_medium=community_content&utm_campaign=week1_free_content