48-Hour Field Notes: The Stream Looked Healthy Right Up Until the Proxy Hung Up

The bug that only showed up in the last 200 bytes

Have you ever had a streaming call that finished without an error, returned HTTP 200, and still handed you half a JSON object? I spent two days on exactly that shape of failure, and the annoying part was that nothing in my logs looked wrong. The client printed "done", the exception handler stayed quiet, and the downstream parser was the first thing to complain. So which component was actually lying to me?

The answer, in my case, was none of them individually — my client and the proxy simply had different ideas about when a stream is finished. My client treated a closed socket as success, while the proxy treated a long-running response as a resource leak worth killing. I wrote this up as field notes because the fix was less interesting than the instrumentation that made the kill visible.

Three clocks are running on one connection

A streaming HTTP response is governed by at least three independent timers, and most client libraries expose only one of them.

• Idle gap: how long the socket can sit without a byte arriving before the read raises a timeout.
• Total request budget: how long the whole response may stay open, enforced by proxies, load balancers, or your own job runner.
• Generation time: how long the producer genuinely needs, which you do not control and cannot infer from the first chunk's speed.

In Python's requests, the timeout= tuple is (connect, read), and the read value is per socket read, not per request. That distinction matters enormously here: a stream that emits a keepalive every five seconds will never trip a 30-second read timeout, even if it runs for twenty minutes and gets guillotined by a proxy at some unrelated limit.

What I tried, in order

Hours 0–6: trust the status code

My first pass wrapped the call in try/except, checked response.status_code, and moved on. This is the default posture for most of us, and it is wrong for streaming, because a truncated body still arrives with a 200 and a valid header block. Errors at the end of a stream are frequently invisible to status-code checks.

Hours 6–14: timestamp every chunk

Next I added per-chunk arrival times, which is the single highest-value change in this whole exercise. Once I had a trace, I could distinguish "slow producer" from "wire died" without guessing. Here is the collector I now keep in a scratch directory and reuse whenever a stream misbehaves:

#!/usr/bin/env python3
"""stream_trace.py - record per-chunk arrival times from an SSE endpoint.
Adjust the terminator line to whatever your provider actually sends."""
import json
import sys
import time

import requests


def trace(url, headers, payload, out_path, idle_timeout=30.0):
    rows = []
    start = last = time.monotonic()
    terminator = False
    try:
        with requests.post(
            url, headers=headers, json=payload,
            stream=True, timeout=(5, idle_timeout),
        ) as resp:
            resp.raise_for_status()
            for raw in resp.iter_lines(decode_unicode=True):
                if raw is None:
                    continue
                now = time.monotonic()
                if raw.strip() == "data: [DONE]":
                    terminator = True
                rows.append({
                    "t": round(now - start, 3),
                    "gap": round(now - last, 3),
                    "bytes": len(raw.encode()),
                    "head": raw[:120],
                })
                last = now
    except requests.exceptions.ReadTimeout:
        rows.append({"t": round(time.monotonic() - start, 3), "error": "read-timeout"})
    except requests.exceptions.ChunkedEncodingError as exc:
        rows.append({"t": round(time.monotonic() - start, 3), "error": f"chunked:{exc}"})
    finally:
        with open(out_path, "w", encoding="utf-8") as fh:
            for row in rows:
                fh.write(json.dumps(row) + "\n")
        print(json.dumps({
            "elapsed_s": round(time.monotonic() - start, 3),
            "lines": len(rows),
            "terminator_seen": terminator,
            "trace": out_path,
        }))


if __name__ == "__main__":
    trace(
        url=sys.argv[1],
        headers={"Accept": "text/event-stream"},
        payload={"stream": True},
        out_path=sys.argv[2],
    )

A quick analyzer turns the trace into a decision instead of a vibe. I am deliberately not quoting my own numbers here, because free-tier behaviour changes and my trace is not your trace:

import json, statistics, sys

rows = [json.loads(line) for line in open(sys.argv[1], encoding="utf-8")]
gaps = [r["gap"] for r in rows if "gap" in r]
errors = [r for r in rows if "error" in r]
print(json.dumps({
    "lines": len(rows),
    "max_gap_s": max(gaps) if gaps else None,
    "median_gap_s": round(statistics.median(gaps), 3) if gaps else None,
    "head_gaps_s": gaps[:10],
    "errors": errors,
}, indent=2))

If the head gaps are tiny and the tail gap is huge, you have a producer that went quiet. If the gaps are uniformly small and the trace simply stops, something outside your process cut the connection — and that "something" is usually a total-budget rule you never configured.

Hours 14–30: move the collector somewhere it can survive

Running a 20-minute trace from a laptop that sleeps is a waste of a day. I moved the collector into a long-running worker and used MonkeyCode's free server option to host it, driving the requests through free model access so I could iterate on instrumentation without paying per attempt. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat the free access as a way to reproduce this workflow, not as a performance claim — quotas and availability are operator-supplied and can change, so re-read the current docs before you build a schedule around them.

Hours 30–48: define a completion contract

This was the actual fix. Instead of asking "did the call return", I now ask three separate questions and refuse to collapse them into one boolean:

import json


def completion_state(transport_closed, terminator_seen, payload_text):
    if not transport_closed:
        return "transport-open"
    if not terminator_seen:
        return "closed-without-sentinel"
    try:
        json.loads(payload_text)
    except json.JSONDecodeError:
        return "sentinel-but-invalid"
    return "complete"

closed-without-sentinel is the state that burned me, and it is also the state most retry logic silently treats as success. Once that label existed in my logs, the failure stopped being mysterious and started being countable.

Where each timeout belongs

Layer	Setting to make explicit	Failure it prevents	Symptom when misconfigured
Client	read/idle timeout per socket read	hanging on a dead peer	worker pinned open for hours
Client	wall-clock budget in your job runner	runaway generations	partial payload marked done
Proxy / LB	idle timeout between bytes	slowloris-style stalls	502 mid-stream
Proxy / LB	total response budget	long-held connections	clean close, no sentinel
Application	sentinel + payload validation	silent partial output	parser crash downstream

What I would repeat

1. Trace first, theorise second — chunk timestamps ended two days of speculation in about ten minutes.
2. Log the terminal state as a string, not a boolean, so partial completions are greppable.
3. Reconcile a sentinel before parsing, and fail loudly when it is absent.
4. Keep the streaming path and the batch path on the same validator, so a resumed run cannot smuggle in half a record.

What I would not repeat

I would not tune retries before I knew which clock fired, because backoff on a total-budget kill just multiplies the number of truncated payloads I have to clean up. I would also not run long traces from a machine that sleeps, and I would not assume a keepalive means progress — it only means the socket is still warm.

Limitations and who should skip this

If your provider returns a structured finish_reason and your client surfaces it, you already have most of a completion contract and this instrumentation is overkill. If your streams are short, synchronous, and always consumed inside a single request handler, a simple total timeout is easier to reason about than a trace pipeline. And if you cannot change proxy settings, this workflow will still identify the cut, but the remedy moves to your retry and resume logic rather than your infrastructure.

So — when your next stream returns "successfully", do you actually know whether the last chunk arrived, or merely that the socket closed politely?