DEV Community

jidonglab
jidonglab

Posted on

requests timeout=30 Isn't a Deadline: One LLM Stream Ran 31 Minutes

I set timeout=30 on every request in my batch job. One of those requests ran for 31 minutes, and nothing in the stack considered that a problem.

No exception and no log line. One of my eight worker threads sat there, receiving a few bytes every 15 seconds, for half an hour. The Python requests timeout did exactly what it's documented to do. I just hadn't read the documentation closely, and I suspect most people calling LLM streaming endpoints with requests haven't either.

Here's the setup, the numbers, and the fix that brought my worst case from 31 minutes down to 105 seconds.

TL;DR

  • The Python requests timeout is not a total deadline. It caps how long the client waits for the next bytes from the socket. Every byte that arrives resets the clock.
  • Streaming LLM responses never trip it. Tokens trickle in every few hundred milliseconds, so a 9-minute generation never goes 30 seconds without a byte.
  • SSE keep-alive pings make it worse. My proxy (sse-starlette) sends a : ping comment every 15 seconds by default. A stalled upstream looked alive forever.
  • In 2,000 streamed calls: 49 ran past 30s, 0 raised ReadTimeout, the longest ran 31 minutes.
  • Fix: keep the read timeout, add your own wall-clock deadline checked inside the stream loop (or use asyncio.timeout with httpx). Worst case becomes deadline + one read interval.

The setup

The job: summarize and tag 2,000 support transcripts with a self-hosted model. The pieces:

  • A vLLM server running an 8B instruct model on one GPU
  • A small FastAPI gateway in front of it that handles auth and re-emits the stream using sse-starlette
  • A Python client using requests.post(..., stream=True, timeout=30) with a ThreadPoolExecutor of 8 workers
  • max_tokens=8192, because some transcripts were long and I was lazy

My mental model was "any single call that takes over 30 seconds gets killed, so the job finishes in about 25 minutes." The job took 58 minutes. That's what made me start logging.

What does the timeout parameter in Python requests actually do?

The timeout in Python requests is two separate idle timers: a connect timeout (how long to wait to establish the TCP connection) and a read timeout (how long to wait between bytes from the server). Passing timeout=30 sets both to 30 seconds. Neither one limits the total duration of the request.

The requests docs say this plainly: the read timeout is the number of seconds the client will wait for the server to send a byte. Strictly speaking it's the time between socket reads. A server that sends one byte every 29 seconds can keep a request open until the heat death of the universe.

For a normal JSON API this barely matters. The server computes, then dumps the whole response at once. Idle time and total time are roughly the same thing.

For a streaming LLM endpoint they are completely different things.

Why don't streaming LLM responses ever hit the read timeout?

Streaming LLM responses send a new chunk every time the model emits a token, typically every 20 to 100 milliseconds. The read timer resets on each chunk, so the request only times out if the model goes silent for the full timeout window. A slow or endless generation never does.

I added timing to every call: start time, first-byte time, end time, bytes received, and how many of those bytes were SSE comment lines. Here's what 2,000 calls looked like:

Bucket Calls What was happening
Finished under 30s 1,951 Normal
30s to 2 min 40 Long transcripts, legitimately long outputs
4 to 9 min 8 Repetition loops running all the way to 8,192 tokens
31 min 1 Upstream stalled, proxy kept pinging
Raised ReadTimeout 0

Zero. Not one call in 2,000 tripped the timeout I thought was protecting me.

The 8 repetition loops were the classic small-model failure: the summary ends, then the model starts repeating its last bullet point, over and over, until it hits max_tokens. At around 30 tokens per second under load, 8,192 tokens takes about four and a half minutes. The slowest one took 9 minutes and 14 seconds because it overlapped with the other loops and throughput dropped.

None of those were a timeout bug in the strict sense. They were the timeout doing its job while I expected it to do a different job.

How did one request run for 31 minutes?

One request ran for 31 minutes because the gateway kept the connection alive with SSE keep-alive comments after the model server stopped sending tokens. Each : ping line arrived every 15 seconds, well inside the 30-second read timeout, so the client treated a dead stream as a healthy one.

This is the one that actually hurt. sse-starlette sends a ping comment every 15 seconds by default. It's a good default: it stops load balancers and browsers from killing idle connections. But it means the gateway is telling the client "still here!" regardless of whether anything upstream is happening.

What happened upstream: the vLLM server hit a rough patch (I never fully pinned it down; it lined up with the 8 repetition loops hogging the batch), and one request stopped producing tokens. The gateway's own call to vLLM had no read timeout configured. So:

  1. vLLM goes quiet on one request
  2. The gateway waits forever on its upstream read
  3. sse-starlette keeps emitting : ping every 15 seconds on the downstream side
  4. The client's 30-second read timer resets every 15 seconds
  5. Nobody ever gives up

My logs showed that request receiving 124 ping comments and zero data events. 124 × 15 seconds is 31 minutes. I killed it by hand when I noticed one worker hadn't logged anything since the first coffee.

That one stuck request held one of my 8 workers for over half the job's runtime. The repetition loops held more. Together they explain most of the gap between the 25 minutes I expected and the 58 minutes I got.

How do you set a total timeout for a Python requests stream?

Python requests has no built-in total timeout, so you enforce a wall-clock deadline yourself. Record time.monotonic() before the call, check it inside the loop that reads the stream, and close the response once the deadline passes. Keep the read timeout too, because it covers the case where no bytes arrive at all.

Here's the client loop after the fix:

import time
import requests

READ_TIMEOUT = 30      # max silence between bytes
DEADLINE = 90          # max total wall-clock time

class DeadlineExceeded(Exception):
    pass

def stream_completion(session, url, payload):
    start = time.monotonic()
    chunks = []
    with session.post(url, json=payload, stream=True,
                      timeout=(5, READ_TIMEOUT)) as resp:
        resp.raise_for_status()
        for line in resp.iter_lines(decode_unicode=True):
            if time.monotonic() - start > DEADLINE:
                raise DeadlineExceeded(f"{DEADLINE}s exceeded")
            if not line or line.startswith(":"):
                continue  # blank separator or keep-alive ping
            if line.startswith("data: "):
                data = line[6:]
                if data == "[DONE]":
                    break
                chunks.append(data)
    return chunks
Enter fullscreen mode Exit fullscreen mode

Two details matter here.

The deadline check runs before the ping filter. If you continue on comment lines before checking the clock, a stream of pings never reaches your deadline check. That's the exact bug I was fixing, so I'd rather not reintroduce it one line later.

The deadline is only checked when a line arrives. If the server goes fully silent, the loop blocks in iter_lines and only the read timeout can save you. That's why you need both. Your real worst case is DEADLINE + READ_TIMEOUT, or DEADLINE + ping interval if pings are flowing. With my numbers: 90 + 15 = 105 seconds.

If you need a hard cap that doesn't depend on bytes arriving, a watchdog thread that calls resp.close() after the deadline works in my testing, but closing a socket from another thread is the kind of thing I'd rather not rely on. If you can go async, it's cleaner:

import asyncio
import httpx

async def stream_completion(client: httpx.AsyncClient, url, payload):
    async with asyncio.timeout(90):  # Python 3.11+
        async with client.stream("POST", url, json=payload,
                                 timeout=httpx.Timeout(30, connect=5)) as resp:
            async for line in resp.aiter_lines():
                ...
Enter fullscreen mode Exit fullscreen mode

Note that httpx's Timeout has the same shape as requests: connect, read, write, and pool timeouts, all per-operation. None of them is a total deadline. asyncio.timeout is what actually cancels the whole thing.

The other two fixes

The client deadline handles symptoms. Two more changes handled causes:

  1. Gateway upstream timeout. The FastAPI gateway now has a 30-second read timeout on its own call to vLLM. If upstream goes silent, the gateway closes the stream and sends an error event instead of pinging forever. Keep-alive pings should mean "I'm alive and my upstream is alive," and that only holds if the gateway is enforcing the second half.
  2. max_tokens sized to the task. My summaries never legitimately exceeded about 900 tokens. I dropped max_tokens from 8,192 to 2,048. A repetition loop now costs about a minute of GPU time instead of four to nine.

The rerun

Same 2,000 transcripts, same model, same 8 workers:

  • Job time: 58 minutes → 27 minutes
  • Longest single call: 31 minutes → 104.8 seconds
  • Calls stopped by the deadline: 11 (8 repetition loops, 2 slow long answers, 1 upstream stall caught by the gateway first)
  • ReadTimeout raised: still 0 on the client, which is now fine, because that's not the timer doing the real work

The 2 slow long answers that hit the deadline were real losses. I retry deadline failures once with a shorter prompt, and both succeeded on retry. A 90-second deadline is a policy choice: pick it from your own latency distribution, not from my numbers.

So is requests timeout=30 a deadline?

No. In Python requests, timeout=30 means the client gives up if the server goes 30 seconds without sending a byte; it does not cap total request time. For streaming LLM calls, tokens and SSE keep-alive pings reset that timer constantly, so a stream can run for minutes or, in my case, 31 minutes without ever raising ReadTimeout. To get a real deadline, track elapsed time with time.monotonic() inside your stream loop (before filtering pings) and abort when it's exceeded, or wrap an async httpx stream in asyncio.timeout. Keep the read timeout as well, and put a read timeout on any proxy that emits keep-alive pings, or a dead stream will look healthy until somebody notices it.


Written by the developer behind Preterview, an interview prep platform.

Top comments (0)