Five Timeout Anti-Patterns That Turn a Slow Model Call Into an Outage

A model call is slow and variable. A small server has few concurrent slots. Most outages live in that gap, not in the model itself.

This post is a catalog, not a war story. Each anti-pattern gets symptoms, a root cause, and a replacement you can implement today. All code below is illustrative and was not executed for this article.

Why the free tier makes deadlines visible

Free-tier hosting usually means a small instance and a shared network path. That is enough for a demo. It is also enough to hide a missing timeout until traffic arrives.

Two costs stack up fast:

• Retries spend tokens. A retried request is a second billed call on most providers.
• Waiting requests hold slots. Ten blocked handlers can exhaust a small worker pool.

MonkeyCode is relevant here because the operator states it offers free model access and a free server option, and that the project is open source. The operator also states a free token allowance on the order of ten million tokens. Treat those as current-as-of-writing operator claims and confirm them on the project's official repository and terms page before you plan capacity around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Anti-pattern 1: The handler that waits forever

Symptoms. Latency graphs show flat plateaus. Restarts clear the problem. Nobody can reproduce it locally.

Root cause. The HTTP client has no timeout, or a timeout longer than the caller's patience. Every slow upstream response becomes a held connection.

Replacement. Give each request a deadline at the edge. Pass the remaining budget down, not a fresh constant.

Anti-pattern 2: Retry on everything

Symptoms. Token spend spikes during incidents. Upstream error rates look worse from your side than from the provider's status page.

Root cause. A retry wrapper catches all exceptions. Rate limits and validation errors get retried like transient failures.

Replacement. Retry only on a small allowlist, at most once, and only if enough deadline remains. Cap output length so the worst case per attempt is bounded.

Anti-pattern 3: Streaming with a buffered body

Symptoms. Users see nothing until the end. A proxy in between adds its own idle timeout.

Root cause. The handler collects the full stream, then returns it. Time-to-first-byte equals time-to-last-byte.

Replacement. Flush each chunk as it arrives, or drop streaming and return a single bounded response. Do not mix the two.

Anti-pattern 4: Cold start counted as a model error

Symptoms. Error dashboards show 502s at low traffic. Logs contain no upstream response body.

Root cause. Instance warm-up, DNS, and TLS handshake are lumped into one except block. The retry policy then punishes the model for your startup cost.

Replacement. Separate phases in your logs: connect, first byte, total. Tag cold-start failures distinctly so retries stay rare.

Anti-pattern 5: Unbounded in-flight requests

Symptoms. Memory climbs, then the process is killed. Throughput drops as concurrency rises.

Root cause. Concurrency is whatever the load generator sends. There is no queue and no admission control.

Replacement. Adds a semaphore with a measured limit. Reject with 429 above the limit instead of queueing without bound.

The replacement pattern in code

One deadline object, one retry budget, one place where the clock is read.

# illustrative only: not executed for this article
import os, time, uuid, asyncio, httpx
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

BASE_URL = os.environ["MODEL_BASE_URL"]
API_KEY = os.environ["MODEL_API_KEY"]
MODEL_ID = os.environ["MODEL_ID"]
DEADLINE_S = float(os.environ.get("REQUEST_DEADLINE_S", "20"))
MAX_OUT_TOKENS = int(os.environ.get("MAX_OUT_TOKENS", "400"))
MAX_IN_FLIGHT = int(os.environ.get("MAX_IN_FLIGHT", "4"))
CASH_OUT = 0.8  # stop starting work past 80% of the budget

app = FastAPI()
slots = asyncio.Semaphore(MAX_IN_FLIGHT)

class Deadline:
    def __init__(self, seconds: float) -> None:
        self.expires = time.monotonic() + seconds
    def remaining(self) -> float:
        return max(0.0, self.expires - time.monotonic())
    def can_retry(self, need: float = 5.0) -> bool:
        return self.remaining() > need

async def call_model(prompt: str, dl: Deadline) -> str:
    timeout = dl.remaining() * CASH_OUT
    async with httpx.AsyncClient(timeout=timeout) as client:
        r = await client.post(
            f"{BASE_URL}/chat/completions",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={
                "model": MODEL_ID,
                "messages": [{"role": "user", "content": prompt}],
                "max_tokens": MAX_OUT_TOKENS,
            },
        )
        r.raise_for_status()
        return r.json()["choices"][0]["message"]["content"]

RETRYABLE = {429, 500, 502, 503, 504}

@app.post("/answer")
async def answer(req: Request):
    trace = uuid.uuid4().hex[:12]
    body = await req.json()
    dl = Deadline(DEADLINE_S)

    if slots.locked():
        return JSONResponse(status_code=429, content={"trace": trace, "error": "busy"})

    async with slots:
        for attempt in (1, 2):
            try:
                text = await asyncio.wait_for(call_model(body["prompt"], dl), dl.remaining())
                return {"trace": trace, "text": text, "attempt": attempt}
            except httpx.HTTPStatusError as e:
                if e.response.status_code not in RETRYABLE or not dl.can_retry():
                    return JSONResponse(status_code=502, content={"trace": trace, "error": "upstream"})
                await asyncio.sleep(0.2 * attempt)
            except (asyncio.TimeoutError, httpx.TimeoutException):
                return JSONResponse(status_code=504, content={"trace": trace, "error": "deadline"})

    return JSONResponse(status_code=504, content={"trace": trace, "error": "deadline"})

Three properties matter more than the framework:

1. Every wait is bounded by the same clock.
2. Retries are capped, so token spend per request has a ceiling.
3. Overload returns quickly instead of queueing.

How to verify it

You need a fake upstream that sleeps on demand. Do not test this against a live model.

Case	Fake upstream delay	Expected result
Normal	below cash-out point	200, one attempt
Slow	above deadline	504 with trace id
Flaky	first attempt 503, second fast	200, attempt 2
Always failing	503 twice	502, no third attempt
Overload	any, many clients	429 above the slot limit

Record four numbers per run: p50, p95, 504 rate, and peak in-flight. Skip all invented targets. Use your own traffic history.

A minimal load loop to drive the table:

# illustrative load generator
import asyncio, httpx

async def one(c, i):
    async with c.stream("POST", "http://localhost:8000/answer", json={"prompt": f"q{i}"}) as r:
        return r.status_code

async def main(n=50):
    async with httpx.AsyncClient(timeout=60) as c:
        codes = await asyncio.gather(*[one(c, i) for i in range(n)])
    print({k: codes.count(k) for k in set(codes)})

asyncio.run(main())

When the free server fits

Workload	Fits?	Reason
Short single-turn requests	Yes	Bounded deadline, stateless
Internal tools and prototypes	Yes	Low concurrency, easy restart
Long batch jobs	No	Keep jobs off the request path
Durable queues and state	No	Needs storage you control
Strict latency SLAs	No	Shared capacity adds variance

Who should skip this

Do not adopt this if you already have a queue, a deadline-aware gateway, and per-tenant quotas. You would be duplicating policy.

Skip it too if your handlers must survive restarts with in-flight work. A deadline makes failures fast, not durable.

If you are evaluating the free tier, start with the verification table above. Measure your own p95, then decide whether the free server option in MonkeyCode fits your request shape.