Who Owns the Deadline? Four Timeout Myths That Break Free Model Calls

Your service calls a free model endpoint. The client timeout is 30 seconds. The model answers in 35.

Who pays the extra five seconds?

Your user does. So does every service behind you.

I run small services against free model servers. MonkeyCode's free model access and free server option keep the barrier low. Low cost is good. Low cost is not low complexity.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This post is a myth-busting FAQ. Each myth is a claim I keep seeing in code. Each myth gets evidence you can reproduce. There is a runnable probe at the end.

The Myths, In One List

• A client timeout is a deadline.
• Longer timeouts are safer.
• Only the model call needs a timeout.
• A connect timeout is enough.

Here is the corrected mental model: timeouts are local limits. Deadlines are end-to-end promises. You need both. Most code only has the first.

Myth 1: A Client Timeout Is a Deadline

What I hear: 'The timeout is 30s, so the function can't hang.'

True. A function cannot hang forever. That is a weak guarantee.

A deadline comes from outside. A user waits two seconds. A queue pops after five. An upstream SLO fails at ten. Your function has no opinion about any of those.

Now chain a few services:

1. Service A waits for Service B. Timeout: 30s.
2. Service B waits for the model. Timeout: 30s.
3. The model is slow. It takes 35s.

What happens? The model finishes at 35s. B abandoned the call at 30s. B returns an error. A waits for a useful answer but now has no time left. The user left after 10s.

The timeout protected nobody. It was local. The deadline was global.

Correction: name the deadline first. Then divide it into slices. Only then pick relative timeouts.

Myth 2: Longer Timeouts Are Safer

What I hear: 'Free servers are slow. Give them a full minute.'

Run the arithmetic. Two nested calls, 60s each. Worst case before your code learns: 120 seconds.

That 120-second window is not free. You hold a connection. You hold a worker. You might hold a database handle. Queueing delays multiply downstream.

A generous timeout does not fix a slow tail. It just waits for the whole tail.

Correction: a timeout is a budget cap. Want to be safer? Fail faster with a useful fallback. A clear 504 beats a late 200.

Myth 3: Only the Model Call Needs a Timeout

What I hear: 'Everything around the call is fast. DNS, JSON, logs — no risk.'

Really? DNS can hang. A cold database can hang. Logs can block under pressure.

Every hop between request and response is latency. None of them is protected by the model timeout.

Correction: measure every hop. Allocate each one a slice. Slice 1: DNS lookup. Slice 2: preprocessing. Slice 3: the model call. Slice 4: validation and response write.

Slices are not guesses. They come from a probe.

Myth 4: A Connect Timeout Is Enough

What I hear: 'If the server is down, connect fails. Timeout on connect only.'

Here is the gap: a server can accept a TCP connection instantly. Then it thinks for a long time. Then it sends the first token.

Or it never sends anything.

Your connect timeout never fires. Your code sleeps on the read. Nobody set a read timeout.

Correction: timeouts on connect, read, and write are separate. A total budget is separate again. In async Python, wrap the whole step in asyncio.timeout.

The Mental Model That Fixes All Four

Write it down before you code.

1. A request arrives. Set a deadline: request_start + 2.0s.
2. Pass the deadline down the stack as an absolute time. Relative timeouts become slices.
3. Each hop asks: How much budget is left?
4. If the answer is zero, stop. Return 504 with the reason.

Here is a small async template:

import asyncio
from time import monotonic

class BudgetExhausted(Exception):
    pass

def remaining(deadline: float) -> float:
    return deadline - monotonic()

async def call_model(client, payload: dict, deadline: float):
    left = remaining(deadline)
    if left <= 0:
        raise BudgetExhausted()
    # The whole I/O step gets one slice of the budget.
    async with asyncio.timeout(left):
        return await client.post('/chat', json=payload)

Not magic. The template forces one question: How much time is really left? Relative timeouts never ask that.

A Probe You Can Run

This is the reproducible part. Point it at any HTTP endpoint you use. It warms up and prints per-hop times. You decide the slices from the output, not from a blog post.

import asyncio
import json
import os
import time
import urllib.request

URL = os.environ.get('MODEL_URL', 'http://127.0.0.1:8000/chat')

def one_call(payload: dict) -> float:
    data = json.dumps(payload).encode()
    req = urllib.request.Request(URL, data=data, method='POST')
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=30) as resp:
        resp.read()
    return time.perf_counter() - t0

async def main():
    payload = {'prompt': 'ping', 'max_tokens': 8}
    # Warm-up beats first-call surprises.
    for _ in range(3):
        one_call(payload)
    times = sorted(one_call(payload) for _ in range(10))
    median = times[len(times) // 2]
    p90 = times[int(len(times) * 0.9) - 1]
    print(f'median={median*1000:.0f}ms p90={p90*1000:.0f}ms')

asyncio.run(main())

Change the URL. Change the payload. Run it twenty times.

If your p90 is triple your median, a fixed 30-second client timeout is a lottery ticket. You want a deadline budget that includes the tail.

Decision Table: When the Budget Runs Out

The hard part of a deadline budget is the response.

Situation	Evidence in your probe	Action
Connect fails fast	OSError within 50ms	Retry once. Connect failures are often transient.
Read times out, first token arrived	Partial body streamed	Do not retry. State is ambiguous. Return 504.
Read times out, nothing arrived	0 bytes in the window	Retry once only if the caller allows it.
Total budget exhausted	Median and p90 confirm it	Stop. Degrade to a fallback or an error.

The table keeps decisions where the data lives. This script measures. It does not retry. Recovery is where your product policy starts.

Who Should Skip This Approach

This is not a universal law. Skip it when:

• You run one-shot batch jobs. A 60-second wait is fine at 2 AM.
• You have no user-facing request. No deadline exists.
• Your chain has one hop. One timeout equals one deadline.

Otherwise the claim holds: your code needs deadlines, not just timeouts. Free model servers are a great place to practice, because the tail is visible. If you handle the budget there, your paid-tier calls will feel boring — and boring is good.

Next time you adjust a timeout, write the deadline down in code. Then time every hop. The answer often surprises you. It will also surprise your users less.