Every morning at 6:00 AM, the first job on my little service failed, and the log line never changed: model_request timed out after 30s. The model provider looked healthy, my code looked correct, and the exit code was still 0 because the retry wrapper swallowed the exception and returned a fallback payload. It took me two full days to realize I had been debugging the wrong layer entirely.
The service itself is deliberately boring: a background job on a MonkeyCode free server that calls a free model endpoint to summarize overnight notifications. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The whole point of the setup is to cost almost nothing, which means the server sleeps when it is idle, and that single fact explained every symptom I was chasing.
The symptom that pointed the wrong way
The error looked like a model problem because it arrived from the model client, complete with a stack trace that named the upstream request:
httpx.ReadTimeout: timed out
My first instinct was to blame the provider, so I added a retry loop, and that made everything worse: three timed-out attempts in a row, each one waiting the full 30 seconds, turned a five-second job into a two-minute failure. I then raised the timeout to 60 seconds, the job started passing, and I felt vindicated. A slow model, right?
Wrong. The 60-second timeout was just long enough to cover the server's boot time, and my "fix" was masking the real problem while making every cold start feel sluggish. The model call was never the bottleneck, and the evidence was sitting in the timing all along. I just had not thought to look at it until the retries made the failure impossible to ignore.
Isolating the layers
The reusable trick here is to measure each hop separately instead of trusting the error message from the deepest layer. I wrote a tiny probe that times the connection and the time-to-first-byte for both my server and the model endpoint, then compared the numbers side by side:
import time
import httpx
def probe(url: str) -> dict:
with httpx.Client(timeout=httpx.Timeout(connect=5.0, read=90.0)) as client:
t0 = time.perf_counter()
with client.stream("GET", url) as response:
ttfb = time.perf_counter() - t0
body = response.read()
total = time.perf_counter() - t0
return {
"ttfb_ms": round(ttfb * 1000, 1),
"total_ms": round(total * 1000, 1),
"status": response.status_code,
}
for url in ["https://my-free-server.example/health", "https://model-endpoint.example"]:
print(url, probe(url))
The numbers told the story immediately. The model endpoint answered in about 400 milliseconds when I hit it directly, but my own server took 38 seconds to return a single byte on the first request of the day. The model was not slow; the server was still booting, and the request to the model was queued inside my own process the entire time.
Why the cold start hid so well
Three separate things conspired to keep the root cause invisible, and each one is a trap you will meet again:
- The timeout surfaced from the model client, so the stack trace pointed at the provider even though the provider never saw the request.
- The retry wrapper returned a fallback payload, so the job exited cleanly with code 0 and nobody noticed the failure.
- The 60-second timeout "fixed" it, which stopped me from asking why a healthy model would ever need a full minute.
That last point is the real lesson. When a fix works but feels wrong, it usually is wrong, and a healthy model call should never take a minute. My acceptance of that latency was the actual bug in my reasoning.
The fix: separate timeouts, warm the server
I made three changes, and none of them involved the model provider:
- Split the client timeout into connect and read phases, so a slow boot fails fast instead of hanging silently.
- Added a lightweight health-check ping every four minutes from a separate cron job, which keeps the server awake during the hours I actually use it.
- Moved retry logic out of the model wrapper and into the job scheduler, with exponential backoff and a hard cap of two attempts.
client = httpx.Client(
timeout=httpx.Timeout(connect=3.0, read=30.0),
limits=httpx.Limits(max_keepalive_connections=2),
)
*/4 * * * * curl -s -o /dev/null https://my-free-server.example/health
The warm-up ping is an honest trade-off: it costs a few requests per day and only makes sense for workloads with predictable windows. If my job ran at random times, I would keep the fast-fail timeout and let the scheduler handle retries instead of keeping a server awake around the clock.
A quick decision table for the next timeout
When a remote call fails, the error message tells you where the exception was raised, not where the time went. This table is the shortcut I wish I had on day one:
| Symptom | Likely layer | Next step |
|---|---|---|
| Timeout after exactly N seconds | Client timeout config | Probe server time-to-first-byte |
| Slow first call, fast repeats | Cold start or warm cache | Measure server boot time |
| Timeout only during peak hours | Upstream model load | Hit the model endpoint directly |
| Fast failure with no error logged | Swallowed exception | Audit fallback and retry paths |
Limitations and who should skip this
This approach assumes a low-traffic workload where a warm-up ping is affordable and the server is allowed to sleep. If you run a public API with real latency requirements, a free server that sleeps is the wrong foundation, and no amount of timeout tuning will save you. The same goes for genuine model degradation: if the provider itself is slow under load, the cold-start fix changes nothing, so verify the model endpoint from an independent machine before you commit to a diagnosis.
The morning job has been green for a week now, and the change that mattered was not a bigger timeout. It was refusing to accept that a healthy model call should take a minute, and measuring the layer above the error message before touching the provider. The next time your logs blame the upstream, time the hop above it first.
Top comments (0)