The Model Responded Fine. My HTTP Client Was Reusing a Dead Connection.

I had a background job that called a free model endpoint every few minutes and posted the reply somewhere. When I ran the script by hand, it worked every single time. When the cron job ran it, roughly one run out of five would hang for thirty seconds and then die with a connection error. Same code, same model, same free server — different outcome.

The setup was simple: a cron job on a free server calling MonkeyCode's free model endpoint, then storing the response. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The inconsistency was the first real clue, even though I didn't recognize it at the time. How could the same request fail one minute and succeed the next?

The symptom that made no sense

The failing runs all followed the same shape. The job would start, the server would pass its health check, and then the model call would sit there for exactly thirty seconds before raising a read timeout. Not a fast failure, not a slow response — a hang that lasted precisely as long as my client timeout. A manual run right after the failure would succeed instantly, which made the whole thing feel like a ghost in the machine.

I checked the obvious suspects first. Cold start? No, the health check passed and the server had been up for hours. DNS? No, resolution was fast and the error was not a name resolution error. Model outage? No, the same request succeeded manually seconds later. I was about to blame the free tier and move on, and that would have been a mistake.

The clue that changed everything

The breakthrough came when I started logging the age of the HTTP connection right before each request. Successful runs always used a connection younger than about sixty seconds. Failed runs always used a connection older than that. The number sixty matched the idle timeout of the gateway in front of the model endpoint, and it matched the hang duration almost perfectly.

That pattern told me the model was probably fine. The real problem was that my HTTP client was reusing a keep-alive connection the gateway had already closed while my job sat idle. The client wrote the request into a dead socket, the gateway never saw it, and the client waited for a response that would never arrive until its own read timeout fired. The model was answering in about a second the whole time; my client was just talking to a ghost.

Reproducing it deterministically

I stopped guessing and wrote a minimal reproduction. The script makes one request to warm up the connection, sleeps longer than the gateway's idle timeout, then makes a second request on the same client.

import asyncio
import httpx

async def main() -> None:
    async with httpx.AsyncClient(base_url="https://api.example.com") as client:
        warm = await client.post("/v1/chat", json={"prompt": "ping"})
        print("warm request:", warm.status_code)

        await asyncio.sleep(75)  # longer than the gateway idle timeout

        try:
            cold = await client.post("/v1/chat", json={"prompt": "hello"})
            print("second request:", cold.status_code)
        except httpx.ReadTimeout as exc:
            print("hung, then raised:", type(exc).__name__)

asyncio.run(main())

With a 75-second sleep, the second request reproduced the hang every single time. With a 30-second sleep, it never failed. That one experiment converted an intermittent production bug into a deterministic one, which is the entire point of a reproduction script.

The fix: recycle connections before the gateway does

The cleanest fix for a low-traffic job like mine is to tell the client to recycle connections before the gateway does. In httpx, that means setting keepalive_expiry on the connection pool.

client = httpx.AsyncClient(
    limits=httpx.Limits(keepalive_expiry=30),
    timeout=httpx.Timeout(10.0, connect=5.0),
)

Now the client closes a connection after 30 seconds of idleness, so the next request opens a fresh one. The cost is an extra TCP and TLS handshake, which is irrelevant for a job that runs every few minutes. I also added a retry that only fires on connection-level errors, because retrying a POST after bytes have been sent can double-execute the model call.

async def call_model(client: httpx.AsyncClient, payload: dict) -> httpx.Response:
    for attempt in range(2):
        try:
            return await client.post("/v1/chat", json=payload)
        except (httpx.ConnectError, httpx.RemoteProtocolError):
            if attempt == 0:
                await client.aclose()
                continue
            raise

Note what this retry deliberately does not catch: ReadTimeout and ReadError are excluded on purpose. Once the request bytes are on the wire, I cannot know whether the gateway processed them, so replaying could charge me twice for tokens or produce a duplicate side effect.

The reusable debugging checklist

This failure looked like a model problem, but it was a client problem. The techniques that exposed it apply to any intermittent network bug:

• Log the age of the connection before each request, not just the response time.
• Match the hang duration against known timeouts: client timeout, gateway idle timeout, proxy timeout.
• Distinguish error types: ConnectError means the socket never opened, RemoteProtocolError means the peer violated HTTP, ReadTimeout means bytes were expected but never arrived.
• Reproduce with a sleep that crosses the suspected threshold, then bisect the threshold.
• Test the fix by disabling keep-alive entirely; if the bug disappears, connection reuse is the culprit.

Who should not use this approach

If your traffic is high, disabling keep-alive or setting a very short keepalive_expiry adds handshake overhead to every request, and my naive retry wrapper is too simple for production load. If your model calls are not idempotent and you cannot tolerate double execution, never retry after the request bytes have been written. And if you are on a shared free server, remember that the gateway's idle timeout is an environment property — measure it yourself instead of copying my sixty-second number.

The model was fine the entire time. My client was just holding a conversation with a connection that no longer existed, and a little connection-age logging caught it in minutes. If you have a similar ghost-connection story, I would love to hear how you traced it.