The Second Request Hung for 61 Seconds Because My Client Reused a Dead Socket

The same endpoint answered in 1.4 seconds on the first call and then made me wait 61 seconds on the second. That pattern never points at a slow model, and it turned out to be something far more embarrassing on my side. I was running a scheduled prompt job on a free server that talks to MonkeyCode's free model endpoint and stores the latest reply.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The first run after deploy finished in about two seconds, which felt great for a free setup. Five minutes later the second run hung for exactly 61 seconds before returning an answer that had clearly been ready almost immediately. My first instinct was to blame the model provider, then the free server's cold start, then the network between them.

All three suspects were innocent, and the evidence was hiding inside my own HTTP client the whole time.

The symptom that ruled out the obvious suspects

Before touching any configuration, I ran three controlled checks:

• A direct curl from my laptop to the same endpoint: fast on every attempt.
• A brand-new Python process on the free server: fast on the first attempt.
• The scheduled job's second request inside the same process: 61 seconds, every single time.

The pattern was consistent, and consistency is a gift when you are debugging. The slowness followed the process, not the model, not the server, and not the network path. So I asked myself an uncomfortable question: what exactly does my HTTP client keep alive between requests?

Timing every stage instead of the whole call

Total time hides where time goes, so I stopped measuring the whole request and instrumented each stage of the transport. I wrapped the session with a small adapter that logs when send() is called and when the first byte of the response finally arrives.

import time
import requests
from requests.adapters import HTTPAdapter

class StageTimer(HTTPAdapter):
    def send(self, request, **kwargs):
        t0 = time.monotonic()
        print(f'[{t0:.3f}] send() called')
        response = super().send(request, **kwargs)
        print(f'[{time.monotonic():.3f}] response started after {time.monotonic() - t0:.2f}s')
        return response

session = requests.Session()
session.mount('https://', StageTimer())

t0 = time.monotonic()
resp = session.post(
    'https://api.example.com/v1/chat/completions',
    json={'prompt': 'Say hello in one sentence.'},
    timeout=(3, 90),
)
print(f'total: {time.monotonic() - t0:.2f}s')

The output told the whole story in two lines:

[100.000] send() called
[161.020] response started after 61.02s
total: 61.03s

The entire gap lived inside send(), between the moment the request went out and the moment the first byte came back. That is exactly the signature of a stale keep-alive connection, and the model had nothing to do with it.

Why the model's reply never reached the waiting client

Free servers and the proxies in front of them close idle connections aggressively, often after 60 seconds or so. My client's connection pool did not notice, because the close arrives as a TCP FIN that the pool only discovers on the next read.

The failure sequence looks like this:

1. The first request opens a fresh connection and completes in 1.4 seconds.
2. The server closes that idle connection after its keep-alive timeout.
3. The pool still holds the dead socket and hands it to the second request.
4. The client writes the request, and the write succeeds locally because the kernel buffers it.
5. The client waits for a response that will never come until the read timeout finally fires.

The model actually answered in about 1.4 seconds, but its reply landed on a connection the server had already forgotten. My client was not waiting for the model at all; it was waiting for a ghost.

The frustrating part was that every log line looked healthy. The request was sent, the connection was established, and the response eventually arrived, so my dashboard showed a slow call instead of a failed one. Only the stage timers revealed that the socket had been dead before the request was even written.

The fix: stop trusting sockets the server has forgotten

Once the root cause is clear, the fix becomes boring, and boring is a good sign in production. I made three changes, and any one of them would have solved the symptom on its own.

# Option 1: disable keep-alive for this endpoint
session.headers['Connection'] = 'close'

# Option 2: drop the whole pool before each attempt
session.close()

# Option 3: short read timeout, then retry on a fresh connection
for attempt in range(3):
    try:
        resp = session.post(url, json=payload, timeout=(3, 10))
        break
    except requests.exceptions.ReadTimeout:
        session.close()
        continue

Option 1 is the simplest because it tells the server to close the socket after every response, so the pool never keeps a zombie. Option 3 is the most robust because it also covers proxies that silently kill connections without a clean FIN. I also added a log line that prints the age of the pooled connection, so the next occurrence becomes obvious in seconds instead of after another afternoon of guessing.

Retrying is only safe if the call is idempotent or if a duplicate reply is harmless. My job just stores the latest answer, so a duplicate was fine; if your job charges credits or appends records, add a request ID and deduplicate on the server side.

The reusable debugging checklist

This failure cost me an afternoon, and most of that time went to blaming the wrong layer. The next time a remote call hangs, I will run through this list first:

• Measure each stage, not just the total, before blaming any service.
• Reproduce in a fresh process to separate process state from network state.
• Check whether the slow request reused a pooled connection.
• Treat an instant write plus a hanging read as the signature of a dead socket.
• Blame your own client last, but only after you have proven it innocent.

Limitations and who should not copy this approach

Connection reuse is only one failure mode, and my fix trades a little latency for a lot of reliability. If your workload needs every millisecond, disabling keep-alive adds a round trip to every call, so measure the trade-off before you adopt it.

A free server that sleeps between requests is also not a real-time platform, and this approach does not change that. If your users wait synchronously for a model reply, you need a warm instance and proper retries, not a background job that can tolerate a 61-second hang. This workflow fits scheduled jobs where a slow retry is acceptable and a silent failure is not.

The model was never the problem, and the free server was doing exactly what free servers do. The bug lived in my client's polite habit of trusting a socket that had already been buried, and now every connection is guilty until it proves itself alive.