Why Did Every Model Call Timeout After an Hour of Success?

Have you ever watched a service work flawlessly for an hour, then suddenly every request times out for no obvious reason? That happened to me with a small proxy I built to reach MonkeyCode's free model access from a free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The CPU was idle, memory was half empty, and the model API's status page was green. Yet every call failed. The answer was hiding in a place I rarely check: the number of open file descriptors.

The Symptom: An Hour of Success, Then Total Failure

I deployed a simple Python proxy that accepted requests from my local machine, forwarded them to the model endpoint, and streamed the response back. For the first 40 minutes, everything worked. Then requests started timing out. Not all of them, but enough to make the service unusable. The pattern was suspicious: the failures began only after a burst of traffic, and they didn't stop even when traffic dropped.

I restarted the process, and everything worked again. For 40 minutes. That was my first clue that something was accumulating over time.

The Obvious Suspects (and Why They Were Innocent)

I checked the usual suspects. CPU usage was below 10 percent. Memory was stable. The network looked fine. The model API's status page showed no incidents. I even increased the timeout from 10 seconds to 30 seconds, which made the problem worse, because now requests waited longer before failing.

Then I ran ss -s and saw something odd: the number of established connections was around 1,000, even though my proxy should only have a handful of active requests. I checked ulimit -n and found the limit was 1,024. That was the ceiling.

The Real Culprit: A Connection Pool Leak

My proxy was using a requests.Session to call the model API, but I had never configured the connection pool size. Worse, I was reading the response body but never closing the response object. In requests, if you don't close a response, the underlying connection is not returned to the pool. Over time, every request leaked a connection, and once the pool hit the file descriptor limit, every new request failed.

The model API was fine. My code was slowly eating its own sockets.

What a Connection Pool Does (and Doesn't Do)

A connection pool reuses TCP connections to avoid the overhead of a new handshake for every request. But a pool is not a garbage collector. If you take a connection and never give it back, the pool grows until the operating system says "no more." On a free server, that limit is often 1,024 file descriptors, and once you hit it, even a healthy model looks down.

Reproducing the Leak in 20 Lines of Python

You don't need a model API to reproduce this. Here's a minimal script that simulates the leak by creating sockets and never closing them:

import socket
import time

sockets = []
try:
    for i in range(1100):
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.connect(('example.com', 80))
        sockets.append(s)
        if i % 100 == 0:
            print(f'Open sockets: {i}')
        time.sleep(0.01)
except OSError as e:
    print(f'Failed at socket {len(sockets)}: {e}')
finally:
    for s in sockets:
        s.close()

On a typical Linux box with ulimit -n set to 1,024, this script will fail around socket 1,021 with "Too many open files." That's exactly what my proxy hit.

The Fix: A Bounded, Reusable Connection Pool

The fix was simple: use a Session with an explicit connection pool size, and always close the response. Here's the corrected pattern:

import requests
from requests.adapters import HTTPAdapter

session = requests.Session()
adapter = HTTPAdapter(
    pool_connections=10,
    pool_maxsize=10,
    max_retries=3,
)
session.mount('https://', adapter)

def call_model(prompt):
    with session.post(MODEL_URL, json={'prompt': prompt}, timeout=30) as response:
        return response.json()

The with block closes the response, which returns the connection to the pool. The pool is bounded, so even if something goes wrong, the number of open connections can't grow forever. I also added a health check that watches the file descriptor count and alerts me if it climbs above 800.

A Quick Decision Table for Connection Pooling

Scenario	Recommended pool size	Notes
Low traffic (1-5 req/s)	10	Enough for bursts, low memory cost
Medium traffic (10-50 req/s)	50	Watch FD count, set alerts
High traffic (100+ req/s)	100+	Consider a message queue instead
Streaming responses	10-20	Keep connections open longer, monitor carefully

Remember: pool size is not the same as concurrency. You can have 100 concurrent tasks sharing a pool of 10 connections; the rest will wait. If you need true parallelism, you need a larger pool, but every connection costs a file descriptor.

Limitations and Who Should Skip This

This fix works for long-running processes on a single server. If you're using a serverless platform that manages connections for you, or if your traffic is so low that you never hit the limit, you may not need to touch the pool at all. But if you're on a free server with a strict ulimit, and you're calling any external API, check your connection count before you blame the upstream.

The model API was never the problem. My connection pool was leaking, and the file descriptor ceiling turned a small bug into a total outage.

So, when was the last time you checked ss -s on your server? It might be telling you a story you don't expect.