The 503 That Wasn't a Rate Limit: A Layered LLM Gateway Debugging Retrospective

Most LLM gateway failures look like upstream rate limits but are actually client-side connection problems, and a recent batch evaluation run proved the point. The run, which sent two hundred prompts against a free-tier model endpoint, failed with intermittent 503s for forty minutes before the real cause surfaced. The session ran on MonkeyCode, an open-source project that offers free model access and a free server option for eval and agent workloads, and the root cause was a stale keep-alive socket pool in the client, not the provider. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Symptom

The job was a small model-eval loop: two hundred prompts, one model, one server, one client process. It failed at prompt forty-seven, then at fifty-one, then at sixty-three, in a pattern that looked random. Each failure consumed the full thirty-second timeout before the retry logic ran, and the retries sometimes succeeded and sometimes failed again. The client log showed 503 Service Unavailable with no Retry-After header, which made the failure look like a hard upstream rejection.

The first instinct was to blame the model provider, because a 503 from a free tier usually means throttling. A check of the status page, the account dashboard, and the token budget showed no incident, no rate-limit event, and no quota exhaustion. That mismatch between the client's error and the provider's telemetry was the first real clue: a 503 without a Retry-After header and without a matching provider-side event is usually not a rate limit at all.

The Wrong Hypothesis

The first twenty minutes were spent assuming the free tier had throttled the workload, and every adjustment targeted that assumption. Adding exponential backoff, reducing concurrency from four to two, and re-running the batch moved the failures but did not remove them. That behavior is the classic sign of a stateful transport problem, because a server-side throttle responds consistently to the same request shape. Changing concurrency changes socket reuse patterns, which changes which requests fail, which is exactly what happened.

Layered Isolation

The fix came from isolating layers in a fixed order instead of guessing, and that discipline turned a forty-minute mystery into a five-minute diagnosis. The routine below is the reusable part of this retrospective.

1. Reproduce outside the loop. Run a single curl request with the same headers, model, and payload. If the single request succeeds while the loop fails, the server is healthy and the problem lives in client state.
2. Inspect the connection pool. Check keep-alive settings, pool size, and socket age in the client library; most HTTP clients reuse sockets indefinitely unless told otherwise.
3. Read the server's request logs. If the server never saw the failing requests, the failure sits between the client and the gateway, not at the model.
4. Compare timing. Measure time-to-first-byte separately from total time; a long connect phase points at the network, while a long response phase points at the model.

A quick decision table makes the routine concrete:

Observation	Likely layer	Next step
Single curl succeeds, loop fails	Client connection state	Shorten keep-alive expiry, recreate the client
Failing requests never reach server logs	Gateway or DNS	Check DNS TTL and egress IP changes
Requests reach the server but respond slowly	Upstream model	Inspect latency percentiles, not averages
503 with a Retry-After header	Real rate limit	Respect the header and back off

Root Cause

The free server had recycled its egress connection during a routine restart, and the client's keep-alive pool still held sockets to the old address. Every few requests, the pool handed the loop a dead socket; the request waited for the full timeout, the server never saw it, and the client surfaced the failure as a 503. The provider dashboard showed nothing because the requests never arrived anywhere, which is why the failure looked random. It depended entirely on which socket the pool happened to hand out.

The Fix

The fix was a short-lived transport with a bounded keep-alive expiry plus a health probe before the batch, and it required no changes on the server side. The Python snippet below uses httpx with a custom transport that caps socket lifetime:

import httpx

class ShortLivedTransport(httpx.AsyncHTTPTransport):
    def __init__(self, *args, **kwargs):
        kwargs.setdefault("keepalive_expiry", 30)
        super().__init__(*args, **kwargs)

limits = httpx.Limits(max_keepalive_connections=5, max_connections=10)
transport = ShortLivedTransport(limits=limits)
client = httpx.AsyncClient(
    transport=transport,
    timeout=httpx.Timeout(15.0, connect=5.0),
)

The batch loop reused this client, and a probe ran before the first request to fail fast when the endpoint was unreachable:

curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"model":"<configured-model>","messages":[{"role":"user","content":"ping"}]}' \
  "$ENDPOINT"

Retries stayed in place, but with jitter and a per-attempt reason code in the log, so the next failure would show which layer actually rejected the request. The batch completed in under four minutes after the change, and the same client configuration has handled later runs without a single timeout.

The Reusable Checklist

• Reproduce outside the loop before touching any configuration.
• Compare client telemetry with server logs; a mismatch points at the transport layer.
• Check socket age and keep-alive expiry, not just status codes.
• Add a health probe to the deploy pipeline so dead endpoints fail fast.
• Log retry reason codes, not just retry counts.

Limitations

This routine assumes a single client talking to a shared server, so it will not help with multi-region latency tuning, prompt-level model failures, or security reviews. Teams with production SLAs should not build on a free server tier without an explicit fallback path, and workloads with sensitive data need a self-hosted or contractual option. The free model access, the current 10-million-token allowance, and the free server are accurate as of this writing, but terms change; verify the project's live documentation before committing a workflow to them.

The same isolation routine is worth running against any free-tier LLM setup, and MonkeyCode's free server is a convenient place to try it. The next time a 503 appears, check the socket before blaming the model.