Three Nights of Empty LLM Responses — and the Isolation Trick That Finally Caught the Culprit

After three nights of watching my eval harness log empty responses that the API insisted were successful, I was ready to blame the model. The model was innocent. The real culprit turned out to be a stale TCP connection, a VPN proxy with a short patience, and one overly broad except Exception that quietly converted a crash into a blank string. This post walks through the symptom, the wrong turns, and the isolation trick that found the root cause in about twenty minutes.

The symptom that made no sense

Every night at 2 AM, a batch job sends a few hundred prompts to an LLM endpoint and scores the replies. Around seven percent of them came back as empty strings, always clustered in the middle of the run, never at the start. The HTTP status was 200, the JSON parsed cleanly, and the choices array was simply empty. My first instinct was to blame the model, because an empty success is the most confusing failure mode an API can produce.

The wrong turns I took first

I swapped prompts, shortened them, and padded max_tokens to give the model more room. Then I blamed rate limits and added retries with exponential backoff, which made the failure rate worse because every retry reused the same poisoned connection. I even suspected my scoring code, but the raw logs showed the empty choices array before any scoring logic ran. That was the moment I realized I had been guessing instead of isolating.

The isolation trick

Step one was building a minimal repro that stripped away every layer of my harness. No database, no scoring, no retries — just a loop that posts one prompt and prints the raw response shape.

import httpx

client = httpx.Client(timeout=30.0)
for i in range(20):
    r = client.post(
        "https://api.example.com/v1/chat/completions",
        json={
            "model": "some-model",
            "messages": [{"role": "user", "content": "Say hello in one sentence."}],
            "max_tokens": 20,
        },
    )
    body = r.json()
    print(i, r.status_code, body.get("choices"))

Step two was changing exactly one variable at a time. Same script, same prompt, same model: from my laptop with the VPN on, it failed roughly one time in fifteen. From a colleague's machine on a completely different network, it passed every single time. That pointed hard at my network path, but I still needed a second independent environment to be sure the model endpoint was not somehow involved.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. That context matters here because MonkeyCode's free model access and free server option gave me precisely that second environment — a different endpoint to rule out model-side behavior, and a separate server to rule out my laptop's network stack. A few hundred requests from that clean box produced zero empty responses, which told me exactly where to keep digging.

The actual root cause

The proxy in front of my VPN kills idle keep-alive connections after about sixty seconds. My HTTP client kept a pooled socket open between requests, and when the proxy silently closed it, the next request rode a dead connection into an error page that the gateway returned as a 200 with an empty choices array. My harness then indexed into body["choices"][0], raised IndexError, and the broad except Exception turned that crash into text = "". Every retry reused the same dead socket, which is why backoff made things worse instead of better.

The fix, in three small changes

# Before: one shared client, broad except, silent empty string
for prompt in prompts:
    try:
        body = client.post(...).json()
        text = body["choices"][0]["message"]["content"]
    except Exception:
        text = ""

# After: fresh connection, narrow except, loud failure
for prompt in prompts:
    with httpx.Client(timeout=30.0, headers={"Connection": "close"}) as c:
        resp = c.post(...)
        resp.raise_for_status()
        body = resp.json()
    if not body.get("choices"):
        raise RuntimeError("empty choices array — do not swallow this")
    text = body["choices"][0]["message"]["content"]

First, narrow the exception handling so a malformed response crashes the run instead of becoming a silent empty string. Second, validate the response shape before you parse it, because choices can be present but empty. Third, stop reusing pooled connections in long batch jobs, or set an idle timeout shorter than whatever proxy sits in front of you.

The reusable checklist

1. Reproduce with the smallest script that still triggers the failure, and print the raw response shape, not just the text.
2. Change one variable at a time — network, endpoint, client, prompt — and keep a log of which combination fails.
3. Move the repro to a second environment, ideally one with a different network path, to separate transport problems from model problems.
4. Log status codes and response bodies at the boundary of your code, before any parsing or scoring can hide the evidence.
5. Never catch broad exceptions around parsing; let a malformed response fail loudly so you cannot miss it.

Who should not copy this approach

If your failures are about output quality — wrong answers, hallucinations, format drift — a connection fix will not help you, and this checklist will waste your evening. A clean scratch server is also the wrong place to benchmark latency or throughput, because your real users will never share that network path. Use the free tier for isolation experiments, not for production assumptions, and treat every environment as a suspect until proven otherwise.

So the next time your LLM pipeline returns something that looks like a model problem, ask yourself a rude question: did I actually see the raw response, or did my client swallow the evidence? That question saved me three more nights, and I would honestly love to hear what your empty-response culprit turned out to be.