Treat an Empty 200 as a Protocol Violation Before You Debug the Model

Last week I moved a small agent evaluation harness onto a free server and pointed it at a free model endpoint, expecting a cheap way to validate a design. The harness stalled at turn 14 of 20, and the log showed something infuriating: HTTP 200, a normal latency, and a state machine that refused to advance. I spent an hour blaming the model before I checked what the client actually received, and the response was a 200 with an empty content field. The model was never the problem; my client-side assumption was.

I ran the harness on MonkeyCode's free server option with its free model access, because the point was to measure a design without paying for compute. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The failure I hit is provider-agnostic, and it will bite you against any endpoint that sits behind a proxy.

The Symptom: A Turn That Vanished

My harness was a simple loop with four steps:

1. Append the user message to the transcript.
2. Call the model endpoint and read the response.
3. Extract the assistant content and append it to the transcript.
4. Advance the cursor and repeat.

At turn 14, the loop logged a 200, extracted nothing, and moved on, so the transcript gained a gap where an assistant turn should have been. The evaluation metrics were quietly wrong, and I had no idea why. What do you do when the status code says success and the payload says nothing?

The Wrong Hypotheses

I tried three explanations before finding the real one. Hypothesis one was context truncation, which collapsed when I checked the request size and found it tiny. Hypothesis two was rate limiting disguised as a 200, which collapsed when the next turn succeeded instantly. Hypothesis three was the dangerous one, because it stopped me from looking at the transport: the model simply returned nothing.

The reusable technique here is to classify the response before you judge the model, separating transport validation from content validation. That one habit forces you to ask what the client actually received instead of what the model meant to say. It is the difference between debugging the network and debugging a ghost.

The Root Cause: Connection Reuse

The client kept a keep-alive connection open, and the proxy on the other side closed it after an idle timeout. The next request was written to a half-open connection, and the gateway's error path answered with a bare 200 and no payload.

Client            Proxy
  |-- keep-alive connection ------>|
  |   (idle timeout, proxy closes) |
  |-- next request on dead socket->|
  |<-- 200 OK, empty body ---------|

The exact mechanism varies by proxy, and that is the point: you cannot tell from the status code alone. My state machine treated any 200 as a valid turn and advanced the cursor, so the empty turn was silently consumed. The model never saw the turn, the transcript gained a gap, and every downstream metric inherited the error. The scary part is how normal the logs looked, because a 200 with no body is easy to miss when you scan for red status codes.

The Invariant

An empty 200 is a protocol violation, not a valid model turn. Declared assumptions: the endpoint returns JSON with a content field; a valid turn must have non-empty content; the harness advances the cursor only on a valid turn; network failures can surface as 200s. The fix is to classify every response into one of four classes: valid, empty_200, throttled, failed. Only valid advances the cursor, while empty_200 triggers a fresh connection and a bounded retry.

If retries exceed three, the harness fails loudly instead of corrupting the transcript. A silent gap in an evaluation run is worse than a failed run, because a failed run forces a decision. That asymmetry is the whole argument for the invariant.

The Reproducer

Here is the minimal client I used to confirm the failure class, and it works against any OpenAI-compatible endpoint. It reuses a session, classifies every response, and reconnects on an empty 200. The buggy version is the one-liner I started with: if status == 200: extract content, with a silent else that drops the turn.

import json
import time
import requests

ENDPOINT = "https://your-free-endpoint.example/v1/chat/completions"
API_KEY = "replace-me"

def classify(status, body):
    if status != 200:
        return "throttled" if status == 429 else "failed"
    try:
        payload = json.loads(body)
        content = payload["choices"][0]["message"]["content"]
    except (KeyError, IndexError, json.JSONDecodeError):
        return "malformed"
    if not content or not content.strip():
        return "empty_200"
    return "valid"

def call(session, messages):
    resp = session.post(
        ENDPOINT,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"model": "any", "messages": messages},
        timeout=30,
    )
    return resp.status_code, resp.text

messages = [{"role": "user", "content": "Count from 1 to 5."}]
session = requests.Session()

for turn in range(20):
    status, body = call(session, messages)
    cls = classify(status, body)
    print(f"turn={turn} status={status} class={cls}")
    if cls == "empty_200":
        print("stall detected; closing the half-open connection")
        session.close()
        session = requests.Session()
        continue
    if cls != "valid":
        raise SystemExit(f"unexpected class: {cls}")
    content = json.loads(body)["choices"][0]["message"]["content"]
    messages.append({"role": "assistant", "content": content})
    time.sleep(0.2)

The fixed version names the failure class, reconnects, and fails loud after the retry budget. The exact turn where the stall appears depends on the proxy's idle timeout, not on the model, so do not expect a stable reproduction point. Run it against a free endpoint and watch how often the empty_200 class shows up.

The Decision Table

Once you classify responses, the action table writes itself.

Observed response	Class	Action
200 with non-empty content	valid	advance the cursor
200 with empty or whitespace content	empty_200	reconnect, retry up to 3, then abort
200 with choices: []	empty_200	reconnect, retry up to 3, then abort
429	throttled	backoff and retry
5xx or timeout	failed	retry with backoff, abort after N

Notice that empty_200 and malformed share the same recovery path, because both mean the transport lied to you.

Failure Analysis and Tradeoffs

Ignoring this failure class costs you silent gaps in transcripts, skewed evaluation metrics, and wasted tokens on the next turn because the model never saw the previous one. The tradeoff table is small, and the cheapest option is the one most teams skip.

Choice	Cost	Benefit
Trust any 200	zero checks, lowest latency	risks silent stalls
Validate content	one JSON parse per turn	catches empty_200
Reconnect on empty_200	one TCP/TLS handshake	avoids half-open reuse
Fail loud after retries	run aborts	no silent metric corruption

The JSON parse is nanoseconds, the reconnect is milliseconds, and the abort is a decision you make on purpose. That is a good trade.

How to Validate This Yourself

1. Run the reproducer against any OpenAI-compatible endpoint and watch for the empty_200 class.
2. Add the classification to your real harness and run it twenty times, counting stalls before and after.
3. Apply the acceptance rule: zero unclassified stalls and zero silent gaps in the transcript.

You do not need my endpoint to reproduce this, because any proxy can return a bare 200.

pip install requests
python reproducer.py

The free server option is a convenient place to leave the harness running without watching a cloud bill. If a different failure class shows up first in your logs, I would genuinely like to know, because that is how these invariants get sharpened.

Limitations

This guard is transport-level, not a model-quality fix, and it will not help if the model genuinely returns empty content for a valid prompt. If you are writing a one-shot script that keeps no state, the classification is overkill. If you are on a paid endpoint with strict SLAs, empty_200 is rarer, but the check still costs almost nothing. I am also not quoting token quotas or server specs here, because those numbers change and stale data is worse than no data.

Check the current docs before you plan capacity, and treat any blog number as a snapshot, not a contract.

The Counterexample

Which event order breaks the invariant now: an empty 200 after a reconnect, or a 200 with whitespace-only content that slips past a naive if content check? Should your harness reject the turn, replay it on a fresh connection, or compensate by marking the transcript gap? The answer depends on your workload, but the decision has to be explicit, because the default is silent corruption.