The API Said 200. The Body Said 'Rate Limited'.

I ran a batch of 50 prompts through a client I had just finished polishing, pointed at a free model server — the free server option in MonkeyCode — and the job summary looked perfect: 50 requests, 50 HTTP 200s, zero timeouts, zero 429s. Then I opened the output file and found that 14 of those responses were error objects wearing a 200 status code. My client checked response.status_code == 200 and moved on, so the errors flowed downstream as if they were completions, and my carefully tuned retry logic never had a chance to fire.

That is the failure mode I want to dissect today, because it is common on free and queued model endpoints, and because the fix is not about retrying harder. It is about deciding what success actually means before you write the next line of client code. The free server was just the stage; the real bug was my definition of a successful response.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The symptom that looked like a model problem

The first thing I noticed was that the failures were not random. Requests one through twenty came back clean, then results started arriving with empty text fields, and a few responses were missing their choices arrays entirely. My first instinct was to blame the model, and I almost rewrote the prompt template before I thought to dump the raw response bodies.

Here is what one of those "successful" responses actually contained:

{
  "id": "req_9f2c1a",
  "object": "chat.completion",
  "error": {
    "type": "rate_limit_exceeded",
    "message": "Too many requests. Try again in 12 seconds."
  }
}

Notice what is missing: there is no choices field, no usage field, and no HTTP 429 anywhere in sight. The transport layer accepted the request and returned 200, while the application layer reported a rate limit inside the body. My code never looked past the status code, so it parsed the JSON, found no choices, and returned an empty result to the caller.

Why my retry logic stayed silent

My retry logic was built around the assumption that the server would tell me when it was overloaded. I had a decorator that retried on requests.exceptions.Timeout and on response.status_code == 429, with exponential backoff and jitter, and it worked beautifully in local tests against a mock server. The mock server was the problem, in hindsight, because it was too well-behaved to expose the real failure.

It returned 429 the way the textbooks describe, so my tests proved that my retry logic handled 429s correctly. They never proved that my client could detect an error arriving inside a 200 response, because I had never taught it to look there. How could a rate limit slip through a client that was explicitly built to handle rate limits?

# The bug, simplified
def complete(prompt: str) -> str:
    response = httpx.post(API_URL, json={'prompt': prompt}, timeout=30)
    if response.status_code != 200:
        raise TransientError(f'HTTP {response.status_code}')
    data = response.json()
    return data['choices'][0]['text']  # KeyError when the body is an error

The KeyError was the giveaway in hindsight. A rate limit did not raise a retryable exception; it raised a KeyError that my batch script swallowed and logged as a failed generation. The retry decorator never saw the exception because the exception happened after the decorator had already decided the request was a success.

The fix: validate the envelope, then classify the error

Step one: validate the envelope before touching fields

The first change was to stop treating the HTTP status code as the single source of truth. For JSON APIs, the body is the contract, and the status code is just a transport hint, so I added an envelope check that runs before any field access. It felt like overkill at the time, and it turned out to be the cheapest insurance I wrote that week.

def parse_completion(data: dict) -> str:
    if 'error' in data:
        err = data['error']
        if err['type'] == 'rate_limit_exceeded':
            raise RateLimitError(err['message'])
        raise ModelError(err['message'])
    if 'choices' not in data or not data['choices']:
        raise MalformedResponseError('No choices in response')
    return data['choices'][0]['text']

Step two: classify errors with a visible decision table

The second change was to classify errors explicitly instead of guessing from the status code. I built a small decision table that maps error types to actions, and it made the retry behavior visible in one place. Now the question "should I retry this?" has a single answer that does not depend on my mood at 2 a.m.

Error type	Retryable?	Action
rate_limit_exceeded	Yes	Exponential backoff; honor Retry-After if present
overloaded / server_busy	Yes	Backoff with jitter and a longer cap
invalid_request	No	Fail fast and log the payload
authentication_error	No	Fail fast and alert a human
Missing choices	No	Fail fast and treat it as a client or server bug

Step three: write a test that simulates the sneaky 200

The third change was the one that actually caught this bug class in the future. I added a test that simulates a 200 response with an error body, using a tiny mock server, so the client now has to prove it can detect the failure. That test is the artifact I would hand to anyone who tells me their client handles rate limits correctly.

The test that would have caught it in five seconds

Here is the minimal mock server I used to reproduce the original bug and verify the fix. It returns a 200 with a rate_limit_exceeded envelope for every request, which is exactly the case my old client silently mis-handled.

from http.server import BaseHTTPRequestHandler, HTTPServer
import json

class SneakyHandler(BaseHTTPRequestHandler):
    def do_POST(self):
        body = json.dumps({
            'error': {
                'type': 'rate_limit_exceeded',
                'message': 'Too many requests. Try again in 12 seconds.'
            }
        }).encode()
        self.send_response(200)  # The trap
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass  # Keep the test output clean

server = HTTPServer(('127.0.0.1', 8765), SneakyHandler)
server.serve_forever()

Then the test asserts that the client raises a retryable error instead of returning an empty completion:

def test_error_envelope_inside_200_is_detected():
    client = CompletionClient(base_url='http://127.0.0.1:8765')
    with pytest.raises(RateLimitError):
        client.complete('hello')

That single test would have caught the original bug in about five seconds, and it cost me far less time than the hour I spent staring at output files. It also made the fix durable, because anyone who breaks the envelope check later will watch the suite go red. That is the difference between fixing a bug and fixing the process that let the bug exist.

What I changed about my debugging process

The deeper lesson was not about this one endpoint. I now treat every HTTP integration as having two layers that both need validation: the transport layer, where status codes and timeouts live, and the application layer, where the actual contract lives.

• Dump raw response bodies when results look wrong, before blaming the model.
• Validate the response envelope before accessing fields, and raise typed exceptions.
• Classify errors as retryable or not, and keep the decision table visible.
• Write a test for every failure mode you can imagine, including the rude ones.

Limitations and who should not use this approach

This approach assumes the API returns JSON error envelopes, which is common but not universal, and it assumes you can distinguish error types from the body. If the server returns plain-text errors or ambiguous messages, the classification table needs to be adapted. Retrying on rate_limit_exceeded only helps if the server actually clears the limit; on an aggressively throttled free tier, the honest answer may be to slow the whole batch down or queue requests locally instead of hammering the endpoint.

And if you are building a quick internal script where a failed generation is acceptable, you probably do not need this machinery. But the moment your pipeline treats empty results as successful completions, you are one silent error away from training a report on garbage, and that is a debugging session nobody enjoys. If you want to reproduce this class of bug against a real free endpoint, MonkeyCode's free model access is a convenient place to try — just bring your own envelope validation.