A Free Model Endpoint Returned 200 With an Empty Body. The Fix Was a Lease, Not a Retry.

A realistic failure mode starts with a team that runs a nightly triage job on a small free server. The job groups stack traces from three services. It sends a compact prompt to a free model endpoint. The HTTP client sees a 200 response. The body is empty. The job marks the call as success. The next morning, two pages go to the wrong owner. Nobody knows the model returned nothing.

That is not a model failure. It is a contract failure.

MonkeyCode offers free model access and a free server option, which makes this class of bug cheap to test. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The case below is a reduced reproduction, not a production incident or a statement about any specific API contract.

The first fix made it worse

The first instinct was to retry. The team added one retry for any 200. That doubled the dead time. It also hid the poison case. A second call can also return an empty 200. Worse, it can return a different but still malformed body.

Retry is the wrong lever. Transport success and semantic success are different states. HTTP 200 only means a server responded. It does not mean the response can drive a routing decision.

The fix was a lease: a small local contract around one call. The lease has four parts.

1. A total timeout.
2. A response byte cap.
3. A schema guard.
4. A deterministic fallback.

The wrapper below uses libcurl and nlohmann/json. It is a reduced reproduction. The response shape is generic; adapt the schema check to the real endpoint contract.

#include <curl/curl.h>
#include <nlohmann/json.hpp>
#include <chrono>
#include <cstdlib>
#include <iostream>
#include <string>
#include <utility>

using json = nlohmann::json;

struct Lease {
    std::chrono::milliseconds connect_timeout{2500};
    std::chrono::milliseconds total_timeout{9000};
    size_t max_response_bytes = 4096;
};

size_t bounded_write(char *ptr, size_t size, size_t nmemb, void *userdata) {
    auto *state = static_cast<std::pair<std::string*, size_t>*>(userdata);
    size_t bytes = size * nmemb;
    if (!state || !state->first) return 0;
    if (state->first->size() + bytes > state->second) return 0;
    state->first->append(ptr, bytes);
    return bytes;
}

enum class Outcome { Accepted, Empty, SchemaViolation, TransportError };

Outcome parse_response(const std::string& body) {
    if (body.empty()) return Outcome::Empty;

    try {
        auto doc = json::parse(body);
        if (!doc.contains("choices") || !doc["choices"].is_array() || doc["choices"].empty()) {
            return Outcome::SchemaViolation;
        }

        const auto& first = doc["choices"][0];
        if (!first.contains("message") || !first["message"].is_object()) {
            return Outcome::SchemaViolation;
        }

        const auto& message = first["message"];
        if (!message.contains("content")) return Outcome::SchemaViolation;

        const auto content = message["content"].get<std::string>();
        if (content.empty()) return Outcome::Empty;

        return Outcome::Accepted;
    } catch (const json::exception&) {
        return Outcome::SchemaViolation;
    }
}

Outcome query_endpoint(const std::string& url,
                       const std::string& prompt,
                       const Lease& lease,
                       std::string& out_content) {
    CURL* curl = curl_easy_init();
    if (!curl) return Outcome::TransportError;

    json req = {{"prompt", prompt}, {"max_tokens", 128}};
    std::string req_body = req.dump();
    std::string response;
    auto state = std::make_pair(&response, lease.max_response_bytes);

    struct curl_slist* headers = nullptr;
    headers = curl_slist_append(headers, "Content-Type: application/json");

    curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
    curl_easy_setopt(curl, CURLOPT_POST, 1L);
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, req_body.c_str());
    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, bounded_write);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, &state);
    curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT_MS, static_cast<long>(lease.connect_timeout.count()));
    curl_easy_setopt(curl, CURLOPT_TIMEOUT_MS, static_cast<long>(lease.total_timeout.count()));
    curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L);

    CURLcode res = curl_easy_perform(curl);
    long http_code = 0;
    if (res == CURLE_OK) {
        curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code);
    }

    curl_slist_free_all(headers);
    curl_easy_cleanup(curl);

    if (res != CURLE_OK) return Outcome::TransportError;
    if (http_code < 200 || http_code >= 300) return Outcome::TransportError;

    Outcome result = parse_response(response);
    if (result == Outcome::Accepted) {
        auto doc = json::parse(response);
        out_content = doc["choices"][0]["message"]["content"].get<std::string>();
    }

    return result;
}

int main(int argc, char** argv) {
    if (argc < 2) return 2;
    std::string prompt = argv[1];

    const char* url = std::getenv("FREE_MODEL_URL");
    if (!url) return 2;

    Lease lease;
    std::string content;
    Outcome result = query_endpoint(url, prompt, lease, content);

    switch (result) {
        case Outcome::Accepted:
            std::cout << content << std::endl;
            return 0;
        case Outcome::Empty:
            std::cerr << "lease outcome: empty 200" << std::endl;
            std::cout << "severity=unknown; owner=oncall" << std::endl;
            return 1;
        case Outcome::SchemaViolation:
            std::cerr << "lease outcome: schema violation" << std::endl;
            std::cout << "severity=unknown; owner=oncall" << std::endl;
            return 1;
        case Outcome::TransportError:
            std::cerr << "lease outcome: transport error" << std::endl;
            std::cout << "severity=unknown; owner=oncall" << std::endl;
            return 1;
    }

    return 1;
}

The bounded writer is the important detail. A free endpoint can sit behind a proxy. It can send a redirect. It can return a large body. The byte cap stops the local process from holding an unbounded response. The timeout caps the wall time. The schema guard separates I got JSON from I got a usable answer.

The outcomes are a decision table

Outcome	Evidence	Local action
Accepted	non-empty choices[0].message.content	pass to routing
Empty	HTTP 200, empty body or empty content	log, use fallback, no retry
SchemaViolation	invalid JSON or missing fields	log, use fallback, no retry
TransportError	timeout, DNS failure, non-2xx, byte overflow	log, use fallback, no retry

The no-retry rule matters. A free endpoint can be slow or degraded. Retries can turn one bad call into three bad calls. They also make latency hard to predict. For an idempotent one-shot triage job, a single call with a fallback is better than multiple calls with hope.

Free server constraints are part of the lease

A free server often has a small memory and CPU allowance. The process should not fork a retry loop. The shell wrapper keeps the outer boundary simple.

#!/usr/bin/env bash
set -euo pipefail

ulimit -v 262144
timeout 20s ./triage_client "$1" > result.txt 2> error.log
status=$?

if [ "$status" -ne 0 ]; then
  echo "severity=unknown; owner=oncall" > result.txt
fi

This wrapper does not guess why the call failed. It only guarantees that the local process exits and leaves a safe fallback file. The C++ layer and the shell layer are two independent brakes. If one misses a hang, the other can still stop the job.

Limitations

The schema guard checks shape, not correctness. A confident wrong owner in valid JSON will pass. The fallback can hide chronic failure if nobody watches the logs. Add a counter for each Outcome value. Alert when Empty, SchemaViolation, or TransportError dominates. The wrapper is built for a routing suggestion, not for generated code or user-facing answers. It assumes a response shape with choices[0].message.content; adapt it to the actual endpoint.

Who should not use this approach

Do not use it for high-risk decisions. Do not use it when a wrong label creates legal, safety, or financial exposure. Do not use it when the team cannot define a safe fallback. Do not use it when latency requirements are tighter than the lease can support. A free model endpoint is an input, not a service-level agreement.

If a free model endpoint already feeds an internal job, copy the lease before adding a retry. The empty 200 bug was not solved by better prompting or a larger model. It was solved by making the client expect less and check more. The free tier can still be useful. The local contract is the part that makes it safe.

Comments

[Marcus Kim]

Separating an empty 200 from a missing choices[0].message.content makes the failure visible at the point where the routing decision is made. The 4 KB response cap, nine-second total timeout, and severity=unknown; owner=oncall fallback give this small nightly job a bounded failure mode instead of multiplying uncertainty through retries. I'd also track outcome rates per endpoint and trip a temporary circuit breaker when failures cluster; deterministic fallback protects the current run, but without that operational signal it can quietly normalize a provider contract that has been broken for days.

[Alex Shev]

A lease is the right shape when the problem is ambiguous ownership, not transient failure. Retries can hide the symptom while making the state machine less honest.