Your 503 Is Not an Outage: Four Error-Code Myths on Free Model Servers

A status code is not a verdict. It's a data point.

Your endpoint just returned 503. Is the machine dead? Is the queue full? Did a cold container eat your request?

Most error-handling code maps codes to blame. Free-tier servers don't cooperate with that. One code can mean three different things, depending on when and how you probe it.

Here are four myths I keep hearing. Each one costs queue time. Each one has a cheap test.

Myth 1: 429 always means "you sent too many requests"

429 can mean several things on a shared free tier:

• A per-key rate limit.
• A quota that's already spent.
• Admission control while the host is busy.

The test: send one request. Wait five seconds. Send one request again. If a single lonely request still gets 429, you are not the problem. The tier is.

Check the Retry-After header too. If it's present, the server is telling you exactly when to retry. Respect it. If it's absent, you need more evidence before assigning blame.

Myth 2: 503 means the server is down

RFC 9110 defines 503 as "Service Unavailable." That's a temporary state, not a tombstone.

Free servers spin containers up and down. Cold starts produce 503s. Rolling deploys produce 503s. Capacity drains produce 503s. None of those mean the endpoint is gone.

The test: retry after five seconds, then after thirty. If a retry lands fast, the 503 was a transient moment. If every node returns 503 for a full minute, that's a different investigation.

Log the elapsed time of the 503 itself. A fast 503 and a slow 503 tell different stories.

Myth 3: 504 means my network is broken

504 is a Gateway Timeout. An upstream didn't answer in time.

On a model server, that upstream is usually the inference process. Your laptop can have perfect connectivity while the model is still generating. Long prompts make this worse. Long generations make this worse.

The test: send the same prompt with a shorter payload. If small prompts pass and large ones hit 504, the bottleneck is generation time, not your internet. Shrink the payload. Tune the gateway window. Split the work. Don't restart your router.

Myth 4: a timeout is just another 5xx

This one is dangerous.

A client timeout is not an error. It's an unknown. The model may have finished. The bytes may be stranded in a closed connection. You can't tell from the client side.

Retrying a timed-out inference is usually harmless. Retrying a timed-out write is not. If work can be duplicated, you need an idempotency key or a lookup path. Not a blind retry loop.

The test: capture the request ID on every attempt. Then you can ask "did that request complete?" instead of guessing. Guesswork is not an error-handling strategy.

The probe

Here's a classification probe you can point at any free-tier endpoint. Standard library only.

#!/usr/bin/env python3
"""classify.py - label free-tier model server outcomes."""
import json
import sys
import time
import urllib.error
import urllib.request

def probe(url, payload, timeout=30.0):
    started = time.monotonic()
    req = urllib.request.Request(
        url,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            return {
                "outcome": "success",
                "status": resp.status,
                "elapsed_s": round(time.monotonic() - started, 3),
                "retry_after": resp.headers.get("Retry-After"),
                "request_id": resp.headers.get("X-Request-Id"),
            }
    except urllib.error.HTTPError as exc:
        return {
            "outcome": "http_error",
            "status": exc.code,
            "elapsed_s": round(time.monotonic() - started, 3),
            "body": exc.read(200).decode(errors="replace"),
            "retry_after": exc.headers.get("Retry-After"),
            "request_id": exc.headers.get("X-Request-Id"),
        }
    except urllib.error.URLError as exc:
        if isinstance(exc.reason, TimeoutError):
            return {"outcome": "timeout", "elapsed_s": timeout}
        return {"outcome": "network_error", "reason": str(exc.reason)}
    except TimeoutError:
        return {"outcome": "timeout", "elapsed_s": timeout}

Run it like this:

python3 classify.py https://your-endpoint.example/v1/complete

Then classify the result:

def classify(entry):
    if entry["outcome"] == "success":
        return "healthy"
    if entry["outcome"] == "http_error":
        if entry["status"] == 429:
            ra = entry.get("retry_after")
            if ra:
                return "rate_limit_or_quota: back off " + ra + "s"
            return "admission_control: probe again with a single request"
        if entry["status"] == 503:
            return "capacity_or_cold_start: retry in 5s, then 30s"
        if entry["status"] == 504:
            return "upstream_timeout: shrink payload, check gateway window"
    if entry["outcome"] == "timeout":
        return "unknown: log request_id, avoid blind retry"
    return "unclassified: " + entry.get("reason", "?")

I point this probe at whatever free endpoint I'm staging on. If you're trying the free server option or free model access from MonkeyCode, the same classification applies. The script doesn't care about the brand.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The one-hour drill

Classify any endpoint in under an hour:

1. Send ten probes, one at a time, five seconds apart.
2. Send ten probes concurrently.
3. Compare the buckets.

Keep the rate equal. Change the concurrency. If concurrency produces 429s that singles don't, you found a concurrency cap. If both batches fail the same way, you found a quota or capacity issue. That difference changes your whole retry strategy.

What to log

A status code alone is too thin. Capture at least five fields:

• status and outcome
• elapsed_s measured from the client
• retry_after if present
• request_id if present
• a short body excerpt

Store them as JSON Lines. After a few days, you can ask real questions. Does 503 cluster at the same hour? Does 429 follow your cron, or the platform's queue? Patterns beat vibes.

Decision table

Status	Likely meaning on a shared free tier	Retry?	First thing to check
429 + Retry-After	Rate limit or quota	After header value	Header value vs your pacing
429, no header	Admission control	No, re-probe once	Single vs concurrent results
503	Cold start / capacity	Yes, with backoff	Burst duration
504	Upstream too slow	Only when retry is safe	Payload size vs gateway window
Client timeout	Unknown, not an error	Only with idempotency	Request ID lookup

Who shouldn't use this

This classification is not a benchmark. It won't size a production fleet. It won't score model quality.

If you need an SLA, a free tier is the wrong home. That's true for any provider. And don't infer server topology from one sample. One 503 is a clue, not an architecture diagram.

Use the probe for one question only: "Which failure class am I in?" Answer that first. Then tune the retry. Point it at your own endpoint and let the buckets do the talking.