No Response Is a Response: Five Silent-Drop Myths on Free Model Servers

Some failures never raise an error. Your client waits. The deadline passes. You blame the model server. But what if the request never arrived? What if it arrived and then vanished? That is a silent drop. Retry counters will not catch it. Logs usually won't either.

The failure no status code describes

An HTTP error is honest. 429 says back off. 503 says try later. 500 says something broke. A silent drop says nothing. You get no status code. You get no body. You get a timeout that you invented yourself. That distinction matters.

A timeout is a client-side decision. A status code is a server-side statement. Confusing them is where the myths start.

What I mean by silent drop

A silent drop is a request that disappears between your process and the model. The server may never see it. The server may see it and then lose it. Either way, your application sees nothing. No answer. No code. No trace.

Free-tier endpoints hide these drops well. The symptom looks exactly like slowness. That's why retry loops and timeout tweaks don't fix it. You are measuring the wrong layer.

To measure this properly, you need an endpoint that accepts arbitrary JSON. I use MonkeyCode's free model access and its free server option as a test target. That gives me a real model endpoint and a server I can restart. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The probe

Stop guessing. Build a probe. The probe sends requests with a unique request ID. Then it compares client results against server logs. The gap between those two is your drop rate.

You can test locally first. No cloud account required. Two files. Five minutes.

Part 1 - A drop server

This small server simulates four failure modes. echo is the healthy control. hold/N delays the answer past the client deadline. cut accepts the request, then closes the connection without answering. phantom drops the request before the app ever records it.

Run it in terminal one:

# drop_server.py
import asyncio, json

SEEN = []

def http_response(body: bytes, status: str = "200 OK") -> bytes:
    return (
        f"HTTP/1.1 {status}\r\n"
        f"Content-Length: {len(body)}\r\n"
        f"Connection: close\r\n\r\n"
    ).encode() + body

async def handle(reader, writer):
    try:
        header_bytes = await asyncio.wait_for(
            reader.readuntil(b"\r\n\r\n"), timeout=3
        )
        headers = header_bytes.decode(errors="replace").split("\r\n")
        request_line = headers[0] if headers else ""
        parts = request_line.split(" ")
        path = parts[1] if len(parts) > 1 else "/"

        request_id = None
        for line in headers:
            if line.lower().startswith("x-request-id:"):
                request_id = line.split(":", 1)[1].strip()

        # Simulate drops that the app never sees.
        if path.startswith("/phantom"):
            await asyncio.sleep(0.2)
            writer.close()
            return

        if request_id:
            SEEN.append({"request_id": request_id, "path": path})

        if path.startswith("/hold/"):
            seconds = float(path.split("/")[-1])
            await asyncio.sleep(seconds)
            data = json.dumps({"ok": True}).encode()
            writer.write(http_response(data))
            await writer.drain()
        elif path == "/cut":
            # Claim receipt, then close without replying.
            await asyncio.sleep(0.1)
            writer.close()
            return
        elif path == "/echo":
            data = json.dumps({"ok": True}).encode()
            writer.write(http_response(data))
            await writer.drain()
        elif path == "/logs":
            data = json.dumps(SEEN).encode()
            writer.write(http_response(data))
            await writer.drain()
        else:
            writer.write(http_response(b"{}", "404 Not Found"))
            await writer.drain()
    except Exception:
        pass
    finally:
        try:
            writer.close()
        except Exception:
            pass

async def main():
    server = await asyncio.start_server(handle, "127.0.0.1", 8001)
    print("drop server running on http://127.0.0.1:8001")
    async with server:
        await server.serve_forever()

if __name__ == "__main__":
    asyncio.run(main())

Part 2 - The client

The client sends N requests with a request ID. Then it asks the server which IDs arrived. The table shows the quiet ones. Run it in terminal two:

# probe.py
import json, time, uuid
import requests
from concurrent.futures import ThreadPoolExecutor

BASE = "http://127.0.0.1:8001"
PAYLOAD = {"model": "probe-model", "prompt": "ping"}

def send_one(request_id: str, path: str):
    headers = {"X-Request-ID": request_id, "Content-Type": "application/json"}
    started = time.monotonic()
    result = "ok"
    try:
        r = requests.post(f"{BASE}{path}", json=PAYLOAD, headers=headers, timeout=5)
        if r.status_code != 200:
            result = f"http_{r.status_code}"
    except requests.exceptions.Timeout:
        result = "timeout"
    except requests.exceptions.ConnectionError as exc:
        result = "reset" if "reset" in str(exc).lower() else "connection_error"
    elapsed = round(time.monotonic() - started, 2)
    return request_id, result, elapsed

def run_probe(path: str, count: int = 12, workers: int = 4):
    ids = [f"req-{i:03d}" for i in range(count)]
    with ThreadPoolExecutor(max_workers=workers) as pool:
        results = list(pool.map(lambda rid: send_one(rid, path), ids))
    seen = requests.get(f"{BASE}/logs", timeout=3).json()["seen"]
    seen_ids = {item["request_id"] for item in seen}
    print(f"{'request_id':<12} {'client':<14} {'elapsed':<8} {'server_log'}")
    for rid, result, elapsed in results:
        in_log = "yes" if rid in seen_ids else "no"
        print(f"{rid:<12} {result:<14} {elapsed:<8} {in_log}")

if __name__ == "__main__":
    print("--- echo: healthy control ---")
    run_probe("/echo")
    print("--- hold/10: slow beyond client deadline ---")
    run_probe("/hold/10")
    print("--- cut: arrived, then vanished ---")
    run_probe("/cut")
    print("--- phantom: dropped before the app ---")
    run_probe("/phantom")

Sample output shape (your timing will differ):

--- echo: healthy control ---
request_id   client         elapsed   server_log
req-000      ok             0.21      yes
req-001      ok             0.19      yes

--- hold/10: slow beyond client deadline ---
req-002      timeout        5.01      yes

--- cut: arrived, then vanished ---
req-003      reset          0.11      yes

--- phantom: dropped before the app ---
req-004      reset          0.20      no

How to read the results

client saw	server logged	meaning	move
ok	yes	healthy round trip	keep going
timeout	yes	arrived, response lost	raise read timeout or stream
timeout/reset	no	dropped before the app	recreate connection, retry with ID
reset	yes	connection killed mid-flight	check body size and keep-alive
4xx/5xx	maybe	explicit error	handle status codes normally

Do not count all timeouts as drops. A timeout with a server log is a slow response. A timeout without a log is a true silent drop. The distinction changes your fix. Slow means wait longer. Dropped means resend. The request ID makes that choice safe.

Five myths, corrected

Myth 1: A timeout means the server is thinking

Timeouts fire before inference even starts. DNS can stall. TCP can stall. A proxy can close an idle connection. Your timeout is a measurement, not a diagnosis. It cannot see inside the network path. Treat it as a symptom, not proof.

Myth 2: An empty server log means a clean bill of health

Some servers log only completed work. Some gateways drop before the app sees the request. Log sampling hides rare events. Absence of evidence is not evidence of absence. That is why this probe checks the server side explicitly.

Myth 3: Retry after timeout is always safe

Is it safe when the first request arrived? No. You create a duplicate. The server may process both. Request IDs put a pin in that problem. Without an ID, a retry is a guess. With one, it is an idempotent replay.

Myth 4: Drops on free tiers are random noise

Free-tier traffic is not uniform. Drops cluster around idle connections. They cluster around large prompts and long generations. Classify your samples before calling them random. Patterns show up once you look.

Myth 5: No error means no problem

The quiet failures are the expensive ones. One drop is noise. Five percent is a user-visible bug. If your dashboard only watches 4xx and 5xx, it is blind. Add the no-response category to your monitoring.

When this probe is not enough

This probe is a spot check, not a full observability stack. It misses trace-level issues and slow cascading failures. For high throughput, use distributed tracing. For security boundaries, use a proper gateway. And remember: the probe creates its own load. Watch for new 429s. They are data, not bugs. If you cannot add a request ID to the request, this method falls apart.

Run it

Start the drop server. Point the probe at your endpoint. Let it run for one hour. Then read the table with fresh eyes. The empty rows tell more than the error codes. I run this before trusting any new free-tier endpoint. It takes five minutes and saves a day of confusion.