The Free Endpoint Does Not Belong on the Interactive Path

Free capacity rarely fails by returning errors. It fails as a queue, and the queue surfaces as latency on a request a user is already waiting for. That asymmetry is the whole argument: MonkeyCode's free model access and free server option fit background lanes and batch jobs, and they fit badly behind a spinner in a product surface.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The offering is described by the operator as free model access with a 10 million token allowance, plus a free server option. Nothing below assumes a model name, a concurrency limit, hardware, or how long the offer lasts. Those are exactly the unknowns the harness is built to measure rather than guess.

Errors are the wrong signal

A saturated free endpoint still answers 200. It answers slowly. Your dashboard shows a clean error rate while your users watch a cursor blink, and the first thing that breaks is not the model call but the request timeout wrapped around it.

Then the retries arrive. An SDK retries twice, your service retries once, and a single slow call becomes four queued calls competing for the same capacity. Latency is self-inflicted from that point on, and the error rate you were watching finally moves — after the damage.

Interactive paths punish this behaviour twice. The user pays the latency directly, and the retry storm you generate lands on shared free capacity that other lanes also depend on. Background work absorbs the same queue as throughput; a person cannot.

Measure the thing that actually degrades

First-byte latency, not success rate, is the number that decides path placement. The harness below is deliberately endpoint-agnostic: it takes a full URL and an exact request body from the environment, so it measures whatever you actually intend to ship instead of an invented API shape. It is an unexecuted reference implementation — run it against your own endpoint before trusting any number in it.

# probe_free_endpoint.py -- endpoint-agnostic latency ledger
import json, os, time, urllib.request

URL = os.environ["PROBE_URL"]          # full URL, supplied by you
TOKEN = os.environ["PROBE_TOKEN"]
PAYLOAD = json.load(open(os.environ["PROBE_BODY"]))
LEDGER = os.environ.get("PROBE_LEDGER", "ledger.jsonl")
N = int(os.environ.get("PROBE_N", "20"))
TIMEOUT = float(os.environ.get("PROBE_TIMEOUT", "60"))

def one(i):
    req = urllib.request.Request(
        URL, data=json.dumps(PAYLOAD).encode(), method="POST",
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {TOKEN}"},
    )
    t0 = time.monotonic()
    first, nbytes = None, 0
    try:
        with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
            status = r.status
            while True:
                chunk = r.read(1024)
                if not chunk:
                    break
                if first is None:
                    first = time.monotonic() - t0
                nbytes += len(chunk)
    except Exception as e:
        return {"i": i, "ok": False, "err": type(e).__name__,
                "total_s": round(time.monotonic() - t0, 3)}
    return {"i": i, "ok": True, "status": status, "bytes": nbytes,
            "first_byte_s": round(first, 3) if first else None,
            "total_s": round(time.monotonic() - t0, 3)}

def pct(xs, p):
    xs = sorted(x for x in xs if x is not None)
    if not xs:
        return None
    k = max(0, min(len(xs) - 1, round((p / 100) * (len(xs) - 1))))
    return xs[k]

if __name__ == "__main__":
    rows = [one(i) for i in range(N)]
    with open(LEDGER, "a") as f:
        for r in rows:
            f.write(json.dumps(r) + "\n")
    ok = [r for r in rows if r["ok"]]
    print(json.dumps({
        "n": N,
        "ok_rate": round(len(ok) / N, 3),
        "p50_first_byte_s": pct([r.get("first_byte_s") for r in ok], 50),
        "p95_first_byte_s": pct([r.get("first_byte_s") for r in ok], 95),
        "p95_total_s": pct([r.get("total_s") for r in ok], 95),
    }, indent=2))

Run it the way you would run a smoke test, not a benchmark:

PROBE_URL="$FREE_URL" PROBE_TOKEN="$FREE_TOKEN" \
PROBE_BODY=./request.json PROBE_N=30 \
python probe_free_endpoint.py

Append-only JSONL is the point. A single run tells you almost nothing; thirty runs across a morning tell you whether the p95 moves when the rest of the world wakes up. The ledger is what makes the admission decision auditable six weeks later, when nobody remembers why the lane was split.

Admit only with headroom

A free lane should clear the path's latency objective with margin to spare, because you cannot buy your way out of a queue mid-incident. The rule below encodes that asymmetry: first byte must land inside half the budget, and total time inside eighty percent, before the lane is allowed anywhere near a user.

# admission.py -- unexecuted; bounds come from your own SLO, not from this article
def admits(stats, slo_first_byte, slo_total, min_ok_rate=0.99):
    if stats["ok_rate"] < min_ok_rate:
        return False, "ok_rate below floor"
    if (stats["p95_first_byte_s"] or 9e9) > 0.50 * slo_first_byte:
        return False, "first byte has no headroom"
    if (stats["p95_total_s"] or 9e9) > 0.80 * slo_total:
        return False, "total has no headroom"
    return True, "admitted with margin"

A lane that passes this rule is a candidate for interactive use. A lane that fails it is not slow, it is misrouted. Those are different bugs and they get different fixes.

Lane placement, decided once

Lane	Human waiting	Free capacity fits	Required fallback
Autocomplete, chat reply	Yes	Only with measured headroom	Paid or local endpoint
PR review comments	No, minutes	Yes	Requeue, then drop
Nightly summarisation	No, hours	Yes	Next window
Anything holding a lock	Depends	No	Unbounded path only
Write path with side effects	Yes	No	Idempotent local path

The fourth row is the trap. Free capacity inside a critical section holds a lock for a duration nobody controls, and every waiter inherits the queue.

Red flags, as prose

Five signals should stop a design review cold. The first is a spinner: if a person is watching the call resolve, free capacity needs measured headroom, not optimism. The second is a retry policy written before the endpoint was measured, because two layers of retries against a queue multiply rather than divide. The third is state: if the free server also holds a queue, a cron table, or a lock that the request path depends on, the model's latency and the scheduler's latency become one number you cannot separate. The fourth is privilege: a free endpoint should never be the thing that approves, merges, deletes, or pays. The fifth is opacity: if you cannot reproduce the same request body from a fixture file, you are not measuring the thing you ship.

Better alternatives exist for each red flag

For the spinner, move the call off the request path and stream a cached answer while the slow lane works. For the retry storm, cap attempts at one per layer and let the caller decide. For shared state, keep the free server stateless — every artefact reconstructed from your own durable store. For privilege, put a human or a deterministic rule at the boundary and let the model draft only. For opacity, the probe harness above is the fix, and it is cheaper than the incident review it prevents.

Exit criteria, written before adoption

Exit when the probe's p95 first-byte latency exceeds half the path's budget in three consecutive measurement windows. Exit when the fallback rate crosses the headroom you measured, because at that moment the free lane is no longer an optimisation but a dependency. Exit, most of all, when a single engineer would need to be paged to keep the lane alive — that is the definition of the free tier becoming load-bearing, and it is a staffing decision disguised as an infrastructure one. Record the exit date and the number that triggered it in the same ledger file.

Limitations, and who should skip this

The harness measures one client from one location. It says nothing about behaviour under your real concurrency, nothing about correctness of responses, and nothing about data handling, which is a review you must do separately before sending anything sensitive to any third-party endpoint. Latency measured at 09:00 may not describe 21:00.

Teams with a hard real-time contract, teams handling regulated data without a completed review, and teams whose on-call rotation cannot absorb a new dependency should not adopt a free lane at all. Neither should anyone who needs a duration guarantee; the operator describes free access and a free server, not a service level, and this article deliberately asserts no numbers beyond the stated allowance.

Start by measuring. Point the probe at the free model access endpoint and, separately, at whatever the free server option actually serves, then let the ledger rather than the announcement decide which lane each one earns.