Five Anti-Patterns at the Model–Server Boundary (and a Preflight That Catches Them)

The core conclusion first: free models and free servers rarely fail alone. They fail at the seam between them.

One side retries. The other side times out. Nobody owns the counter. Sounds familiar?

I keep writing about free-tier footguns because the same shape keeps showing up. Not a broken model. Not a broken box. A broken contract between the two.

This post is a catalog. Five anti-patterns, each with symptoms, root cause, and a replacement pattern. Then a small stdlib script that fails loudly before your users do.

Why the seam, specifically?

A model call has a token budget. Your gateway has an idle deadline. Your job queue has a wall clock.

That's three clocks. If they are not ordered, you get ghost work: the request you cancelled but still paid for.

Free resources make this worse for one boring reason. You did not size them. You inherited them.

Anti-pattern 1 — The Timeout Hand-Me-Down

Symptoms

• Users see 504 while your logs show a completed completion.
• The same job row appears twice, once empty, once full.
• Retry storms every time the provider is slow, not down.

Root cause. Your client timeout is longer than your gateway timeout. The gateway gives up first. Your client keeps waiting, then retries into a queue that already did the work.

Or worse: nobody set a number at all, so every layer uses its default. Defaults are not a policy.

Replacement pattern. A strict ladder, in config, with an assertion.

"timeouts": { "client_s": 20, "gateway_s": 25, "job_s": 90 }

Client must be shortest. The job must outlive both, so a retry has somewhere to land. If you cannot state all three numbers, you do not have a budget.

Anti-pattern 2 — Retry Without a Key

Symptoms

• Two identical answers, two rows, one user.
• Your "dedupe" is a SELECT after the fact.
• Costs double only on the slow days, so nobody notices.

Root cause. The retry loop retries the transport, but the server sees a brand-new request. Nothing links attempt one to attempt two.

Replacement pattern. Derive a deterministic key from the request itself, then retry only on 429, 5xx, and timeouts. Twice, with jitter.

import hashlib, json

def key_for(prompt: str, params: dict) -> str:
    raw = json.dumps([prompt, params], sort_keys=True).encode()
    return hashlib.sha256(raw).hexdigest()[:32]

Same prompt plus same params equals same key. That is the whole trick. Pass it as a header and let the far side decide what a duplicate means.

Anti-pattern 3 — The Ceiling That Lives in a Spreadsheet

Symptoms

• "We have plenty left" is a sentence, not a number.
• The cap is real but only enforced by attention.
• One batch job eats the month in eleven minutes.

Root cause. The limit exists in a dashboard. The code path never asks.

Replacement pattern. Admit or reject before the call, in code, using a projected token count. Then log the admission.

def admit(estimate: int, spent: int, cap: int) -> None:
    if spent + estimate > cap:
        raise RuntimeError(f"cap: {spent}+{estimate} > {cap}")

A rejection you can read is better than a mystery 429 at 2 a.m. Log {key, estimate, spent, model_id} on every admitted call. Now the ledger and the code agree.

Anti-pattern 4 — The Key in the Client Bundle

Symptoms

• A long string in dist/app.js.
• A screenshot with a header visible.
• "It's only a free key, who cares."

Root cause. Proxying felt like extra work, so the browser talked to the model directly.

Replacement pattern. A thin server-side proxy with a fixed model allowlist and a per-IP rate limit. Then grep for leaks in CI:

grep -rInE "(sk-|api[_-]?key|Bearer )" --include="*.js" dist/ \
  | grep -v "process.env" && exit 1 || true

Free keys still cost trust. Rotate on suspicion, not on proof.

Anti-pattern 5 — Config Drift Between Two Dashboards

Symptoms

• Works on the laptop, 429s on the server.
• Two different model id strings, both "correct".
• Nobody can say which max_tokens is live.

Root cause. Two sources of truth: your shell env and the server env. They diverge on day three.

Replacement pattern. One committed boundary.json, loaded by both paths, printed at boot.

print(json.dumps(cfg, indent=2, sort_keys=True), file=sys.stderr)

If the boot log does not show your limits, your limits are folklore.

The artifact: boundary_preflight.py

Here is the checker I keep in the repo. Stdlib only. Four static assertions plus one timed probe.

Treat it as scaffolding, not a certified test suite. I have not run it against your deployment.

#!/usr/bin/env python3
"""Boundary preflight. stdlib only. Exit 1 on any FAIL.
Static checks read boundary.json. --live adds one timed probe."""
import json, os, sys, time, urllib.request

PROBE = "Reply with exactly: ok"


def load(path):
    with open(path, encoding="utf-8") as fh:
        return json.load(fh)


def check_timeout_ladder(cfg):
    t = cfg["timeouts"]
    ok = t["client_s"] < t["gateway_s"] < t["job_s"]
    return ok, f"client={t['client_s']} gateway={t['gateway_s']} job={t['job_s']}"


def check_retry_policy(cfg):
    r = cfg["retry"]
    ok = r["max_attempts"] <= 2 and r["idempotency_key"] is True
    return ok, f"max_attempts={r['max_attempts']} key={r['idempotency_key']}"


def check_cap(cfg):
    cap, where = cfg["cap"]["tokens"], cfg["cap"]["enforced_at"]
    ok = isinstance(cap, int) and cap > 0 and where == "client"
    return ok, f"cap={cap} enforced_at={where}"


def check_secret_surface(cfg):
    ok = cfg["secrets"]["in_client_bundle"] is False
    return ok, f"in_client_bundle={cfg['secrets']['in_client_bundle']}"


CHECKS = [
    ("timeout-ladder", check_timeout_ladder),
    ("retry-policy", check_retry_policy),
    ("token-cap", check_cap),
    ("secret-surface", check_secret_surface),
]


def probe(cfg):
    base = os.environ["MODEL_BASE_URL"].rstrip("/")
    payload = json.dumps({
        "model": cfg["model_id"],
        "messages": [{"role": "user", "content": PROBE}],
        "max_tokens": 8,
    }).encode("utf-8")
    req = urllib.request.Request(
        base + cfg.get("path", "/chat/completions"),  # adjust to your API
        data=payload, method="POST",
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + os.environ["MODEL_API_KEY"]})
    started = time.monotonic()
    with urllib.request.urlopen(req, timeout=cfg["timeouts"]["client_s"]) as res:
        json.loads(res.read())
    return time.monotonic() - started


def main():
    cfg = load("boundary.json")
    failed = 0
    for name, fn in CHECKS:
        ok, detail = fn(cfg)
        failed += 0 if ok else 1
        print(f"{'PASS' if ok else 'FAIL'}  {name:15s} {detail}")
    if "--live" in sys.argv:
        elapsed = probe(cfg)
        budget = cfg["timeouts"]["client_s"]
        ok = elapsed < budget * 0.5
        failed += 0 if ok else 1
        print(f"{'PASS' if ok else 'FAIL'}  {'live-probe':15s} "
              f"{elapsed:.2f}s of {budget}s budget")
    print(f"\n{failed} failing check(s)")
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

Run it:

export MODEL_BASE_URL="https://your-endpoint"
export MODEL_API_KEY="..."
python3 boundary_preflight.py boundary.json --live

Sample output from a config I deliberately broke:

PASS  timeout-ladder  client=20 gateway=25 job=90
FAIL  retry-policy    max_attempts=5 key=False
PASS  token-cap       cap=20000 enforced_at=client
PASS  secret-surface  in_client_bundle=False
FAIL  live-probe      9.84s of 20s budget

2 failing check(s)

Why budget * 0.5? Because a probe that uses half your client budget has no headroom for a real prompt. Slow p50 is a warning, not a pass.

Decision table: where should this workload live?

Workload	Laptop + free model	Free server + free model	Paid
One-off prompt exploration	Yes	Overkill	No
Internal tool, 5 users, tolerant of retries	Yes	Yes, with the preflight	No
Anything holding PII	No	Only with redaction in the proxy	Probably
Sub-second user-facing latency	No	No	Yes
Uptime promise in a contract	No	No	Yes
Overnight batch, resumable	Yes	Yes	Maybe

Read the table as a starting hypothesis. Your latency probe is the tiebreaker.

Who should not use this approach

• Teams with a data-processing agreement that names a specific region.
• Products where a 30-second tail latency is a bug report.
• Anyone who cannot redact prompts before they leave the box.
• Workloads that need a written SLA, not a best effort.

Free capacity is a fine place to prove a seam. It is a bad place to promise one.

Where this fits for me

MonkeyCode offers free model access and a free server option. That combination is exactly the setup this preflight was written for: two resources, one boundary, no invoice to blame.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I would not quote a token number or a hardware spec here. Quotas and machine shapes move, and a stale number in a tutorial is just a future bug report. Check the project's current docs before you size anything.

What I would do instead: stand up the free server, point MODEL_BASE_URL at the free model endpoint, and run the five checks on day one. Keep the ones that fail.

The takeaway

Free things do not fail loudly. They fail at the join.

Write the join down. Assert it in code. Print it at boot.

Which of the five would your preflight fail first?