Last month a green checkmark lied to me. An API call returned 200 OK. The model that answered was not the model I asked for. Nothing in my logs said so.
I now keep a provenance log for every endpoint I touch. It started as paranoia. It is now the first file I open when something looks wrong.
Below are five claims I used to repeat. Each one cost me real debugging time.
Myth 1: "A 200 OK means the model I asked for answered"
The status code describes the HTTP layer. It does not describe the model layer. Gateways accept a model string and route wherever they choose.
Ask yourself: did the response body echo my model id back? Even when it does, that field is a claim made by the server. It is not proof of anything.
So log both sides. Store requested_model and served_model in one record. When they diverge, you are looking at an alias, a fallback, or a silent routing change.
Myth 2: "Free model access and a free server are the same kind of free"
They are two different budgets, and they fail in different ways. One is an entitlement: how much inference you may spend. The other is compute: where your process actually runs.
- An entitlement usually fails loudly. You get
429,402, or a stream that stops mid-sentence. - A compute allocation fails quietly. The box sleeps, the disk fills, the port closes, the clock drifts.
Do not merge them into one mental model. Do not merge them into one retry policy either. One needs backoff. The other needs a health check.
Myth 3: "If it runs on my laptop, it runs on the free box"
Environment drift is not a vibe. It is a diff you failed to capture. My laptop has a different Python minor version and a different PATH order than any remote machine.
Record the boring facts with the request: interpreter version, working directory, and the env var names you depend on. Never log the values. Names are evidence. Secrets are liabilities.
python -c "import sys, platform; print(sys.version, platform.platform())"
node -v && uname -a && pwd
Run those on both sides and diff them. Two minutes, no guessing.
Myth 4: "Rate limits are a billing problem"
No. Rate limits are a control-flow problem. A 429 tells you your program's timing is wrong, not your wallet.
A retry loop without jitter creates a thundering herd of one. Here is the shape I use. It is a template, not a certified client.
import random, time
def with_backoff(call, attempts=4):
rec = None
for i in range(attempts):
rec = call() # call() logs every attempt itself
if rec["status"] != 429:
return rec
wait = min(2 ** i, 30)
time.sleep(wait + random.uniform(0, 0.5))
return rec
Notice that call() still writes a log line on every attempt. A retry you cannot count is a retry you cannot explain.
Myth 5: "Free means I can skip the provenance log"
This is exactly backwards. When you pay, the invoice is your record. When you do not pay, the invoice disappears.
Your own append-only file becomes the only artifact that says what ran, when, and against which model. That file is cheaper than an incident review.
The artifact: a provenance probe you can run
This script targets any OpenAI-compatible chat endpoint. Set three env vars and run it. Treat header names as provider-specific; I have not tested it against every gateway.
"""probe.py - append-only provenance record for one chat call."""
import json, os, time, requests
BASE = os.environ["MC_BASE_URL"].rstrip("/")
KEY = os.environ["MC_API_KEY"]
MODEL = os.environ["MC_MODEL"]
LOG = os.environ.get("PROVENANCE_LOG", "provenance.jsonl")
def probe(prompt="Reply with the single word: ok") -> dict:
started = time.perf_counter()
resp = requests.post(
f"{BASE}/chat/completions",
headers={"Authorization": f"Bearer {KEY}"},
json={"model": MODEL, "max_tokens": 8,
"messages": [{"role": "user", "content": prompt}]},
timeout=60,
)
served = None
try:
served = resp.json().get("model")
except ValueError:
pass # HTML error page; keep the raw status instead
rec = {
"ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"requested_model": MODEL,
"served_model": served,
"status": resp.status_code,
"latency_ms": round((time.perf_counter() - started) * 1000),
"request_id": resp.headers.get("x-request-id"),
"retry_after": resp.headers.get("retry-after"),
}
with open(LOG, "a", encoding="utf-8") as fh:
fh.write(json.dumps(rec) + "\n")
return rec
if __name__ == "__main__":
print(json.dumps(probe(), indent=2))
No Python? The same evidence fits in one shell pipeline.
curl -sS -D headers.txt -o body.json \
-H "Authorization: Bearer $MC_API_KEY" \
-H "Content-Type: application/json" \
-d "{\"model\":\"$MC_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"ok\"}],\"max_tokens\":4}" \
"$MC_BASE_URL/chat/completions"
grep -iE '^(x-request-id|retry-after|x-ratelimit)' headers.txt
jq -r '.model' body.json
How to read the output
| Field | What it proves | What it does not prove |
|---|---|---|
status |
The HTTP layer answered | Which model generated tokens |
served_model |
What the server claims it used | That the claim is true |
request_id |
A handle for a support ticket | Anything about reproducibility |
latency_ms |
One sample, one path | Your p50, your p99, your user's day |
retry_after |
The server asked you to wait | How many attempts the SDK made |
One row is an anecdote. Sixty rows sorted by time are a baseline.
Decision table: which "free" do you actually need?
| Your situation | Entitlement matters most | Compute matters most | Why |
|---|---|---|---|
| Prompt iteration and prompt diffing | Yes | No | You need many cheap calls |
| CI smoke test with no secrets | Yes | No | Short-lived, stateless |
| Long-running worker with a queue | No | Yes | Uptime beats burst budget |
| Anything holding production credentials | Neither | Neither | Free tiers are not prod |
| Audit or compliance evidence | Partly | Partly | Your own log is the artifact |
Who should not use this approach
- Anyone needing an uptime guarantee. Neither kind of free provides one.
- Teams handling regulated data on shared infrastructure.
- Projects where a silent model swap changes legal or safety outcomes.
Limitations, stated plainly
A provenance log records what a server claimed, not what executed. It cannot detect a quantized variant behind the same alias. It cannot prove the weights. It also adds I/O to every call, so keep it out of hot loops.
I will not quote quotas, hardware, or model names for any provider here. I cannot verify them today, and yesterday's number is tomorrow's misinformation. Measure your own endpoint with the probe above.
Ten-minute verification checklist
- Run the probe once. Confirm the script wrote one JSONL line.
- Compare
requested_modelandserved_model. Note any mismatch. - Force a
429with a burst. Confirm your backoff honoredretry-after. - Diff interpreter and OS facts between laptop and remote.
- Re-run the probe tomorrow and diff the two files with
jq -S.
If you want somewhere disposable to run steps one through five, I used MonkeyCode's free model access and free server option for exactly this kind of probe. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
A green checkmark means a socket accepted bytes. That is all it ever meant. The rest is your log's job.
Top comments (1)
I’d also record a stable request fingerprint and the gateway region alongside the model fields. Those two details make a later mismatch actionable: you can group silent fallbacks by route, distinguish a provider change from client variation, and reproduce the probe without retaining prompt contents.