A 200 OK Is Not Evidence: Five Hosted-Model Myths and a Three-Gate Probe

One successful request feels like proof. It is not.

You send a prompt. You get valid JSON back. You wire it into a nightly job.

Then the job runs somewhere you did not set up. Different Python. Different packages. Different timeout. The model did not change. Your pipeline broke anyway.

So what did that first 200 OK actually prove? Almost nothing about the system around it.

That gap is where most "the model broke my build" stories live. Here are five myths that keep it open, and a probe that closes it.

Myth 1: "It answered once, so it works."

One success proves the happy path exists. It does not prove the path repeats.

Concurrency, long inputs, and empty inputs fail in different ways. Each one deserves its own case.

Corrected model: a reply is a sample, not a contract. Collect counts, not vibes.

Myth 2: "Free model access and a free server are one thing."

They are two independent resources. They fail independently.

Endpoint access can answer while your box is out of disk. Your box can be healthy while the endpoint rejects you. If you debug both at once, you learn nothing.

Corrected model: access and compute are two separate columns in your receipt. Never collapse them.

Myth 3: "Temperature 0 makes a replay check meaningful."

No. Temperature 0 lowers randomness. It does not promise byte-identical text.

So hashing model output is a false gate. It goes red for reasons that are not your bug. You then spend an afternoon chasing nothing.

Corrected model: hash the environment, not the sentence. Compare machine state, package versions, and status sets.

Myth 4: "A failed probe means the provider is bad."

A probe fails at a specific gate. Every gate has a specific owner.

Transport failures belong to the endpoint or your network. Contract failures are usually your prompt or your parser. Environment failures are your image. Blaming the vendor hides all three.

Corrected model: a red gate is a work item with an owner, not a verdict.

Myth 5: "Free tiers are only for throwaway code."

Free is a proving ground. It is a terrible place to run production.

But it is an excellent place to discover that your workflow is not yet a workflow. That discovery is cheap. Making it in production is not.

Corrected model: use free access to earn the receipt. Use the receipt to decide when to pay.

The artifact: three gates, one receipt

I keep one script. It runs three gates and prints a single JSON receipt.

• Gate A - transport. Send N requests. Count statuses. Record wall clock only so regressions are visible.
• Gate B - contract. Send the edge inputs you actually care about. Count shape violations.
• Gate C - environment replay. Fingerprint the machine. Never the model text.

Here is the whole thing. It talks to any OpenAI-compatible endpoint.

# probe.py - three-gate pre-flight for any OpenAI-compatible endpoint
import hashlib, json, os, subprocess, sys, time, urllib.error, urllib.request

BASE  = os.environ["PROBE_BASE_URL"].rstrip("/")
KEY   = os.environ["PROBE_API_KEY"]
MODEL = os.environ["PROBE_MODEL"]
N     = int(os.environ.get("PROBE_N", "8"))

EDGE_PROMPTS = [
    "",                                    # empty input
    'Reply with JSON: {"ok": true}',       # contract ask
    "\U0001F642" * 200,                     # unicode weight
    "x " * 4000,                           # long input
]

def call(prompt, timeout=45):
    payload = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        BASE + "/chat/completions",
        data=payload,
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + KEY},
    )
    start = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            body = json.loads(resp.read())
            status = resp.status
    except urllib.error.HTTPError as err:
        return {"status": err.code, "seconds": time.monotonic() - start, "text": None}
    except Exception:
        return {"status": 0, "seconds": time.monotonic() - start, "text": None}
    text = body.get("choices", [{}])[0].get("message", {}).get("content")
    return {"status": status, "seconds": time.monotonic() - start, "text": text}

def gate_a():
    """Transport: does it answer at all, under my concurrency?"""
    runs = [call("ping") for _ in range(N)]
    return {
        "requests": N,
        "ok": sum(1 for r in runs if r["status"] == 200),
        "statuses": sorted({r["status"] for r in runs}),
        "wall_seconds": round(sum(r["seconds"] for r in runs), 2),
    }

def gate_b():
    """Contract: does the shape hold on the inputs I really send?"""
    violations = []
    for prompt in EDGE_PROMPTS:
        r = call(prompt)
        if r["status"] != 200:
            violations.append({"case": prompt[:24], "why": "status %s" % r["status"]})
            continue
        if prompt.startswith("Reply with JSON"):
            try:
                json.loads(r["text"] or "")
            except Exception:
                violations.append({"case": prompt[:24], "why": "not parseable JSON"})
    return {"cases": len(EDGE_PROMPTS), "violations": violations}

def gate_c():
    """Environment replay: fingerprint the machine, never the model text."""
    pkgs = subprocess.run([sys.executable, "-m", "pip", "freeze"],
                          capture_output=True, text=True).stdout
    return {
        "probe_sha": hashlib.sha256(open(__file__, "rb").read()).hexdigest()[:12],
        "packages_sha": hashlib.sha256(pkgs.encode()).hexdigest()[:12],
        "python": sys.version.split()[0],
    }

if __name__ == "__main__":
    receipt = {"gate_a": gate_a(), "gate_b": gate_b(), "gate_c": gate_c()}
    print(json.dumps(receipt, indent=2, sort_keys=True))

Then run it twice, from two clean places:

# environment 1: your laptop
PROBE_BASE_URL="$BASE" PROBE_API_KEY="$KEY" PROBE_MODEL="$MODEL" \
  python probe.py > receipt-a.json

# environment 2: a clean container, clean venv
PROBE_BASE_URL="$BASE" PROBE_API_KEY="$KEY" PROBE_MODEL="$MODEL" \
  python probe.py > receipt-b.json

# compare the machine, not the sentence
diff <(jq -S .gate_c receipt-a.json) <(jq -S .gate_c receipt-b.json)
diff <(jq -S .gate_a.statuses receipt-a.json) \
     <(jq -S .gate_a.statuses receipt-b.json)

Three things to notice. Gate A gives you a status set, not an average. Gate B gives you named violations, not a score. Gate C gives you two hashes you can diff.

No receipt should contain your API key, and none should contain real customer prompts. Redact before you commit anything.

The decision table

I read the receipt in this order. A is cheapest, C is the one people skip.

Gate A (transport)	Gate B (contract)	Gate C (environment)	Verdict
Pass	Pass	Pass	Safe to automate, with a fresh receipt per release
Pass	Fail	-	The endpoint is fine. Fix the prompt or the parser first
Fail	-	-	Keep it interactive. Do not put it in CI yet
Pass	Pass	Fail	Pin the image. Free compute drifted, not the model
Fail	Fail	Fail	Nothing is pinned. Close one gate at a time

The middle row is the one I hit most often. The endpoint was never the problem. My schema expectations were.

Limitations, stated plainly

This probe measures liveness, shape, and environment parity. It does not measure answer quality. A green receipt tells you the pipe is connected, not that the water is clean.

• N=8 is a smoke test, not a load test. Raise it, but keep it cheap enough to run on every release.
• Free endpoints change. A receipt is a dated photograph, not a promise. Re-run it.
• Gate C compares environments on purpose. If you want text comparison, do it in your own test suite where you control the assertions.
• The wall-clock number in Gate A is a regression signal only. It is not a latency benchmark.

Who should not use this approach

Skip it if you need a contractual uptime figure. A free tier cannot give you one, and no probe fixes that. Buy the thing with a support clause.

Skip it if your prompts carry regulated data. Do not send them to a shared endpoint while you are still learning the gate boundaries.

Skip it if nobody on the team will read the receipt. An unread artifact is just log noise with extra steps.

Where free access earns its place

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. I used both while this workflow was still unproven, and that is exactly the right moment for them. Paying for capacity before you have a receipt is guessing. Paying for it after is a decision.

The probe still applies. Free access and the free server fail independently, which is why Myth 2 matters here. When Gate A went red for me, it was my own timeout. When Gate C went red, it was the image, not the endpoint. That is the whole point of splitting them.

Free is a proving ground, not a plan. Use it to find out what you do not know yet.

The habit that matters

Stop treating a single successful response as evidence. Collect a status set. Name your shape violations. Fingerprint the machine.

If you build this probe, commit the receipt next to the workflow it describes. Future you will want the date on it.