Can You Trust a Free AI Stack? Five Verification Myths, Tested

You ran one prompt. It worked. You wrote it into the design doc.

Three weeks later a teammate reruns the same script and gets a timeout. Nobody changed anything. So what did you actually verify?

I have hit that wall twice. Both times the root cause was a myth I had accepted without testing. Here are the five I hear most often, plus the small harness I now use to kill them.

Where I run these probes

I do my cheap experiments on MonkeyCode's free model access and its free server option. Both are operator-supplied availability claims, not my promises.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free-tier details move. Check the current project page before you build on any number. That is exactly why the artifact below records a fingerprint and a date instead of trusting my memory.

Myth 1: "It's free, so I can fan out 50 requests"

Free describes a price, not a capacity plan.

Shared capacity is normal on free endpoints. What you observe at 10 a.m. may not hold at 10 p.m.

How to test it:

1. Send 20 sequential requests, then 20 in parallel.
2. Log status code, latency, and retry count for each.
3. Compare the failure rate, not just the average.

Corrected model: Free access is a queue you share with strangers. If your job needs guaranteed concurrency, free is the wrong tier — and no benchmark from last month will save you.

Myth 2: "A free server is my staging environment"

It runs a container. That is not the same as owning an SLO.

Staging implies an owner, a pager, and a rollback path. A free server usually means none of those.

How to test it:

• Start a long job, then drop your connection mid-run.
• Reconnect and check whether the process still exists.
• Write a file, restart, and confirm what survives.

Whatever the answer is, write it down as a property of today, not a guarantee.

Who should not use this approach: anything customer-facing, anything with an uptime clause, and anything holding secrets you cannot rotate in an hour.

Myth 3: "Two identical runs prove reproducibility"

Two matching outputs are one sample, not determinism.

You did not reproduce a result. You observed it twice under unknown conditions.

How to test it: record a fingerprint before every run.

• Python or runtime version
• Endpoint URL and, if exposed, a model revision string
• Prompt hash and temperature
• A timestamp

Corrected model: A run is reproducible only relative to a recorded fingerprint. Without that record, "it worked yesterday" is folklore.

The account's earlier piece on this pattern is worth reading if you want the longer argument. Here I only need the habit.

Myth 4: "My token allowance guarantees throughput"

An allowance is a ceiling. Throughput is a rate.

They are different units, and confusing them breaks capacity math on a Friday.

How to test it:

1. Log the usage field on every response, if the endpoint returns one.
2. Sum tokens per call over 20 calls.
3. Use the p90 call cost, never the mean, for planning.

Corrected model: Budget math is about the worst realistic call, not the friendliest one. One verbose tool response can eat a day of headroom.

Myth 5: "It's open source, so I've verified my deployment"

Reading source code and verifying a running instance are different jobs.

Readable code tells you what could run. It says nothing about what you are actually talking to.

How to test it: pin what you run.

• Record the commit or release you deployed.
• Log response headers and any version field you can see.
• Re-check that pin weekly; free deployments drift.

Corrected model: Your deployment is a separate artifact with its own evidence trail. Version pinning is your job, not the project's.

The artifact: a claim ledger and one probe script

I stopped arguing about free tiers and started writing ledgers. A ledger is one JSON file per claim, with an expiry date.

{
  "claim": "p90 latency under 3s for short prompts",
  "method": "probe.py, N=20, sequential, 1-word prompt",
  "evidence": "summary.p90_ms = 2180",
  "verdict": "holds-today",
  "recorded": "2026-09-15",
  "expires": "2026-09-22"
}

The rule is simple. No expiry, no claim.

Here is the probe that fills in the evidence field. It is runnable as written.

#!/usr/bin/env python3
# probe.py - record latency + usage for one endpoint. No secrets in the file.
import hashlib, json, os, platform, sys, time, urllib.request

ENDPOINT = os.environ["PROBE_ENDPOINT"]
API_KEY = os.environ.get("PROBE_KEY", "")
PROMPT = "Reply with the single word: pong"
N = int(os.environ.get("PROBE_N", "20"))

def fingerprint():
    return {
        "python": sys.version.split()[0],
        "os": platform.platform(),
        "endpoint": ENDPOINT,
        "prompt_sha": hashlib.sha256(PROMPT.encode()).hexdigest()[:12],
        "started": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }

def call(i):
    body = json.dumps({
        "messages": [{"role": "user", "content": PROMPT}],
        "temperature": 0,
    }).encode()
    headers = {"Content-Type": "application/json"}
    if API_KEY:
        headers["Authorization"] = "Bearer " + API_KEY
    req = urllib.request.Request(ENDPOINT, data=body, headers=headers)
    t0 = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=60) as r:
            payload = json.loads(r.read().decode())
            return {
                "i": i,
                "status": r.status,
                "ms": round((time.perf_counter() - t0) * 1000, 1),
                "ok": True,
                "usage": payload.get("usage"),
            }
    except Exception as e:
        return {
            "i": i,
            "status": getattr(e, "code", 0),
            "ms": round((time.perf_counter() - t0) * 1000, 1),
            "ok": False,
            "error": type(e).__name__,
        }

records = [call(i) for i in range(N)]
lat = sorted(r["ms"] for r in records)
summary = {
    "fingerprint": fingerprint(),
    "n": N,
    "ok": sum(1 for r in records if r["ok"]),
    "p50_ms": lat[len(lat) // 2],
    "p90_ms": lat[min(len(lat) - 1, int(len(lat) * 0.9))],
    "max_ms": lat[-1],
}
print(json.dumps({"summary": summary, "records": records}, indent=2))

Run it against whatever you actually use:

export PROBE_ENDPOINT="https://your-endpoint.example/v1/chat/completions"
export PROBE_KEY="$YOUR_KEY"
python3 probe.py > ledger-$(date +%F).json
python3 -c "import json;print(json.load(open('ledger-$(date +%F).json'))['summary'])"

For the parallel myth, I use a threaded variant. I have not included it here because I have not run it against every endpoint shape, so treat it as a sketch:

# sketch only - not executed here
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=20) as ex:
    parallel = list(ex.map(call, range(20)))

Decision table: what a result actually tells you

Observation	Proves	Does not prove
20/20 successful calls	The endpoint worked in that window	It will work at your peak hour
p90 under your threshold today	A usable baseline right now	A trend, or next week's number
Process alive after reconnect	That lifecycle on that day	Persistence as a documented feature
Source code public	You can read the logic	Your instance matches that commit
Usage field present	The endpoint reports tokens	Your allowance covers your real workload

Copy that table into your own repo. Edit the right-hand column until it stops flattering your design.

Limitations and honest edges

• A 20-call probe is a smoke test. It cannot detect weekly patterns.
• Latency measured from my network is not latency from yours. Always keep your own ledger.
• Free-tier behavior changes without notice. A ledger without an expiry date quietly becomes misinformation.
• This workflow verifies availability and cost shape. It does not verify output quality. For that you need labeled evaluation data, which is a different article.

What I do every Monday

1. Re-run the probe and diff the summary against last week.
2. Mark stale ledgers as expired instead of deleting them.
3. Promote only claims that survived three weeks to the design doc.
4. Keep a parallel fallback path wired up before I need it.

That is it. Fifteen minutes, one file, and my README stops lying to my teammates.

If you want to compare ledgers, the MonkeyCode project page lists the current free access details. Run the script before you trust my numbers — or anyone's.

What is the oldest latency claim in your repo right now? If you cannot put a date on it, it is already expired.