Free AI Servers Hide Their Worst Behavior. This Probe Exposes It.

Free AI servers hide their worst behavior behind a single curl command. The blog screenshot shows one good answer; production sees queues, rate limits, and truncation. If you're building an agent on a free tier, you need a probe that measures the system under load, not one lucky call.

I see this all the time with tools like MonkeyCode, which offers free model access and a free server option. People paste a demo, assume the infra is solid, and don't test until a Friday 7 PM incident. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This post gives you a reproducible probe that quantifies four things a free server rarely advertises: latency percentiles, error rate, truncation rate, and output consistency. Run it before you trust any free endpoint—MonkeyCode's included.

The Probe

The script below uses the OpenAI-compatible chat completions interface. You set two environment variables: MC_API_URL and MC_API_KEY. It fires runs requests at concurrency workers, records the latency, status, and whether the response was cut off by finish_reason == "length". It also repeats the same prompt at least twice so you can see how stable the output really is.

# probe_free_server.py
import os, json, time, statistics
from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
from difflib import SequenceMatcher

def one_call(url, key, prompt):
    headers = {"Authorization": f"Bearer {key}"}
    payload = {
        "model": "free-tier",  # change to whatever model string your provider uses
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 200,
        "temperature": 0.0,
    }
    start = time.perf_counter()
    try:
        r = requests.post(url, headers=headers, json=payload, timeout=60)
        latency = time.perf_counter() - start
        data = r.json()
        if r.status_code != 200:
            return {"status": r.status_code, "latency": latency, "error": str(data)}
        choice = data["choices"][0]
        return {
            "status": 200,
            "latency": latency,
            "content": choice["message"]["content"],
            "truncated": choice.get("finish_reason") == "length",
            "usage": data.get("usage", {})
        }
    except Exception as e:
        return {"status": -1, "latency": time.perf_counter() - start, "error": str(e)}

def probe(url, key, prompt, concurrency=5, runs=10):
    with ThreadPoolExecutor(max_workers=concurrency) as ex:
        futures = [ex.submit(one_call, url, key, prompt) for _ in range(runs)]
        results = [f.result() for f in as_completed(futures)]

    ok = [r for r in results if r["status"] == 200]
    latencies = sorted(r["latency"] for r in ok)
    report = {
        "total_requests": runs,
        "ok": len(ok),
        "error_rate": (runs - len(ok)) / runs,
        "latency_p50": latencies[len(latencies)//2] if latencies else None,
        "latency_p95": latencies[int(len(latencies)*0.95)-1] if len(latencies) > 1 else None,
        "truncation_rate": sum(r["truncated"] for r in ok) / len(ok) if ok else None,
    }
    if len(ok) >= 2:
        a, b = ok[0]["content"], ok[1]["content"]
        report["consistency"] = SequenceMatcher(None, a, b).ratio()
    return report

if __name__ == "__main__":
    url = os.environ.get("MC_API_URL", "https://api.example.com/v1/chat/completions")
    key = os.environ.get("MC_API_KEY", "")
    prompt = "Explain in two sentences why a token budget matters more than model size on a shared free server."
    print(json.dumps(probe(url, key, prompt), indent=2))

Run it like this:

export MC_API_URL="https://your-provider.example/v1/chat/completions"
export MC_API_KEY="your_key_here"
python probe_free_server.py

Don't hardcode secrets. Use a .env file or your CI secrets manager.

What the Numbers Mean

The probe returns six fields. Here are the thresholds I use as a sanity check. They are deliberately strict because free tiers have to earn trust operationally, not just by answering a prompt.

Metric	Trust it	Investigate	Run away
p95 latency @ 5 concurrency	< 5s	5–15s	> 15s
error rate	< 1%	1–5%	> 5%
truncation rate	0%	< 10%	> 10%
consistency (same prompt, temp=0)	> 0.8	0.5–0.8	< 0.5

Latency p95 catches the queue that p50 hides. A free server with 20 concurrent users will have a fat tail; your agent's retry loop may blow up before you see a single slow response.

Error rate matters for any automation. A 3% error rate means your 300-step migration job has nine failures you didn't plan for.

Truncation rate is the silent killer. The server returns a 200 OK, the model hits max_tokens, and your code parses half a JSON blob. The finish_reason field is the only warning you get.

Consistency at temperature 0 tells you if the server is doing something nondeterministic under the hood, like batching your request with a different model or restarting a container mid-call. Two identical prompts should produce near-identical text; if they don't, your tests will be flaky before you write a single assertion.

Why This Probe Works

It isolates infrastructure from model quality. You're not asking "is the model smart?" You're asking "can this shared server return predictable results under load?" If it can't, prompt engineering won't save you. You'll be debugging timeouts and half-repaired responses at 2 AM.

The probe also gives you a baseline. Run it once when you deploy, once after a month, once when a new free model is visible in the org. Free tiers change quotas and routing rules more often than paid ones. Data from three months ago is trivia, not evidence.

Limitations

This script measures plumbing, not thinking. It won't tell you whether the model can write secure code or understand your domain. For that you still need a labeled bug set or a pair of human eyes.

It also ignores time-of-day effects. A free server at 3 AM is not the same product as a free server at 9 PM. If you're building an agent that runs during business hours, run the probe multiple times across a full week. One run is a snapshot, not a distribution.

Finally, this probe assumes an OpenAI-compatible endpoint. If your free server exposes a different API, adapt the payload and the response parsing. The metric definitions stay the same.

Who Should Skip This

If you're using a free server for occasional autocomplete, skip the probe. The cost of measurement outweighs the benefit. But if you're wiring it into a CI pipeline, a Slack bot, or an autonomous agent that makes secret decisions, this probe is the minimum due diligence.

Take it one step further: run the probe, save the JSON, and re-run it after you've been a user for a week. The numbers will drift. When they do, you'll know exactly which part of your trust boundary broke.

Drop your p95 in the comments if you run it against any free endpoint—MonkeyCode's included. I'd like to see how the tail behaves for the rest of the world.