Jordan Huang

Posted on Sep 1

Why Your Free-Tier Benchmark Is Misleading (and What to Run Instead)

#ai #testing #api #python

You got a fast response once. So the free tier must be fine, right?

Then you deploy it, and the next request takes eleven seconds. Or it returns HTML instead of JSON. Or it works for three hours then 503s for a day.

Your benchmark was misleading. Let me show you what actually happened.

I maintain this field protocol as a developer evaluating MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The steps are generic. You can use them on any provider.

The Three Misleading Metrics

Most "free tier is solid" conclusions come from three lazy measurements.

Myth 1: Average Latency Is the Truth

You send ten requests, compute the mean, get 400ms. Feels great.

The average hides tails. Free tiers share CPU, network, and request queues. The distribution is a long curve, not a spike.

Ask instead: What's the p90? The p99? What happens at noon or midnight?

Myth 2: A Correct-Looking Output Means Correct Output

You ask a model for JSON. It returns something that parses. Done, right?

But the model may skip a required field, or wrap the answer in markdown, or emit an empty string that parses as valid.

You need contract tests. Validate structure and types, every time.

Myth 3: One Success Window Proves Stability

You ran the test at 2 a.m. on a Tuesday. Everything passed.

Shared infrastructure changes by the hour. A single run proves nothing about evenings, weekends, or release days of the underlying platform.

Repeat the same test across at least three different time windows before you trust it.

The Correct 15-Minute Protocol

Here's a reproducible script that catches all three myths.

Save this as free_tier_probe.py:

import os, time, json, statistics, sys
import httpx

url = os.environ.get("API_URL")
key = os.environ.get("API_KEY")
if not url:
    sys.exit("Set API_URL")

headers = {"Authorization": f"Bearer {key}"} if key else {}
payload = {"prompt": "Explain the CAP theorem in one sentence."}

samples = int(os.environ.get("SAMPLES", "20"))
times = []
statuses = []
structure_errors = 0

with httpx.Client(timeout=30) as client:
    for i in range(samples):
        start = time.perf_counter()
        try:
            r = client.post(url, json=payload, headers=headers)
            elapsed = time.perf_counter() - start
            times.append(elapsed)
            statuses.append(r.status_code)

            data = r.json()
            # Minimal contract: response must contain a non-empty "text" string
            if not (isinstance(data, dict) and isinstance(data.get("text"), str) and data["text"].strip()):
                structure_errors += 1
        except Exception as e:
            times.append(time.perf_counter() - start)
            statuses.append(0)
            structure_errors += 1

if not times:
    sys.exit("No responses recorded")

ts = sorted(times)
print(f"samples: {len(ts)}")
print(f"p50: {statistics.median(ts):.2f}s")
print(f"p90: {ts[int(len(ts)*0.9) - 1]:.2f}s")
print(f"p99: {ts[min(int(len(ts)*0.99) - 1, len(ts)-1)]:.2f}s")
print(f"error_rate: {structure_errors / samples:.2%}")
print(f"status_codes: { {k: statuses.count(k) for k in set(statuses)} }")

Install the client:

pip install httpx

Run it three times, hours apart:

export API_URL="https://your-endpoint" export API_KEY="secret"
SAMPLES=30 python free_tier_probe.py
sleep 21600  # wait six hours, run again
sleep 21600  # wait another six hours, run again

Why 30 samples? Enough to see tail behavior without eating your free quota. Why three runs? Because one window is a vibes check, not evidence.

Reading the Results

Look at the error rate first. If it's above 5%, stop. The free tier is not ready for your flow.

Then compare p50 and p90. A p90 that is 4x the p50 means you're getting queue bursts. That's normal for shared servers, but you must expect it in your timeout budget.

Finally, check the structure_errors. Even with HTTP 200, a malformed response is a failure for your pipeline.

For a more advanced contract, use JSON Schema with the jsonschema library. But the snippet above is enough to expose the three myths.

Limitations of This Approach

This protocol tests reliability, not semantic quality. A model can return valid JSON that is factually wrong.

It also measures a single endpoint and a single prompt. Different payload sizes change latency. Test with your real use case.

Free tiers can change quotas or routing with no notice. Re-run the probe weekly. Treat results as a snapshot, not a permanent guarantee.

Who Should Skip This

If you're building a one-off demo and just need a few responses, don't bother. Manual eyeballing is fine for a weekend script.

If you're planning a real product or an automated pipeline, this is the minimum viable evidence. You still need retries, fallbacks, and timeouts regardless of what the benchmark shows.

What I Learned

My first benchmark was a lie too. I saw one quick response, wrote code around it, and woke up to a queue of broken calls.

Since then I always run the protocol above before touching a new free-tier provider. It takes me a quarter of an hour and saves me a day of blame-shifting.

Want to adapt this for your own platform? Point the script at any compatible endpoint. The numbers don't care where they came from.

And if you're curious how MonkeyCode's free model and free server hold up, run this exact probe. You'll know more in 15 minutes than most blog posts will tell you.

DEV Community