One Script, Five Signals: My Free Model Server Scorecard

A free model server looks like a gift. Until it returns garbage at 3 a.m.

Every week, a new model drops. Every week, someone wires it into production on day one. That's how incidents start.

I've spent weeks probing free endpoints. Concurrency. Structured output. Time-of-day variance. The pattern never changes: free works, until it doesn't.

So I built a scorecard. One script. Five signals. Fifteen minutes. Here's the evaluation.

This week I pointed the harness at MonkeyCode's free tier. MonkeyCode is an open-source project. It ships with 10 million free tokens and a free server option. I wanted one answer: where does it hold up, and where does it break?

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The five signals

A single benchmark run tells you almost nothing. Five signals, repeated three times, tell you a lot.

1. TTFT — time to first token. This is your perceived latency.
2. Output rate — words per second after the first token.
3. Error rate — HTTP 4xx, 5xx, timeouts, exceptions.
4. Output integrity — is the response complete and parseable?
5. Consistency — variance across repeated runs.

Why five signals? Because latency alone lies. A server can answer fast and still be broken. It can answer slowly and still be fine. The combination tells the real story.

One good run proves nothing. Three runs show a pattern. That's the whole trick.

The experiment design

Fixed prompts. Fixed order. Fixed client. No retries.

Retries hide failures. I want the raw failure. So the harness records everything and retries nothing.

• Five tasks: summarize, code, JSON, Q&A, rewrite.
• Three repetitions per task. Fifteen calls total.
• One machine. One network. One time window.
• Every result lands in a JSON array.

Why fifteen calls? Enough to expose gross failures. Small enough to run during a coffee break.

The script

Here's the whole harness. It targets any OpenAI-compatible chat endpoint.

# scorecard.py — 15-minute free model server evaluation
import json, os, statistics, time
import httpx

BASE_URL = os.environ["MC_BASE_URL"]   # e.g. https://host/v1
API_KEY = os.environ["MC_API_KEY"]
MODEL = os.environ["MC_MODEL"]

PROMPTS = {
    "summarize": "Summarize eventual consistency in three bullets.",
    "code": "Write a Python retry helper with exponential backoff.",
    "json": 'Return JSON only: {"name": "Ada", "role": "engineer"}',
    "qa": "What is idempotency? Answer in one short paragraph.",
    "rewrite": "Rewrite 'utilize' as 'use' in one short sentence.",
}

def run_one(task, prompt):
    t0 = time.perf_counter()
    first_token = None
    text = ""
    try:
        with httpx.stream(
            "POST", f"{BASE_URL}/chat/completions",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={"model": MODEL,
                  "messages": [{"role": "user", "content": prompt}],
                  "stream": True},
            timeout=60.0,
        ) as r:
            if r.status_code != 200:
                return {"task": task, "error": f"HTTP {r.status_code}"}
            for line in r.iter_lines():
                if not line or not line.startswith("data:"):
                    continue
                data = json.loads(line[5:])
                delta = data["choices"][0]["delta"].get("content", "")
                if delta:
                    if first_token is None:
                        first_token = time.perf_counter() - t0
                    text += delta
    except Exception as exc:
        return {"task": task, "error": type(exc).__name__}

    total = time.perf_counter() - t0
    words = len(text.split())
    return {
        "task": task,
        "ttft_s": round(first_token, 2) if first_token else None,
        "total_s": round(total, 2),
        "words": words,
        "words_per_s": round(words / max(total - (first_token or 0), 0.01), 1),
    }

def summarize(results):
    clean = [r for r in results if "error" not in r]
    return {
        "calls": len(results),
        "errors": len(results) - len(clean),
        "median_ttft_s": round(statistics.median(r["ttft_s"] for r in clean), 2),
        "median_words_per_s": round(statistics.median(r["words_per_s"] for r in clean), 1),
    }

if __name__ == "__main__":
    results = [run_one(t, p) for t, p in PROMPTS.items() for _ in range(3)]
    print(json.dumps({"runs": results, "score": summarize(results)}, indent=2))

Run it:

export MC_BASE_URL="https://your-endpoint/v1"
export MC_API_KEY="your-key"
export MC_MODEL="the-model-name"
python scorecard.py

Output looks like this (illustrative — your numbers will differ):

{
  "runs": [
    {"task": "summarize", "ttft_s": 1.8, "total_s": 9.4, "words": 82, "words_per_s": 10.7}
  ],
  "score": {"calls": 15, "errors": 0, "median_ttft_s": 1.9, "median_words_per_s": 11.2}
}

Word count is a proxy. Stream chunks are not tokens. If the endpoint returns usage fields, prefer those.

How to read the scorecard

Thresholds are workload decisions. Here are mine.

Signal	Pass	Warn	Fail
TTFT median	< 2 s	2–5 s	> 5 s
Output rate	> 20 w/s	5–20 w/s	< 5 w/s
Error rate	0%	< 20%	≥ 20%
Integrity	all clean	one bad	many bad
Consistency	p95 < 2× median	< 3×	≥ 3×

Take a hypothetical run. Median TTFT of 1.8 s is a Pass. Median output rate of 11 w/s is a Warn. Zero errors is a Pass. That's one Warn. Yellow flag. Proceed with caution.

A "Fail" in any row is a red flag. Two "Warn" rows are a yellow flag. Everything else is a maybe.

Free tiers don't come with SLAs. The scorecard is your SLA.

Where free model servers break

I've seen the same failure modes across several free endpoints.

• Silent truncation. HTTP 200 with an empty or half-finished body. The worst one, because logs look clean.
• Burst rate limits. 429s appear only under load. My earlier concurrency probe on a comparable free server broke at 16 parallel calls.
• Cold starts. The first call after idle is painfully slow. The second is fine.
• Degraded output. The server stays up, but the quality drops. No error. No warning.

How to confirm silent truncation

Silent truncation hides behind HTTP 200. Here's the check.

1. Compare the response length against what the prompt asked for.
2. Look for an unfinished final sentence. A missing period is a tell.
3. Re-run the same prompt. Repeated truncation is a server bug, not a fluke.

The scorecard catches all four modes. That's why error rate and integrity are separate signals.

What I concluded

The conclusion is the method, not my numbers.

One window of fifteen calls proves nothing permanent. It proves the server can survive fifteen calls. That's it. Run the scorecard twice, on different days. If both runs pass, you have a usable free tier. If not, you have evidence. That's more than most teams have.

What surprised me? Nothing. Free infrastructure behaves like free infrastructure: generous at the start, vague at the edges. The scorecard turns vague into concrete. That's the value.

When to use a free model server

Use it for:

• Prototypes and internal demos.
• Batch jobs with retries and a dead-letter queue.
• Drafting, summarization, classification — where a bad output is cheap.
• Testing your own client code against a real endpoint.

Don't use it for:

• User-facing latency with a contract.
• Regulated or sensitive data. Free tiers carry no guarantees.
• Anything where a silent empty 200 is worse than a loud error.

The rule is simple: match the workload to the guarantee. Free tiers guarantee nothing. So give them nothing critical.

Extending the harness

Fifteen calls is the floor, not the ceiling. Add what your workload needs.

• Larger prompts. Feed it a 2,000-word document.
• Longer streams. Ask for 500 words and watch the tail.
• Different hours. Run the same script at 9 a.m. and 9 p.m.
• Concurrent calls. My earlier probe showed concurrency is where free servers die first.

Keep the scoring logic identical. That's how you compare runs honestly.

Limitations of this evaluation

The harness is small by design. Fifteen calls catch gross failures. They don't prove reliability.

• One client, one region, one time window.
• Word rate is a proxy, not token throughput.
• Free tiers change. Re-run the scorecard before you trust it.

Try it

The script is the deliverable. Point it at any endpoint.

If you want a first target, MonkeyCode's free tier and free server option are a reasonable place to start. The project is open source, so the client code is inspectable.

Run the scorecard twice. Then decide. That's the whole evaluation.