A Cost-Per-Token Benchmark Is Not Enough: A Reproducible Harness for Free and Cheap Model Endpoints

A 20-step coding-agent loop that waits 900 ms per model call adds 18 seconds of idle time no matter whether each token costs $0.10 or $0.00. That is why a new model announcement that emphasizes only price per token is not enough information to switch. The more important numbers are latency distribution, failure rate, token accounting, and truncation behavior. This article provides a small harness for capturing those numbers against any OpenAI-compatible endpoint, including a free model access tier or free server option you are evaluating.

The metrics that predict whether a model is usable

The table below separates the numbers that usually predict whether a model is usable in a real workflow from the numbers that only tell you how cheap it could become.

Metric	Why it matters	Failure signal
p50/p95 latency	p95 determines the worst common experience; a 20-step loop is held back by its slowest step.	p95 more than 2.5x p50, or p95 above 3000 ms in a short synchronous task.
Failure/429 rate	Retry storms can make a free tier expensive in engineering time.	Failure rate above 2% in a 30-sample run, or 429s before concurrency 4.
Prompt/completion token accounting	Cost estimates are meaningless if the endpoint does not report usage.	prompt_tokens or completion_tokens missing, unstable, or clearly wrong.
Finish reason	A short task ending in length means the response was truncated.	finish_reason is not stop for a summary or classification task.

A reproducible endpoint harness

The point of the harness is to compare the same inputs across every model identifier you are considering. The names deepseek-v4-pro-0813 and gork-4.6 below are placeholders, not claims that those exact strings exist on your endpoint. Replace them with the routed identifier your provider actually exposes.

import os
import time
import json
from collections import defaultdict
from openai import OpenAI

ENDPOINT = os.environ.get('BASE_URL')
API_KEY = os.environ.get('API_KEY', 'not-needed')
# Replace these example identifiers with the exact model strings your endpoint exposes.
EVAL_MODELS = os.environ.get(
    'MODELS',
    'deepseek-v4-pro-0813,gork-4.6,free-tier-model',
).split(',')

PROMPTS = [
    'Summarize this RFC in 3 bullets: ',
    'Return the call graph for this function: ',
    'Review this diff for null handling: ',
]

client = OpenAI(base_url=ENDPOINT, api_key=API_KEY)

def run_one(model, prompt, max_tokens=256):
    started = time.perf_counter()
    try:
        resp = client.chat.completions.create(
            model=model,
            messages=[{'role': 'user', 'content': prompt}],
            max_tokens=max_tokens,
            temperature=0.0,
        )
        latency_ms = (time.perf_counter() - started) * 1000
        usage = getattr(resp, 'usage', None)
        return {
            'model': model,
            'ok': True,
            'latency_ms': round(latency_ms, 1),
            'prompt_tokens': getattr(usage, 'prompt_tokens', None),
            'completion_tokens': getattr(usage, 'completion_tokens', None),
            'finish_reason': resp.choices[0].finish_reason,
        }
    except Exception as exc:
        return {
            'model': model,
            'ok': False,
            'latency_ms': round((time.perf_counter() - started) * 1000, 1),
            'error': type(exc).__name__,
        }

report = defaultdict(list)
for model in EVAL_MODELS:
    for prompt in PROMPTS:
        report[model].append(run_one(model, prompt))

print(json.dumps({m: report[m] for m in report}, indent=2))

Run it with environment variables that match the endpoint you are testing:

BASE_URL='https://your-endpoint.example/v1' \
API_KEY='your-key-or-placeholder' \
MODELS='deepseek-v4-pro-0813,gork-4.6,free-tier-model' \
python eval_endpoint.py

How to read the report

Do not stop at the first successful response. A single prompt returns a single data point, not a trend.

• Calculate failure rate per model as failed requests divided by total requests. A rate above 2% in a 30-sample run is a reason to keep the model out of synchronous user requests.
• Sort latency_ms and take the 95th percentile. A p95 that is more than 2.5 times the median often indicates cold starts, contention, or hidden retry behavior.
• Treat missing prompt_tokens and completion_tokens as unknown cost, not zero cost. If you cannot account for tokens, you cannot compare models on price.
• If finish_reason equals length on a short summary, the endpoint truncated the response. That matters more than a lower price per token.
• Repeat the run at three different time windows. A single burst is not a benchmark.

Where MonkeyCode fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option is useful at the beginning of this workflow because it lets you run the harness before committing a billing method. The value is not that the free tier is faster or better; it is that you can capture your own failure, latency, and token accounting signals against the same API shape your code will use. If the free tier report is clean, the next step is to repeat it against the paid production endpoint. If it is not clean, you have a cheap early signal to stop before spending more engineering time.

Limitations and non-use cases

This evaluation is not a load test and does not verify any specific quota, uptime, hardware, or model name. The placeholder identifiers should not be read as current product names. Free tiers can change without notice, so pin the model string and rerun before any production decision. Do not send regulated, private, or proprietary code to a free server endpoint unless you have verified its data handling policy. Teams with a strict latency SLA, high concurrency, or compliance requirements should use a paid managed endpoint and a dedicated load test rather than relying on this lightweight sanity check.