Stop Benchmarking the Model. Start Benchmarking the Round Trip.

Last week a team I was advising showed me a benchmark they were proud of: forty tokens per second on a fresh model, measured in a clean notebook. Then they deployed the same feature and watched a single user request take eleven seconds from click to rendered answer. The model was fast, and the round trip was a disaster. That gap is not a tuning problem; it is a measurement problem, and it will keep biting you until you change what you measure.

Every AI feature is a chain of handoffs: the browser, the edge, your auth layer, your API, the queue, the server, the model provider, the persistence layer, and the way back. A token-per-second number measures exactly one link in that chain, and it is usually the fastest one. The rest of the chain is where latency, cost, and failure actually live. So why do we keep publishing benchmarks for the one link that never breaks?

There is a healthy argument on DEV right now about what the AI badge actually measures, and the same doubt applies to model benchmarks: they measure the wrong thing with great precision. The number users feel is the round trip, and the only honest way to measure it is to run the whole chain on the weakest legitimate environment you can find. That is where a free server becomes the most valuable tool in your stack, and it is also where most teams stop taking measurements seriously.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That is why I have been running my round-trip tests against MonkeyCode, an open-source project that bundles free model access with a free server option. The model access gives you a real inference endpoint, and the server gives you a real deployment target, which means you can measure the entire chain instead of a notebook cell. Because it is open source, you can read the proxy code instead of trusting a dashboard, which matters when you are debugging the handoff between your server and the model. As of this writing the free grant is advertised at ten million tokens, but do not build a business on that number; quotas change, and the point of the exercise is the measurement, not the allowance.

Here is the reproducible test I run against any AI endpoint before I trust it, and you can run it against the MonkeyCode free server in about ten minutes. Start with a single curl to feel the cold start:

curl -s -o /dev/null -w "cold start: %{time_total}s\n" \
  -X POST https://your-free-server.example/api/chat \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Say hello in one sentence.","stream":false}'

Then run the full script, which measures cold start, warm p50 and p95, concurrent p95, and error rate against a budget you define:

# roundtrip.py — measure the contract, not the model
import asyncio
import statistics
import time

import httpx

URL = "https://your-free-server.example/api/chat"
BUDGET = {"p95_seconds": 3.0, "error_rate": 0.01}
PROMPT = {"prompt": "Summarize this repository in three sentences.", "stream": False}


def p95(values: list[float]) -> float:
    ordered = sorted(values)
    return ordered[min(len(ordered) - 1, int(len(ordered) * 0.95))]


async def one_call(client: httpx.AsyncClient) -> tuple[float, int]:
    start = time.perf_counter()
    try:
        response = await client.post(URL, json=PROMPT, timeout=30)
        return time.perf_counter() - start, response.status_code
    except httpx.HTTPError:
        return time.perf_counter() - start, 0


async def main() -> None:
    async with httpx.AsyncClient() as client:
        cold, cold_code = await one_call(client)              # first call after idle
        warm = [await one_call(client) for _ in range(20)]    # sequential warm calls
        concurrent = await asyncio.gather(*(one_call(client) for _ in range(10)))

    warm_times = [t for t, _ in warm]
    all_codes = [code for _, code in warm + list(concurrent)]
    error_rate = sum(1 for code in all_codes if code >= 400 or code == 0) / len(all_codes)

    print(f"cold start:     {cold:.2f}s (status {cold_code})")
    print(f"warm p50:       {statistics.median(warm_times):.2f}s")
    print(f"warm p95:       {p95(warm_times):.2f}s")
    print(f"concurrent p95: {p95([t for t, _ in concurrent]):.2f}s")
    print(f"error rate:     {error_rate:.1%}")
    print(f"budget met:     {p95(warm_times) <= BUDGET['p95_seconds'] and error_rate <= BUDGET['error_rate']}")


if __name__ == "__main__":
    asyncio.run(main())

Run it once against your local model, once against the MonkeyCode free server, and once against your paid production endpoint. The first run tells you nothing, the second run tells you what your users will actually feel, and the third run tells you what you are paying for. If the free server meets the budget, you do not need the paid tier yet. If it does not, you have two choices: buy more compute and hide the problem, or fix the chain.

In my experience the failure is almost never the model. It is the queue you added to avoid timeouts, the streaming wrapper that buffers the whole response before sending the first byte, or the database write that happens before the answer is returned. The free server exposes all of these because it has no spare capacity to mask them, and that is the entire point. A free server is a worst-case production rehearsal, not a demo environment.

Round-trip result on the free server	What it means	Next move
p95 under budget, no errors	Contract holds at the weakest tier	Stay free until traffic forces you up
p95 over budget, errors low	Latency bug somewhere in the chain	Profile the chain, not the model
Errors high under concurrency	Capacity or timeout design flaw	Fix queue and retry logic before paying

The ten-million-token grant is not a demo allowance; it is a rehearsal budget. Every run of this script spends a few thousand tokens to buy you information about your architecture, which makes it the cheapest load test you will ever run. When the grant runs out, that is a signal to measure again, not a reason to panic. If you burn the whole allowance on notebook benchmarks, you have learned nothing except that the model is fast, which you already knew.

Who should not use this approach? If you are building a real-time voice feature with a hard two-hundred-millisecond budget, a free server is not your rehearsal environment; it is a different product category. If your sustained concurrency already exceeds what a shared free tier can hold, you are past the point where this test helps. And if your compliance rules forbid sending prompts to a third-party endpoint, none of this applies to you. The free server is a rehearsal stage, not a permanent home, and the constraint is the teacher.

So stop benchmarking the model and start benchmarking the round trip. Run this script against the MonkeyCode free server this week, and tell me which layer of the handoff failed first. I want the response code, not the vibes.