Benchmarks Belong on a Server: A Deploy-and-Prove Tutorial

Local benchmarks are contaminated by design. Your network path differs from production. Your laptop sleeps mid-run. Your results are anecdotes with timestamps. Move the harness to a server. Let it run unattended. This tutorial deploys a minimal benchmark runner in six steps. Every step ends with a verification command. Failures point to one layer.

You need three things. A free model endpoint. A server with Docker. SSH access. MonkeyCode's free model access and free server option cover the first two. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What you are building

A small HTTP service. It reads your endpoint URL and key from environment variables. It sends three identical requests on demand. It writes latency and status to JSON. The service runs in Docker. The same image runs locally and on a server.

curl -X POST /run
      |
      v
+-------------+    POST /v1/chat/completions    +------------------+
|  bench API  | ------------------------------> | free model       |
|  (FastAPI)  | <------------------------------ | endpoint         |
+-------------+         JSON response           +------------------+
      |
      v
 /data/results/run-<timestamp>.json

Why a server? Three reasons. Consistent network path. No laptop sleep. No human to start runs. A server turns benchmarks into a time series. That is the whole point.

Before you start

Confirm three facts. Your endpoint accepts the chat payload shape. Your key has permission to call it. Your server has Docker installed. Each fact takes one minute to check. Each saves a debugging session later.

Step 1: Scaffold the harness

Create the project directory.

mkdir bench-server && cd bench-server
mkdir harness

Create harness/main.py. This is the entire service. HTTP calls use only the standard library.

# harness/main.py
import json
import os
import time
import urllib.request

from fastapi import FastAPI

app = FastAPI()
RESULTS_DIR = "/data/results"

MODEL_URL = os.environ["MODEL_URL"]
MODEL_KEY = os.environ["MODEL_KEY"]
PROMPT = os.environ.get("BENCH_PROMPT", "Write a Python function that reverses a list.")


def call_model(payload: dict) -> dict:
    req = urllib.request.Request(
        MODEL_URL,
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": f"Bearer {MODEL_KEY}",
            "Content-Type": "application/json",
        },
    )
    start = time.monotonic()
    with urllib.request.urlopen(req, timeout=60) as resp:
        body = json.loads(resp.read())
    elapsed = time.monotonic() - start
    return {
        "status": resp.status,
        "latency_s": round(elapsed, 3),
        "body": body,
    }


@app.get("/health")
def health():
    return {"ok": True}


@app.post("/run")
def run():
    os.makedirs(RESULTS_DIR, exist_ok=True)
    payload = {
        "messages": [{"role": "user", "content": PROMPT}],
        "temperature": 0,
    }
    results = [call_model(payload) for _ in range(3)]
    path = os.path.join(RESULTS_DIR, f"run-{int(time.time())}.json")
    with open(path, "w") as f:
        json.dump(results, f, indent=2)
    return {"written": path, "results": results}

The payload assumes an OpenAI-compatible chat endpoint. Most free endpoints expose one. If yours differs, change the payload dict. The script does not validate responses. It records what came back. Raw observations are the point.

Create harness/requirements.txt.

fastapi
uvicorn

Pin exact versions in a real project. Unpinned here keeps the tutorial readable.

Step 2: Containerize the service

Create harness/Dockerfile.

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py .
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Create docker-compose.yml at the project root.

services:
  bench:
    build: ./harness
    env_file: .env
    ports:
      - "8000:8000"
    volumes:
      - bench-data:/data

volumes:
  bench-data:

Create .env.example.

MODEL_URL=https://your-endpoint.example/v1/chat/completions
MODEL_KEY=sk-your-key
BENCH_PROMPT=Write a Python function that reverses a list.

Copy it. Fill in real values.

cp .env.example .env

Step 3: Validate locally

Verify the compose file parses.

docker compose config --quiet

Build and start.

docker compose up -d --build

Check the health endpoint.

curl -s http://localhost:8000/health

Expected output: {"ok":true}.

Trigger one run. This needs a real endpoint value in .env. A placeholder URL will fail. That failure is correct behavior.

curl -s -X POST http://localhost:8000/run | jq '.written'

Expected output: a path like /data/results/run-1720000000.json.

Confirm the file exists.

docker compose exec bench ls -la /data/results

This is your baseline. The same image runs on the server.

Step 4: Deploy to a server

You need Docker and SSH on the server. The steps below are server-agnostic. Copy the project to the server.

rsync -av --exclude .env ./bench-server/ user@your-server-ip:~/bench-server/

Or use git. Then SSH in.

ssh user@your-server-ip
cd ~/bench-server

Create the environment file on the server. Never commit .env.

cp .env.example .env
nano .env

Rsync excludes .env on purpose. Your key stays off the repo. If you use git, add .env to .gitignore first.

Paste your real endpoint and key. Start the stack.

docker compose up -d --build

Step 5: Verify the deployed service

From your laptop, hit the server's health endpoint.

curl -s http://your-server-ip:8000/health

Expected output: {"ok":true}.

Open the port in the firewall if it is closed. Use the reverse-proxy domain if one exists. Run one remote benchmark.

curl -s -X POST http://your-server-ip:8000/run | jq '.results[0].latency_s'

Compare this number with your local run. The difference is your network and laptop overhead. That difference is why you deploy.

Step 6: Schedule unattended runs

Add a cron job on the server.

crontab -e

Append this line.

0 */6 * * * curl -s -X POST http://localhost:8000/run >> /var/log/bench-cron.log 2>&1

This runs the benchmark every six hours. Verify the cron job fires.

sleep 60 && tail -n 5 /var/log/bench-cron.log

Then inspect the results directory.

docker compose exec bench ls -la /data/results

Each file is one snapshot. Keep them. They become your trend data.

What the results mean

The JSON records three numbers per request. HTTP status. Latency. The raw body. Look at status first. Non-200 responses are failures. Count them per day. Then look at latency. A rising median means degradation. A rising max means jitter. The raw body tells you why. Maybe a quota message. Maybe a timeout. Maybe a 200 with an error inside. That is why you keep the body.

Latency includes the server's network path. It is not a quality score. It is a liveness probe. Run it for a week. You will see degradation. That is the practical value.

Who should not use this

Do not use this for production traffic. It is a measurement tool, not a gateway. Do not use it for load testing. Three requests are a sample, not a load. Do not use it for rigorous evaluation. That needs seeds, sweeps, and human scoring. This harness gives operational data. Not scientific proof.

Skip this if you need a gateway with retries and fallbacks. Skip this if you need token-level metrics. Skip this if your team needs statistical significance. This is a liveness and latency probe. Nothing more.

Limitations

The payload shape may not match your endpoint. Free endpoints change without notice. Results are snapshots, not guarantees. The server's location biases every latency reading. The script records errors too. That is intentional. Errors are data.

Want a place to run this without renting a VPS? MonkeyCode's free server option is one way to start.