Weekend Build Log: Benchmark Your Free AI Endpoint Before You Build On It

Friday, 21:40. Your demo is Monday at ten. You wired your side project to a free AI endpoint last week.

It felt instant back then. Tonight every request stalls for seconds. The free server went to sleep, and you never measured the wake-up time.

This build log fixes the order of operations. Measure the endpoint first. Build the demo second.

Step 0 — Turn a feeling into a decision

You do not need a dashboard. You need one number and one rule.

Cold-start latency decides whether your demo stalls. That is the whole question.

So ask two things. How slow is a cold call? How fast is a warm one?

Write the rule down before you run anything. Mine was simple: no cold call on the first click.

Step 1 — Fix the two measurements

Measure total latency for a tiny completion. Keep the prompt and max_tokens constant.

Skip streaming for now. Streaming gives you time-to-first-token, which is a different metric.

Hold everything else still too. Same machine, same network, same model id, same prompt.

Otherwise you measure your own output length. That is the classic benchmark mistake.

Step 2 — Run the harness against your endpoint

Where does the endpoint come from? One option is MonkeyCode, an open-source coding project with free model access and a free server lane.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator of this account states the free tier includes 10M tokens and a free server option. Those are operator-supplied availability claims, not results I measured. Check the current terms and limits yourself before you plan a weekend around them.

Here is the harness. It is a reference implementation, and it is unexecuted in this article.

# probe.py -- reference harness, not executed here
import csv, json, os, sys, time, urllib.request

BASE = os.environ["PROBE_BASE_URL"].rstrip("/") + "/chat/completions"
KEY = os.environ["PROBE_API_KEY"]
MODEL = os.environ["PROBE_MODEL"]
RUNS = int(os.environ.get("PROBE_RUNS", "10"))
LABEL = sys.argv[1] if len(sys.argv) > 1 else "warm"

payload = json.dumps({
    "model": MODEL,
    "messages": [{"role": "user", "content": "Reply with the single word: ok"}],
    "max_tokens": 8,
    "temperature": 0,
}).encode()

def one_call():
    req = urllib.request.Request(BASE, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    req.add_header("Authorization", f"Bearer {KEY}")
    started = time.perf_counter()
    with urllib.request.urlopen(req, timeout=120) as resp:
        body = resp.read()
    return (time.perf_counter() - started) * 1000, len(body)

rows = []
for i in range(RUNS):
    try:
        ms, size = one_call()
        rows.append({"label": LABEL, "run": i, "ms": round(ms, 1),
                     "bytes": size, "error": ""})
    except Exception as exc:
        rows.append({"label": LABEL, "run": i, "ms": "", "bytes": "",
                     "error": str(exc)[:200]})
    time.sleep(2)

with open("probe_log.csv", "a", newline="") as fh:
    writer = csv.DictWriter(fh, fieldnames=["label", "run", "ms", "bytes", "error"])
    if fh.tell() == 0:
        writer.writeheader()
    writer.writerows(rows)

Set the environment and run it twice. Once with cold, once with warm.

python3 -m venv .venv && source .venv/bin/activate
export PROBE_BASE_URL="https://your-endpoint.example/v1"
export PROBE_API_KEY="..."
export PROBE_MODEL="your-model-id"
python probe.py cold
python probe.py warm

Do not paste a real key into a shared shell history. Use a local .env and a throwaway key.

Step 3 — Schedule the cold run honestly

You do not know the idle window. So guess wide and label the guess.

Stop the server, then wait past your guess. Twelve hours is a reasonable default for a weekend test.

Run the cold batch first. Then run the warm batch immediately after.

Order matters here. Reversing it turns your cold measurement into a warm one.

Step 4 — Read the numbers, not the vibes

Summarize the CSV with percentiles. Averages hide the stall you actually feel.

# summarize.py -- reference, not executed here
import csv, statistics

buckets = {}
for row in csv.DictReader(open("probe_log.csv")):
    if row["ms"]:
        buckets.setdefault(row["label"], []).append(float(row["ms"]))

for label, values in buckets.items():
    values.sort()
    idx = min(len(values) - 1, int(round(0.95 * (len(values) - 1))))
    print(label, "n=", len(values),
          "p50=", round(statistics.median(values)),
          "p95=", round(values[idx]))

Now compare the cold row against the warm row. Here is the decision table I use.

What you see	What it means	What to do
Cold p95 near warm p50	No real sleep penalty	Keep the endpoint, build the demo
Cold p95 is 2-5x warm p50	Visible wake-up cost	Warm it up before the demo, or show a "warming" state
Cold p95 above 10s, or cold errors	First click is broken	Never put it on the first click
Warm p95 far above warm p50	Unstable tail	Cap retries at one, keep a canned response
Errors only on cold runs	Timeouts, not slowness	Raise the client timeout before changing anything else

One run is not a measurement. Ten runs is a hint. A hundred is a study.

Step 5 — Write the decision into the repo

Providers change quietly. Your numbers expire.

So commit a short DECISION.md next to the code. Date it and paste the raw percentiles.

# Endpoint decision (2026-11-09)
- cold p50 / p95: <fill from your CSV>
- warm p50 / p95: <fill from your CSV>
- runs per batch: 10, max_tokens: 8, temperature: 0
- decision: <keep | warm-up before demo | move>
- re-test after: <date>

Six months from now, this file answers the question your logs cannot.

What I skipped, and why

• No Grafana or Prometheus. One CSV answers one question. Dashboards answer questions you do not have yet.
• No CI job. The endpoint changes faster than a pipeline schedule. Run it by hand when you change hosts.
• No load test. You have no users. You have a demo on Monday.
• No multi-provider bake-off. Two providers double the work and halve the clarity.
• No keep-warm cron on day one. A pinger burns tokens and can look like abuse. Measure the cost before you pay it.
• No retry library. One retry plus a canned response is enough for a weekend build.

Each cut saved an evening. None of them changed the decision.

Limitations and who should skip this

This harness measures one machine on one network. Your users sit elsewhere.

Free tiers move without notice. Treat every number here as perishable.

Skip this approach if you need a latency guarantee. Buy one instead. Free lanes make no promises, and a benchmark cannot create them.

Skip it for regulated data too. Send the smallest possible prompt and no real customer text.

And if your app streams tokens, measure time-to-first-token instead. This harness will mislead you there.

The point of the weekend

You started with a feeling about a free endpoint. You ended with one CSV, one table, and one dated decision.

That is a finished side project. It is also the cheapest insurance your demo will ever get.

If you try a free lane, run the harness before you trust it. Then keep the CSV, because the numbers will change.