Workshop: Benchmark a New Open-Weight Model in 60 Minutes on Free Tokens and a Free Server

When a new open-weight model drops, the first question is always the same: is it better than the one we already run? The second question is the one most teams skip, and it decides whether the first answer means anything at all. A low model score can mean the model is genuinely weak, or it can mean the prompt template is broken, the reference answers are wrong, or the scoring function rewards the wrong words. This workshop teaches you to build a minimal evaluation harness that separates model failures from harness failures, run it on free tokens, and publish the results from a free server.

You will leave with three artifacts: a runnable harness script, a control test that proves the harness itself works, and a static HTML report your team can open in a browser. The total cost is zero, and the total time is about 60 minutes. The workshop uses the open-source MonkeyCode project for both free resources: a free model endpoint and a free server slot. Disclosure: This article was prepared as part of MonkeyCode's product outreach. As of August 2026 the free tier includes a 10M token allowance, but check the project README before you run the session because quotas and endpoints change.

What you will build

The harness is deliberately small: three prompt cases, a keyword-based scoring function, and a JSONL output file. Small matters here because the workshop's real subject is trust, not coverage. You are not trying to prove that a model is production-ready; you are trying to prove that your measurement is not lying to you.

Timing overview

Exercise	Goal	Time
0	Setup: token, template, endpoint check	5 min
1	Write the harness and scoring function	20 min
2	Run the model and capture raw outputs	15 min
3	Run control tests on the harness	10 min
4	Deploy the report to the free server	10 min

Prerequisites

• Python 3.10 or newer with the openai client library installed
• A MonkeyCode account token for the free model endpoint
• A terminal and about 60 minutes of uninterrupted focus

Exercise 0: Setup (5 minutes)

Create a working directory and a virtual environment, then install the client library. Keep the token in an environment variable so it never lands in a script that gets committed to git.

mkdir model-eval-workshop && cd model-eval-workshop
python -m venv .venv && source .venv/bin/activate
pip install openai
export MONKEYCODE_TOKEN="paste-your-token-here"

Verify that the endpoint answers a one-line prompt before you write any real code. A failed check here saves you twenty minutes of debugging later.

Exercise 1: Write the harness (20 minutes)

Save the following file as eval_harness.py. Replace the endpoint URL and model id with the values from the project README; the script itself is the artifact you keep.

# eval_harness.py
import json
import sys
import time

from openai import OpenAI

client = OpenAI(
    base_url="https://ENDPOINT_FROM_README/v1",
    api_key=sys.argv[1],
)

CASES = [
    {
        "prompt": "Explain, in one sentence, the difference between an index and a primary key.",
        "must_contain": ["index", "primary key"],
    },
    {
        "prompt": "What does idempotent mean? Answer in one sentence.",
        "must_contain": ["same result", "repeat"],
    },
    {
        "prompt": "Name one advantage of serverless functions over long-running VMs.",
        "must_contain": ["scale", "zero", "no server"],
    },
]

def score(output: str, case: dict) -> float:
    text = output.lower()
    hits = sum(1 for token in case["must_contain"] if token in text)
    return hits / len(case["must_contain"])

def main() -> None:
    results = []
    for case in CASES:
        start = time.time()
        response = client.chat.completions.create(
            model="MODEL_ID_FROM_README",
            messages=[{"role": "user", "content": case["prompt"]}],
            temperature=0,
        )
        output = response.choices[0].message.content or ""
        row = {
            "prompt": case["prompt"],
            "score": round(score(output, case), 2),
            "latency_s": round(time.time() - start, 2),
            "output": output,
        }
        results.append(row)
        print(f"score={row['score']:.2f} latency={row['latency_s']}s")

    with open("results.jsonl", "w", encoding="utf-8") as fh:
        for row in results:
            fh.write(json.dumps(row) + "\n")

if __name__ == "__main__":
    main()

The scoring function is intentionally crude: it checks whether required tokens appear in the output. That crudeness is a feature for a workshop because it makes the next exercise meaningful.

Exercise 2: Run the model (15 minutes)

Run the harness against the free model endpoint and save the raw outputs.

python eval_harness.py "$MONKEYCODE_TOKEN"

Look at the JSONL file, not just the printed scores. Read every output line and ask whether a low score came from a wrong answer or from a right answer that used different wording. That distinction is the entire point of the workshop.

Exercise 3: Prove the harness, not the model (10 minutes)

This is the heart of the session. Before you trust any model score, you must prove that the harness can award full marks to a good answer and zero marks to an empty one. Save this as control.py and run it.

# control.py
from eval_harness import CASES, score

def perfect_answer(prompt: str) -> str:
    return (
        "An index speeds up lookups, while a primary key identifies a row. "
        "The same result appears on every repeat, so the operation is idempotent. "
        "Serverless functions scale to zero when no server is running."
    )

def empty_answer(prompt: str) -> str:
    return ""

for case in CASES:
    assert score(perfect_answer(case["prompt"]), case) == 1.0, "harness is too strict"
    assert score(empty_answer(case["prompt"]), case) == 0.0, "harness is too lenient"

print("Control passed: the harness can award full marks and zero marks correctly.")

If the control fails, your scoring function is broken and every model score you collected is meaningless. Fix the tokens in CASES first, then re-run Exercise 2. This is the scenario where a team reports a model failure and later discovers that the scoring function was the failure: nobody tested the measuring stick.

Why the control matters

A harness can look perfect while producing useless numbers, because the harness author usually tests the happy path only. The control test adds the two edges that matter: a definitely good answer must score full marks, and a definitely empty answer must score zero. Any model score you collect before those two assertions pass is noise.

Exercise 4: Deploy the report (10 minutes)

Turn the JSONL results into a static HTML report, then push the folder to the free server using the deploy command from the project README.

python - <<'PY'
import html
import json

rows = [json.loads(line) for line in open("results.jsonl")]
cards = "".join(
    f"<li><b>{row['score']:.2f}</b> — {html.escape(row['prompt'])}<br>"
    f"<code>{html.escape(row['output'][:200])}</code></li>"
    for row in rows
)
open("report.html", "w").write(f"<h1>Model eval report</h1><ul>{cards}</ul>")
print("report.html written")
PY

The free server option accepts a static folder, so the exact deploy flag depends on the current CLI. The important habit is that the report lives at a shareable URL instead of a laptop screen, which makes the review a team activity rather than a solo impression.

What the output should look like

The table below shows the report format, not a measured benchmark; your numbers will differ.

Case	Required tokens	Score	Latency	Verdict
index vs primary key	index, primary key	1.0	2.3s	pass
idempotent	same result, repeat	0.5	1.9s	partial: missing "repeat"
serverless advantage	scale, zero, no server	1.0	3.1s	pass

The useful signal is not the average score but the per-case verdicts. A partial score tells you exactly which wording the model missed, and that is the information you need to decide whether to adjust the prompt or reject the model.

Limitations and who should skip this

• The keyword scorer rewards presence, not reasoning quality. Use rubric-based or pairwise scoring before you make a real adoption decision.
• The free token allowance is for experiments. A 10M-token allowance sounds large but disappears quickly if you scale the case count, so meter your spend as you go.
• Free endpoints have no latency or uptime SLA. Do not point production traffic at them, and do not use this workshop's latency numbers in a procurement document.
• Skip this approach entirely if you need data residency, regulated workload handling, or externally citable benchmark results. This harness is for internal triage, not for published claims.

Where to go next

If this workshop looks useful for your team, the MonkeyCode README lists the current sign-up steps for the free model access and the free server. Run all four exercises once before you schedule the session, because token allowances and endpoint details change over time.