Last week a teammate told me their new AI coding tool "scored 94% on HumanEval." I asked three questions: Which model version? What sampling temperature? How many runs? The answers were "I don't know," "default," and "one time." That's not a benchmark. That's a screenshot.
A benchmark without a reproducible methodology is marketing. This post walks through a small but honest benchmark you can run on your own machine — or better, on a free server — using free model tokens. You'll end up with a pass@1 number, a cost per task, and a clear decision for your own codebase.
1. Build your own task set
Start with tasks from your real work. Public benchmarks like HumanEval are useful for broad comparisons, but modern models have likely seen them during training. Your recent git history is a better source.
Take the last 20–50 small functions your team wrote. For each, record:
- The function signature
- A description of intended behavior
- A few assert-based tests
- The reference implementation (for your own verification)
Store them in a tasks.jsonl file. Here's a two-task sample:
{"id": "max_of_two", "signature": "def max_of_two(a, b)", "description": "Return the larger of two integers.", "constraints": "No built-in max()", "test": "assert max_of_two(3, 7) == 7\nassert max_of_two(-1, -1) == -1\nassert max_of_two(0, 0) == 0"}
{"id": "is_palindrome", "signature": "def is_palindrome(s)", "description": "Return True if s reads the same forward and backward.", "constraints": "Ignore case and non-alphanumeric characters", "test": "assert is_palindrome('Racecar') == True\nassert is_palindrome('hello') == False\nassert is_palindrome('') == True"}
Twenty tasks is enough for a quick signal. Fifty gives you a number you can defend in a retro.
2. Pick metrics that matter
The metric most people quote is pass@1: the fraction of tasks where the model's code passes all tests in a single attempt. That's a starting point, not the full story.
You also need:
- Latency — median seconds per task
- Token consumption — input + output tokens per task
- Failure patterns — crashes vs. wrong results vs. timeouts
Cost falls out of token usage. If you know your per-token price (or your free-tier allowance), you can compute cost per task and compare against a human's time.
3. Control the variables
A fair evaluation is boring on purpose. Before you start, pin down these settings:
- Use the same model version across all runs. Record its name in your results file.
- Set temperature to 0. If your API doesn't support that, use greedy decoding.
- Use the exact same prompt template for every task. No extra hints mid-run.
- Clear the conversation context after each task—no memory leaking from previous tasks.
- Run the whole suite at least twice to see variance.
- Kill any process that runs longer than 10 seconds. Timeouts are failures too.
These controls make your results comparable and retestable.
4. The benchmark harness
The script below is intentionally dependency-light. It reads your tasks.jsonl, calls an OpenAI-compatible chat completions endpoint, runs generated code in a subprocess, and writes results to CSV.
#!/usr/bin/env python3
import json, sys, time, csv, subprocess, requests
API_URL = "http://localhost:8080/v1" # swap for your endpoint
API_KEY = "YOUR_KEY"
MODEL = "your-model"
PRICE_PER_10K_TOKENS = 0.0 # set this to your real cost
def estimate_tokens(text):
return len(text) // 4
def generate(prompt):
resp = requests.post(
f"{API_URL}/chat/completions",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0
}
)
resp.raise_for_status()
return resp.json()["choices"][0]["message"]["content"].strip()
def run_tests(code, test_code):
with open("solution.py", "w") as f:
f.write(code + "\n" + test_code)
try:
result = subprocess.run([sys.executable, "solution.py"], capture_output=True, timeout=10)
return result.returncode == 0
except subprocess.TimeoutExpired:
return False
def main():
with open("tasks.jsonl") as f:
tasks = [json.loads(line) for line in f if line.strip()]
rows = []
for task in tasks:
prompt = (
f"Write a Python function named {task['signature']}.\n"
f"Task: {task['description']}\n"
f"Constraints: {task.get('constraints', 'none')}\n"
"Return only the code, no explanations."
)
t0 = time.time()
try:
code = generate(prompt)
latency = time.time() - t0
passed = run_tests(code, task["test"])
except Exception as e:
code = ""
latency = time.time() - t0
passed = False
print(f"{task['id']}: error {e}")
input_tokens = estimate_tokens(prompt)
output_tokens = estimate_tokens(code)
rows.append({
"task_id": task["id"],
"passed": passed,
"latency": round(latency, 2),
"input_tokens": input_tokens,
"output_tokens": output_tokens,
})
print(f"{task['id']}: pass={passed} latency={latency:.2f}s")
with open("results.csv", "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=rows[0].keys())
writer.writeheader()
writer.writerows(rows)
total = len(rows)
passed = sum(r["passed"] for r in rows)
avg_latency = sum(r["latency"] for r in rows) / total
total_tokens = sum(r["input_tokens"] + r["output_tokens"] for r in rows)
cost = total_tokens / 10_000 * PRICE_PER_10K_TOKENS
print(f"\nPass@1: {passed}/{total} = {passed/total:.2%}")
print(f"Avg latency: {avg_latency:.2f}s")
print(f"Total estimated tokens: {total_tokens}")
print(f"Estimated cost: ${cost:.4f}")
if __name__ == "__main__":
main()
Run it with python bench.py. The output gives you the numbers in a format you can paste into a PR description.
5. Run it on a free server
Repeating a benchmark requires a place to run it. A laptop works, but a server is better: it stays on, has the same OS every time, and keeps your local environment clean.
One way to get a no-cost setup is to use MonkeyCode's free tier. It ships with 10M free tokens and a free server instance. The API is OpenAI-compatible, so you only need to change API_URL and API_KEY in the script to point at your MonkeyCode workspace. Then you can schedule the benchmark overnight and read the CSV in the morning.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. I verified that the free tier exists; product limits can change, so check their current docs before you plan around them.
Free tiers are perfect for this because your benchmark uses far fewer tokens than a full production workload. Even 1,000 tasks at roughly 1,000 tokens per task consumes only about 1M tokens—leaving room to repeat runs.
6. Read the results like a skeptic
Don't just look at pass@1. Build a simple decision matrix based on your team's tolerance for mistakes:
| Pass@1 | Verdict | Action |
|---|---|---|
| ≥ 0.80 | High confidence | Let it generate code for small, well-tested functions |
| 0.50 – 0.79 | Mixed | Use only for scaffolding or drafts; review everything |
| < 0.50 | Low | Keep it for pseudo-code and exploration, not real tasks |
Add your own thresholds for latency and cost. If a tool is twice as slow but twice as accurate, that tradeoff might be worth it.
Limitations and who should not use this
This benchmark is narrow by design. It tests single-function generation with explicit tests. It won't tell you how a model handles multi-file refactors, PR comments, or vague requirements.
If you're building an agentic workflow where the AI looks at your whole codebase, this harness is not for you. If you need formal verification or security guarantees, you need something stricter than assert-based tests.
Also, your task set can become stale. Update it every few months, because the difference between tools changes as models improve.
The bottom line
Numbers without methodology are marketing. Build your own benchmark, run it on a free server, and make decisions based on evidence instead of a screenshot.
If you want to try this with MonkeyCode's free tier, the script above is ready to go. Your future self—and your team's codebase—will thank you.
Top comments (0)