Every few weeks a new open-weights model drops — the recent wave of releases from teams like MiniMax is a good example — and my timeline fills with benchmark screenshots. The problem: those benchmarks rarely answer the question I actually have, which is "does this model survive the boring tasks in my repo?"
So instead of reading another leaderboard, I built a small, reusable smoke-test harness. It runs the same fixed set of prompts against any OpenAI-compatible endpoint, scores the outputs with cheap deterministic checks, and writes a report I can diff across models. When a new model is announced, I can have a first honest signal in under half an hour — often using free tiers and free hosted compute, so the experiment costs nothing but time.
This post is the harness, the reasoning behind it, and its limits.
Why not just trust the published benchmarks?
Published benchmarks measure aggregate capability. My risks are narrower and weirder:
- Does the model follow my project's formatting conventions (e.g., type hints, no
Any)? - Does it hallucinate imports that don't exist?
- Does it stay coherent on the second turn, when I ask it to modify its own output?
- Is latency acceptable for interactive use, not just batch jobs?
A 20-prompt harness tuned to my actual work answers these better than any public score. And because it's just HTTP calls, it works against any provider that exposes an OpenAI-compatible API — which is now most of them, including the open-weight releases everyone is currently talking about.
Where to run it for free
Two things usually block this kind of experiment: API cost and a place to run the harness. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode happens to remove both blockers for this workflow — it offers free access to a selection of models and a free server option, so you can point the harness at a model you're curious about and run the whole evaluation on hosted compute without a local GPU or a credit card. That's the entire role it plays here; the harness itself works against any compatible endpoint, and you should treat free availability as something that can change, not a guarantee.
There's also a point worth making about the open source spirit here. The reason a harness like this is even possible is that providers converged on open, documented API shapes, and that teams releasing open weights — MiniMax and others — let anyone inspect, host, and poke at their models instead of hiding them behind a black box. MonkeyCode's approach fits the same pattern: free access lowers the barrier to verifying things yourself rather than trusting marketing. Open ecosystems compound because they let outsiders test claims. This harness is my small contribution to that loop — take it, run it against whatever model just launched, and publish your own results.
The harness
The design constraints:
- Fixed prompt set, versioned. Prompts live in a JSON file so results are comparable across models and across time.
- Deterministic scoring where possible. Compile checks, regex checks, import existence — things that don't require a judge model.
- Raw outputs saved. So a human (me) can re-review borderline cases later.
prompts.json — a minimal starter set:
[
{
"id": "py_parse_logs",
"kind": "python_function",
"prompt": "Write a Python function `parse_durations(lines: list[str]) -> dict[str, float]` that reads log lines like 'task=backup duration=1.5s' and returns total seconds per task. Include type hints. No third-party imports.",
"entrypoint": "parse_durations",
"test": "result = parse_durations(['task=a duration=1.5s', 'task=a duration=0.5s', 'task=b duration=2s']); assert result == {'a': 2.0, 'b': 2.0}, result"
},
{
"id": "py_fix_own_code",
"kind": "python_followup",
"prompt": "Now change the function so invalid lines are skipped instead of raising. Return only the updated function.",
"entrypoint": "parse_durations",
"test": "result = parse_durations(['garbage line', 'task=a duration=1s']); assert result == {'a': 1.0}, result"
}
]
smoke.py:
"""Smoke-test an OpenAI-compatible model endpoint against a fixed prompt set.
Usage:
export SMOKE_BASE_URL="https://your-endpoint/v1"
export SMOKE_API_KEY="..."
export SMOKE_MODEL="model-name-as-served"
python smoke.py prompts.json report_dir/
"""
import json
import re
import sys
import time
import urllib.request
from pathlib import Path
def chat(messages: list[dict], base_url: str, api_key: str, model: str) -> tuple[str, float]:
body = json.dumps({
"model": model,
"messages": messages,
"temperature": 0.0,
}).encode()
req = urllib.request.Request(
f"{base_url.rstrip('/')}/chat/completions",
data=body,
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {api_key}",
},
)
start = time.monotonic()
with urllib.request.urlopen(req, timeout=120) as resp:
payload = json.loads(resp.read())
elapsed = time.monotonic() - start
return payload["choices"][0]["message"]["content"], elapsed
def extract_code(text: str) -> str:
"""Pull the first fenced code block, or fall back to the raw text."""
m = re.search(r"```
(?:python)?\n(.*?)
```", text, re.DOTALL)
return m.group(1) if m else text
def run_python_check(code: str, entrypoint: str, test: str) -> tuple[bool, str]:
"""Execute generated code plus its assertion in a throwaway namespace.
NOTE: this executes model output. Run in a disposable environment
(container, free hosted server, VM) — never on a machine with secrets.
"""
namespace: dict = {}
try:
exec(compile(code, "<generated>", "exec"), namespace)
assert callable(namespace.get(entrypoint)), f"{entrypoint} not defined"
exec(compile(test, "<test>", "exec"), namespace)
return True, ""
except Exception as exc: # noqa: BLE001 - we want the failure text in the report
return False, f"{type(exc).__name__}: {exc}"
def main() -> None:
import os
prompts_path, report_dir = Path(sys.argv[1]), Path(sys.argv[2])
report_dir.mkdir(parents=True, exist_ok=True)
base_url = os.environ["SMOKE_BASE_URL"]
api_key = os.environ["SMOKE_API_KEY"]
model = os.environ["SMOKE_MODEL"]
prompts = json.loads(prompts_path.read_text())
results = []
history: list[dict] = []
for item in prompts:
if item["kind"] != "python_followup":
history = [] # fresh conversation per task chain
history.append({"role": "user", "content": item["prompt"]})
try:
reply, latency = chat(history, base_url, api_key, model)
except Exception as exc: # noqa: BLE001
results.append({"id": item["id"], "passed": False, "error": str(exc)})
continue
history.append({"role": "assistant", "content": reply})
code = extract_code(reply)
passed, detail = run_python_check(code, item["entrypoint"], item["test"])
(report_dir / f"{item['id']}.py").write_text(code)
results.append({
"id": item["id"],
"passed": passed,
"latency_s": round(latency, 2),
"detail": detail,
})
passed_count = sum(r["passed"] for r in results)
report = {
"model": model,
"passed": f"{passed_count}/{len(results)}",
"results": results,
}
(report_dir / "report.json").write_text(json.dumps(report, indent=2))
print(json.dumps(report, indent=2))
if __name__ == "__main__":
main()
The whole thing is stdlib-only on purpose: no dependencies means it runs anywhere, including a freshly provisioned free server, with zero setup beyond three environment variables.
How I read the results
The pass count is the least interesting number. What I actually look at:
- Failure mode. A syntax error in extracted code suggests formatting problems; a wrong assertion result suggests reasoning problems; a missing entrypoint suggests instruction-following problems. Different failures disqualify a model for different jobs.
-
Follow-up behavior. The
py_fix_own_codeprompt is where many models quietly degrade — they rewrite the whole thing and break the original behavior, or ignore "return only the updated function." - Latency distribution. One slow response is noise; consistently slow responses rule out interactive use.
-
The saved
.pyfiles. I skim every one. Deterministic checks can't catch code that passes the test but is ugly, fragile, or full of phantom imports elsewhere.
Because each run writes into its own report_dir/, comparing two models — or the same model two months apart — is just diff -ru.
Limitations, and who shouldn't do this
- Twenty prompts is not an evaluation. It's a smoke test. It can disqualify a model for your workflow; it cannot rank models in general. If you need rigorous comparison, look at proper eval frameworks and far larger task sets.
- Executing generated code is dangerous. The harness runs whatever the model emits. Only run it in an environment with no secrets, no network access to internal systems, and no persistent disk you care about. A disposable free server instance is actually a reasonable fit here for exactly this reason.
-
temperature=0.0is not fully deterministic on most serving stacks. Rerun before drawing strong conclusions from a one-run difference. - Free tiers change. Don't build CI that hard-depends on any provider's free model access or free server remaining available; treat them as experiment infrastructure, not production infrastructure. Check the provider's current terms before relying on either.
- If your tasks need judgment-based scoring (design reviews, prose quality), deterministic checks won't get you far — you'd need a judge model or human review, which is a bigger project.
Try it on the next launch
The next time an open model drops and the hot takes start flying, clone this harness, write five prompts from your actual current project, and run it before forming an opinion. Half the value is the harness; the other half is the habit of verifying claims yourself — which is, at its core, what the open source ecosystem is supposed to make easy. If you want a zero-cost sandbox for that first run, MonkeyCode's free model access and free server option are one way to get both the endpoint and the disposable environment in the same place — but the harness doesn't care where you point it.
Top comments (0)