Everyone on DEV this week is talking about agents, orchestration, and multi-agent pipelines. But there's a boring question that comes before all of that: can the model you picked actually do your task? Most of us answer it by pasting a prompt into a chat UI, squinting at the output, and saying "looks fine." That's not an evaluation — and if you iterate against a paid API, it's also a slow leak of money while you decide.
Here's a cheaper pattern I reach for: a tiny, fixed task set with deterministic checks, run against any OpenAI-compatible endpoint. It takes an afternoon to set up, costs nothing if you point it at a free tier, and gives you a pass/fail table instead of a vibe.
The idea
A prompt regression harness has three parts:
- Fixed tasks — a handful of small coding prompts that represent what you actually ask models to do.
-
Deterministic verification — each generated answer is extracted and executed against real
assertstatements. No human squinting. -
Repeatability — run each task N times at
temperature: 0and report a pass rate. One lucky generation shouldn't count.
Because the harness speaks the OpenAI chat-completions format, you can point it at almost anything: a paid API, a local server like llama.cpp or vLLM, or a hosted free tier.
The harness
Stdlib-only Python. Save as model_harness.py:
#!/usr/bin/env python3
"""Score a coding model against a small fixed task set.
Point it at any OpenAI-compatible endpoint:
export LLM_BASE_URL="https://your-endpoint/v1"
export LLM_API_KEY="..." # or a placeholder for local servers
export LLM_MODEL="model-name"
python model_harness.py
"""
import json, os, re, subprocess, sys, tempfile, time, urllib.request
BASE_URL = os.environ.get("LLM_BASE_URL", "http://localhost:8000/v1").rstrip("/")
API_KEY = os.environ.get("LLM_API_KEY", "none")
MODEL = os.environ.get("LLM_MODEL", "change-me")
RUNS = int(os.environ.get("RUNS", "3"))
TASKS = [
{
"name": "dedupe_keep_order",
"prompt": (
"Write a Python function `dedupe(items)` that returns the list "
"with duplicates removed, preserving first-occurrence order. "
"Reply with only a single python code block."
),
"test": "from solution import dedupe\n"
"assert dedupe([1, 2, 1, 3, 2]) == [1, 2, 3]\n"
"assert dedupe([]) == []\n"
"assert dedupe(['a', 'a', 'b']) == ['a', 'b']\nprint('ok')\n",
},
{
"name": "fizzbuzz",
"prompt": (
"Write a Python function `fizzbuzz(n)` returning the classic "
"FizzBuzz list from 1 to n. Reply with only a python code block."
),
"test": "from solution import fizzbuzz\n"
"assert fizzbuzz(5) == [1, 2, 'Fizz', 4, 'Buzz']\n"
"assert fizzbuzz(15)[14] == 'FizzBuzz'\nprint('ok')\n",
},
{
"name": "parse_keyvals",
"prompt": (
"Write a Python function `parse(s)` that parses a string like "
"'a=1, b = 2 ,c=3' into {'a': '1', 'b': '2', 'c': '3'}. "
"Reply with only a python code block."
),
"test": "from solution import parse\n"
"assert parse('a=1, b = 2 ,c=3') == {'a': '1', 'b': '2', 'c': '3'}\n"
"assert parse('x=y') == {'x': 'y'}\nprint('ok')\n",
},
]
def chat(prompt):
body = json.dumps({
"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0,
}).encode()
req = urllib.request.Request(
f"{BASE_URL}/chat/completions",
data=body,
headers={"Content-Type": "application/json",
"Authorization": f"Bearer {API_KEY}"},
)
with urllib.request.urlopen(req, timeout=120) as r:
return json.load(r)["choices"][0]["message"]["content"]
def extract_code(text):
blocks = re.findall(r"```
(?:python)?\s*\n(.*?)
```", text, re.S)
return blocks[0] if blocks else text
def run_once(task):
code = extract_code(chat(task["prompt"]))
with tempfile.TemporaryDirectory() as d:
with open(os.path.join(d, "solution.py"), "w") as f:
f.write(code)
p = subprocess.run([sys.executable, "-c", task["test"]],
cwd=d, capture_output=True, text=True, timeout=30)
return p.returncode == 0
if __name__ == "__main__":
total_pass = total_runs = 0
for task in TASKS:
passed = 0
start = time.time()
for _ in range(RUNS):
try:
passed += run_once(task)
except Exception as e:
print(f" error on {task['name']}: {e}", file=sys.stderr)
total_pass += passed
total_runs += RUNS
print(f"{task['name']:20s} {passed}/{RUNS} passed "
f"({time.time() - start:.1f}s)")
print(f"\nOverall: {total_pass}/{total_runs} "
f"({100 * total_pass / max(total_runs, 1):.0f}%)")
Run it against a local server, then against a hosted endpoint, and compare the tables. The tasks are deliberately trivial — that's a starting point, not a benchmark. Replace them with prompts pulled from your last month of actual usage and the scores start meaning something.
Safety note: this executes model-generated code. For anything beyond toy tasks, run the subprocess inside a container or disposable VM with no network and no credentials.
Where a free tier fits
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
One practical endpoint option for this workflow is MonkeyCode, which currently offers free model access and a free server option. That combination is genuinely useful for exactly the phase described above: you're iterating on prompts and task definitions, you'll burn a lot of calls, and you don't yet know which model deserves your money. Pointing the harness at a zero-cost endpoint means your experimentation budget is time, not dollars. Set LLM_BASE_URL, LLM_API_KEY, and LLM_MODEL per its docs and the script above works unchanged.
Two honest caveats: free tiers can change their quotas, model lineup, or availability, so treat current terms as the source of truth rather than anything written here — and don't route proprietary code or secrets through any third-party endpoint, free or paid, without checking its data policy.
Decision table: which endpoint for which phase
| Phase | Free hosted tier | Paid API | Self-hosted / local |
|---|---|---|---|
| Exploring whether a model fits your tasks | ✅ Best fit — zero marginal cost per experiment | Wasteful at high iteration volume | Good if you already have the GPU |
| CI prompt-regression on every commit | Fine if rate limits allow | ✅ Predictable quotas | ✅ No external dependency |
| Proprietary / regulated code | ⚠️ Check data policy first | ⚠️ Check data policy first | ✅ Best fit |
| Latency-sensitive production traffic | ❌ Not the right tool | ✅ SLAs exist | ✅ If you can operate it |
| Offline / air-gapped work | ❌ | ❌ | ✅ Only option |
Limitations
- Small samples lie. Three tasks times three runs tells you almost nothing statistically. Treat early scores as directional and grow the task set from real failures you observe.
- Passing tests ≠ good code. A model can pass asserts with unreadable, unmaintainable output. Add human review for anything you'd ship.
- Contamination is real. Classic tasks like FizzBuzz are almost certainly in training data, which inflates scores relative to your novel, private tasks.
- Temperature 0 isn't determinism. Providers can and do return different outputs across runs and versions. Re-run periodically and record results with the model name and date.
- Free-tier availability can change. Don't hard-wire a free endpoint into anything you'd miss if it disappeared tomorrow.
Who should skip this
If you already have an eval framework (like a proper benchmark suite or an LLM-judged pipeline), this is a downgrade. If your decision is purely about production latency or compliance, toy pass rates won't answer it. And if you'd only ever ask a model one question, ever — just use the chat UI.
For everyone else in the "which model is even worth paying for" phase: steal the script, swap in your own tasks, and let a table make the decision instead of a hunch. If you don't have spare GPU capacity or an API budget, MonkeyCode's free model access and free server option are one zero-cost way to get an endpoint for exactly this experiment.
Top comments (0)