DEV Community: Dakota Liu

A New Open Model Dropped. Here's My 30-Minute Reproducible Eval Before I Trust It With My Codebase

Dakota Liu — Mon, 10 Aug 2026 10:51:33 +0000

Every few weeks another open-weight model release lights up my feed — lately the chatter has been around MiniMax's newest entries, and before that it was someone else's. The cycle is always the same: impressive launch charts, a wave of hot takes, and then the quiet question that actually matters for my day job: does this thing help me with my code, in my repo, on my tasks?

I've stopped trusting launch benchmarks for that decision. Not because they're dishonest, but because they're answering a different question. So I built a small, repeatable evaluation habit I run whenever a new model shows up. It takes about 30 minutes, costs nothing if you have access to free model hosting, and — more importantly — it produces evidence I can defend in a PR discussion instead of vibes.

This post is that workflow, including the harness. It's tool-agnostic, but I'll note where I run it.

Why launch-day benchmarks don't answer my question

Public benchmarks measure performance on curated tasks with clean prompts. My real usage is messier:

Multi-file context with stale comments and half-migrated APIs
Instructions that contradict each other ("keep the diff minimal" vs. "also fix the naming")
Codebases with conventions no dataset has ever seen

A model can top a leaderboard and still hallucinate our internal logger's method signature. The only benchmark that predicts that is one I run myself, on tasks sampled from my own backlog.

The 30-minute protocol

Step 1 — Freeze a task set (10 min). I keep a standing file of 8–12 real tasks I recently did by hand: a small bugfix, a refactor with a constraint, a test-writing task, a docstring/README task, and one "tricky" task where I already know the correct answer is subtle. Each entry has the prompt I'd actually give a model and a checklist of what a good answer must do.

Step 2 — Run blind-ish (10 min). I run the same prompts against (a) the new model and (b) whatever model I currently use daily, same temperature, same context. I label outputs A/B before scoring so I'm not grading the new shiny thing more generously.

Step 3 — Score against the checklist, not impressions (10 min). Each task scores 0/1 per checklist item. The totals are usually less dramatic than launch-day discourse suggests — and occasionally they reveal a real regression on a task class I care about.

The artifact: a minimal scoring harness

Here's the actual scaffold I use. It's deliberately boring — plain Python, no framework — so anyone can audit and modify it:

# eval_harness.py — run the same task set against two model endpoints, score by checklist.
# This is a template: plug in your own HTTP calls to whatever inference API you have access to.

import json, time
from dataclasses import dataclass, field

@dataclass
class Task:
    name: str
    prompt: str
    checklist: list[str]          # each item is a human-verifiable requirement

@dataclass
class Result:
    task: str
    model: str
    output: str = ""
    scores: dict = field(default_factory=dict)
    latency_s: float = 0.0

def call_model(model: str, prompt: str) -> str:
    """Replace with your provider call. Keep temperature/system prompt IDENTICAL across models."""
    raise NotImplementedError("wire up your endpoint here")

def run(tasks: list[Task], models: list[str]) -> list[Result]:
    results = []
    for t in tasks:
        for m in models:
            start = time.time()
            out = call_model(m, t.prompt)
            results.append(Result(t.name, m, out, latency_s=time.time() - start))
    return results

def score(results: list[Result], tasks: list[Task]) -> None:
    """Manual pass: for each result, mark each checklist item true/false. Keep it blind if you can."""
    for r in results:
        task = next(t for t in tasks if t.name == r.task)
        r.scores = {item: None for item in task.checklist}  # fill in by hand after shuffling

def report(results: list[Result]) -> None:
    by_model = {}
    for r in results:
        done = [v for v in r.scores.values() if v is not None]
        if done:
            by_model.setdefault(r.model, []).append(sum(done) / len(done))
    for m, accs in by_model.items():
        print(f"{m}: mean checklist pass rate = {sum(accs)/len(accs):.2f} over {len(accs)} tasks")

if __name__ == "__main__":
    tasks = [
        Task("bugfix-null-guard",
             "In this diff, find and fix the null-deref without changing the public API:\n<paste>",
             ["identifies the actual root cause", "no public API change", "adds or updates a test"]),
        Task("write-tests",
             "Write unit tests for this function, covering the two edge cases in the comments:\n<paste>",
             ["tests compile", "both edge cases covered", "no tautological assertions"]),
        # ...add 6-10 more from YOUR recent work
    ]
    # results = run(tasks, ["model-a", "model-b"])
    # score(results, tasks); report(results)
    print("Template only — implement call_model() and add your own tasks.")

Two design choices matter more than the code:

Checklist items are binary and verifiable. "Is good" is not an item. "Doesn't touch the public API" is.
The task set comes from my real backlog, refreshed monthly. Synthetic tasks re-import the same bias benchmarks have.

Decision table: what I do with the results

Outcome	My action
New model wins on ≥2 task classes, no regressions	Trial it for a week on low-risk tasks (tests, docs)
Wins some, regresses on one class I use daily	Keep current model; re-test new model after its next update
Roughly tied	Stay put — switching costs (prompt habits, failure modes I already know) aren't worth zero gain
Worse overall	File the eval away; ignore the discourse

The point of writing this down is that "tied → stay put" is a decision I made in advance, not something I rationalize after being impressed by a demo.

Where free model access fits

Running this protocol against every new release only makes sense if the marginal cost of an eval is near zero. I currently run these trials on MonkeyCode, which offers free model access and a free server option, so spinning up a side-by-side comparison doesn't require a budget approval or a credit card experiment.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The eval protocol above predates that relationship and works with any provider — the harness has no MonkeyCode-specific code in it. What I genuinely appreciate, and the reason I'm comfortable writing this, is the open-source posture: the client-side tooling is open, which means I can read how prompts are constructed and context is assembled instead of trusting a black box. For a workflow whose entire value is trustworthy comparison, that transparency isn't a nice-to-have — it's the difference between an instrument and a toy. If you want to try the protocol without touching your paid quota, their free tier is one low-friction way to do it; a local model or any other free endpoint works just as well.

Limitations, and who should skip this

Sample size is small by design. 8–12 tasks tells you about your work, not the model's general ability. Don't cite your personal eval as evidence about the model overall.
Human checklist scoring is still subjective at the edges. Blinding helps; it doesn't eliminate bias.
Latency and cost at your real context length matter more than eval pass rates for daily use, and I haven't covered measuring those here.
Don't bother if you rarely use AI assistance on code, if your tasks are all greenfield one-file scripts (where most models are fine), or if your compliance constraints already fix which models you're allowed to run.

The actual takeaway

The next time a release trends — MiniMax today, someone else next month — the useful response isn't "is it better than X?" It's "does it beat my current setup on my checklist?" Thirty minutes and a boring Python file answer that. Everything else is launch-day weather.

If you've got a standing eval set of your own, I'd be curious what's in it — especially the tasks where every model consistently fails. Those are the interesting ones.

A Reproducible Harness for Evaluating Free AI Coding Models on Your Own Codebase

Dakota Liu — Mon, 10 Aug 2026 08:27:01 +0000

Most "which AI coding model should I use?" advice is based on someone else's codebase, someone else's prompts, and someone else's tolerance for wrong-but-confident answers. With free model access and free hosted servers becoming common across coding platforms, the real bottleneck is no longer access — it's evaluation. This article shows how to build a small, reproducible harness that scores models against tasks drawn from your repository, so your choice is based on evidence you can rerun.

Why public benchmarks don't answer your question

Public benchmarks measure aggregate performance on curated tasks. Your day-to-day work is narrower and stranger: your naming conventions, your framework versions, your tests. A model that tops a leaderboard can still mangle your ORM migrations or hallucinate your internal utility functions. The fix is cheap: a fixed prompt suite, a fixed scoring rubric, and a runner script you can point at any OpenAI-compatible endpoint.

Step 1: Build a task suite from your own repo

Pick 6–10 tasks that represent what you actually ask a coding assistant to do. Good candidates:

Bug localization: paste a failing test plus the relevant file, ask for the root cause.
Small feature diff: describe a change, ask for a patch in unified diff format.
Refactor with constraints: "extract this function, keep the public signature, no new dependencies."
Test authoring: ask for tests for a function with known edge cases.
Explain-the-code: ask for a concise explanation of a module you understand well, so you can judge accuracy yourself.

Store each task as JSON:

{
  "id": "bugfix-session-timeout",
  "type": "bug_localization",
  "prompt": "Given this failing test and module, identify the root cause and propose a minimal fix...",
  "context_files": ["src/session.py", "tests/test_session.py"],
  "rubric": {
    "correctness": "Identifies the stale-cache bug, not the red herring in retry logic",
    "minimality": "Patch touches fewer than 15 lines",
    "no_hallucination": "Does not reference functions that don't exist in the repo"
  }
}

The rubric is the part most people skip, and it's the part that makes results comparable across models and across runs.

Step 2: A minimal runner script

The harness below is a starting point you can adapt. It assumes an OpenAI-compatible chat-completions API — which most hosted coding platforms, including free tiers, expose in some form. Treat it as a template, not a finished product:

import json, time, pathlib
from openai import OpenAI  # pip install openai

def run_suite(endpoint, api_key, model, suite_path):
    client = OpenAI(base_url=endpoint, api_key=api_key)
    tasks = json.loads(pathlib.Path(suite_path).read_text())
    results = []
    for task in tasks:
        context = "\n\n".join(
            pathlib.Path(f).read_text() for f in task["context_files"]
        )
        start = time.time()
        resp = client.chat.completions.create(
            model=model,
            temperature=0,
            messages=[
                {"role": "system", "content": "You are a careful coding assistant."},
                {"role": "user", "content": context + "\n\n" + task["prompt"]},
            ],
        )
        results.append({
            "task": task["id"],
            "model": model,
            "latency_s": round(time.time() - start, 2),
            "output": resp.choices[0].message.content,
        })
    return results

Key decisions, deliberately:

temperature=0 so runs are as reproducible as the model allows.
Raw outputs are saved, not just scores, so you can audit disagreements later.
Latency is recorded because a slightly better answer that's 4x slower changes your workflow choice.

Step 3: Score against the rubric, not vibes

Score each output 0–2 per rubric criterion (0 = wrong, 1 = partially right, 2 = fully meets criterion). Two passes help:

You score it for correctness and hallucination on tasks where you know the answer.
A stronger model scores it as a second opinion — but only for style and completeness, never as the sole judge of correctness, since model-graded correctness tends to inflate scores for fluent-but-wrong answers.

Step 4: Decide with a table, not a feeling

Here's an empty decision table as a template — fill it with your own results:

Criterion	Weight	Model A	Model B
Rubric score (avg)	40%
Hallucination incidents	25%
Latency p50	15%
Cost (here: free vs. paid)	10%
Self-host / data control	10%

Weights are yours to set. The point is that "Model B felt smarter" becomes "Model B hallucinated on 3 of 8 repo tasks, so it stays out of my bug-fix loop."

Where free tiers fit in

Free model access is exactly what makes this workflow practical: you can run a 10-task suite against three or four candidate models without spending anything, and rerun it monthly as models change. This is also where I've been pointing the harness at MonkeyCode — Disclosure: This article was prepared as part of MonkeyCode's product outreach. It offers free access to coding models plus a free server option, which means you can run the suite against hosted endpoints without standing up your own infrastructure first. The free-server angle matters for a specific reason: evaluation harnesses are chatty, and having a hosted endpoint means your laptop isn't the bottleneck while a suite runs in the background.

A few honest caveats: free tiers change. Rate limits, available models, and server availability can shift without notice, so design your harness to treat endpoints and model names as configuration, not constants. And a free server is not the right place to paste proprietary code without checking the provider's data-handling terms — that decision table row about "data control" exists for a reason.

Limitations and who should skip this

Small suites have high variance. Eight tasks can't rank two close models definitively. Treat results as "good enough for my workflow," not as a leaderboard.
Reproducibility is approximate. Even at temperature 0, hosted models can drift between versions. Keep your raw outputs and timestamps.
This is overkill if you only use AI for throwaway snippets, or if your organization has already mandated a tool — in that case, put the effort into prompt quality instead.
Don't do this against production secrets or code you can't share with the hosting provider, whatever the tier costs.

Closing

The durable skill here isn't picking today's best free model — it's having a harness that tells you, in an afternoon, when the answer changes. If you want a zero-cost starting point for the endpoints, MonkeyCode's free models and free server are one option to plug into the runner above; the suite and rubric are what make the choice yours.