DEV Community

Dakota Huang
Dakota Huang

Posted on

I Stopped Trusting Demo Prompts: A Repeatable Smoke Test for Free Coding Models

A few months ago I caught myself doing something embarrassing: judging coding models by how impressive their first answer looked. I'd paste a clever prompt, watch a confident solution stream out, nod, and move on. Then, a week later, I'd actually use the model on real work and discover it couldn't follow a three-line instruction about my project's import style.

Demo prompts are flattering. They sit comfortably inside training distribution, they have one obvious correct answer, and they never come back to haunt you. Real tasks are uglier: half-specified, style-constrained, and entangled with code the model has never seen. So I built a tiny harness that runs the same fixed set of tasks against any model I want to evaluate, scores the results, and writes everything to a log I can diff later. This post is that harness, plus the workflow around it. It works with any API-compatible model, and it's especially handy for deciding whether a free tier is good enough for a specific job before you wire it into your routine.

The principle: fix the tasks, vary the model

Most ad-hoc model comparisons fail because both variables move at once. Different prompt, different model, different mood — you learn nothing. The fix is boring and effective:

  1. Write down 8–12 tasks that look like your actual work, not interview puzzles.
  2. Freeze them in files, with a frozen prompt template.
  3. Run every candidate model against the identical suite.
  4. Score with a rubric you decided on before looking at any output.

The suite is the asset. Models come and go; your task suite should survive them.

A minimal task suite that isn't trivia

Here's what mine contains. Yours should differ — that's the point — but notice the shape:

  • Constraint-following: "Refactor this function. Do not change its signature. Do not add dependencies." (Models love breaking both rules.)
  • Style-matching: a snippet from my real codebase plus "add error handling in the same style as this file."
  • Bug localization: a small file with one planted bug and a failing test description. Does the model find the actual bug or rewrite everything?
  • Explain-then-act: "First say what this code does in two sentences, then modify it." Checks whether the explanation matches the modification.
  • Deliberate trap: a task that is impossible as stated (missing information). A good model asks or flags it; a bad one hallucinates confidently.

That last one is my favorite. Hallucination under ambiguity is the failure mode that costs me the most time, and demo prompts never test it.

The harness

This is a pared-down version of what I run. It's plain Python, one file, no framework:

import json, time, pathlib, urllib.request

ENDPOINT = "https://your-provider.example/v1/chat/completions"  # any OpenAI-compatible API
MODELS = ["model-a", "model-b"]  # candidates under test
SUITE = pathlib.Path("suite")    # one .md file per frozen task

def run_task(model, task_text):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": task_text}],
        "temperature": 0.2,
    }).encode()
    req = urllib.request.Request(
        ENDPOINT, data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer YOUR_KEY"})
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=120) as r:
        out = json.loads(r.read())
    return out["choices"][0]["message"]["content"], round(time.time() - t0, 1)

results = []
for model in MODELS:
    for task in sorted(SUITE.glob("*.md")):
        answer, secs = run_task(model, task.read_text())
        results.append({"model": model, "task": task.name,
                        "seconds": secs, "answer": answer})
        print(f"{model} x {task.name}: {secs}s")

stamp = time.strftime("%Y%m%d-%H%M")
pathlib.Path(f"results-{stamp}.json").write_text(json.dumps(results, indent=2))
Enter fullscreen mode Exit fullscreen mode

Nothing clever here, and that's deliberate. The scoring happens when I read the JSON with a rubric open in another window: for each answer, 0/1 on "followed explicit constraints," 0/1 on "would I merge this after one edit," and a note when the trap task triggered a hallucination. Ten tasks, two models, maybe twenty minutes of scoring. Because the file is timestamped, I can re-run the identical suite in three months and compare honestly.

A few details that matter more than they look:

  • Temperature is pinned. Otherwise you're comparing dice rolls.
  • Latency is logged. A model that's 15% better but 4x slower changes where I'd use it.
  • Tasks live in files, not in the script. Editing a task means a new suite version, which keeps old runs meaningful.

Where free model access fits this loop

The obvious objection to running suites like this is cost: burning paid tokens on evals feels wasteful. This is where free tiers earn their keep in my workflow — I use them as the screening stage. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Concretely, MonkeyCode currently advertises free access to a set of models and a free server option, and I've been pointing the harness above at that endpoint for the screening pass: the suite costs nothing to run, and any model that clears my rubric there graduates to a head-to-head against whatever paid model I rely on for the same tasks.

The honest framing is that a free tier is an evaluation substrate and a second opinion, not a promise. I don't assume the free lineup is static, I don't assume quotas, and I don't benchmark-claim anything I didn't measure in my own logs. The script above is deliberately provider-agnostic so that if the free option disappears or changes, my suite and history survive intact.

What the results actually changed for me

Running this loop for a while produced conclusions I would not have reached by vibes:

  • Models that felt similar diverged hard on the constraint-following tasks. Instruction adherence turned out to be the most differentiating axis for my work — more than raw correctness.
  • The trap task eliminated one model I otherwise liked. It invented the missing function signature with total confidence, twice.
  • For mechanical jobs (rename-and-propagate, test scaffolding), a free-tier model scored the same as my paid default. I now route those tasks to the free option and save the paid calls for the ambiguous stuff.

That last point is the real payoff: the suite doesn't just rank models, it tells you which tasks are cheap.

Limitations, stated plainly

  • Small-N scoring by hand. Ten tasks scored by one person is a smoke test, not a benchmark. It catches dealbreakers; it does not produce leaderboard numbers.
  • My suite reflects my work. It's biased toward Python, refactoring, and test-adjacent tasks. A frontend-heavy developer needs a different suite.
  • Free tiers move. Availability, models, and limits can change without notice. Anything operational you build must tolerate that — one reason the harness hardcodes nothing about the provider beyond the endpoint.
  • Single-turn only. This doesn't measure agentic, multi-step behavior. I treat it as a gate before deeper testing, not a substitute for it.

Who should skip this

If you only use a coding model a couple of times a month for greenfield snippets, the setup cost outweighs the payoff — just use whatever's in front of you. Likewise, if your work is dominated by long multi-file agent sessions, a single-turn suite measures the wrong thing; you'd want trajectory-level evals instead. And if someone else at your company already maintains an eval suite, extend theirs rather than forking a personal one.

The takeaway

Trust is a test suite. Free model access — including MonkeyCode's free models and free server — makes the screening stage of that suite effectively free to run, which removes the last excuse for judging models on demo magic. If you build one thing from this post, make it the frozen task files; the harness around them is an afternoon of work. If you end up writing your own suite, I'd genuinely like to hear which task surprised you — the trap task is the one I keep recommending people steal.

Top comments (0)