Stop Vibes-Testing AI Coding Models: A Repeatable Evaluation Suite You Can Run for Free

Blake Yang — Wed, 05 Aug 2026 06:55:44 +0000

Most developers evaluate a new AI coding model the same way: open a chat, type "write a REST API", nod at the output, and either subscribe or move on. I have done this too, and it is a terrible method. The output always looks competent on the first prompt, and the model's real weaknesses only show up on the fifth refactor, the ambiguous requirement, or the codebase question it answers with confident fiction.

This article is a small, repeatable alternative: a fixed suite of eight prompts, a scoring rubric, and a harness that records everything so you can compare models (or the same model a month later) on evidence instead of vibes. You can run the whole thing on free tiers — I will note one option below — so cost is not an excuse to skip it.

Why one-shot prompts mislead you

A single prompt conflates three different capabilities:

Fluency — producing syntactically plausible code. Nearly every current model passes this.
Specification fidelity — doing what you actually asked, including the boring constraints ("no external dependencies", "must handle empty input").
Honesty under uncertainty — saying "I don't know this API" instead of inventing one.

Fluency is what a demo measures. The other two are what determine whether the model saves you time or creates debugging debt. Your evaluation suite should probe all three separately.

The suite: eight prompts, three categories

Store these as files so the suite is versioned and re-runnable. Adjust the domain to your actual work — a frontend developer should swap the systems prompts for component tasks — but keep the categories:

evals/
  spec_fidelity/
    01_constrained_api.md      # implement X with explicit constraints
    02_refactor_with_tests.md  # change behavior, keep tests green
    03_boring_edge_cases.md    # empty input, unicode, large input
  honesty/
    04_obscure_library.md      # asks about a niche/deprecated API
    05_ambiguous_requirement.md# under-specified task; does it ask or guess?
  workflow/
    06_debug_this.md           # broken code + failing test output
    07_explain_diff.md         # explain a non-obvious diff
    08_multi_step_plan.md      # plan a migration, then execute step 1

Example prompt, 01_constrained_api.md:

Write a Python function `retry_with_backoff(fn, retries, base_delay)`.
Constraints:
- Standard library only.
- Exponential backoff with full jitter.
- Raise the last exception after retries are exhausted.
- Include type hints and one usage example.
Do not explain the code; output code only.

The constraints are the point. A model that adds tenacity as a dependency or skips jitter has failed specification fidelity even if the code runs.

The harness

The harness is deliberately boring: run each prompt, save the raw response, and fill in a scorecard. Automation can check mechanical constraints (did it import a banned package? does the code parse?); judgment calls (did it ask a clarifying question?) stay manual. Here is a minimal Python sketch — label it as a starting point, adapt it to whatever API you are testing:

# eval_runner.py — adapt endpoint/auth to your provider
import json, subprocess, sys, time
from pathlib import Path

SUITE = Path("evals")
RESULTS = Path("results") / time.strftime("%Y-%m-%d_%H%M")
RESULTS.mkdir(parents=True, exist_ok=True)

def run_prompt(text: str) -> str:
    """Send `text` to the model under test, return raw response.
    Replace the body with your provider's SDK or HTTP call."""
    raise NotImplementedError  # wire up your model endpoint here

def mechanical_checks(prompt_path: Path, response: str) -> dict:
    checks = {"response_nonempty": bool(response.strip())}
    if "constrained_api" in prompt_path.name:
        checks["no_external_import"] = "import tenacity" not in response
        checks["mentions_jitter"] = "jitter" in response.lower() or "random" in response
    return checks

for category in ("spec_fidelity", "honesty", "workflow"):
    for prompt_file in sorted((SUITE / category).glob("*.md")):
        prompt = prompt_file.read_text()
        try:
            response = run_prompt(prompt)
        except NotImplementedError:
            sys.exit("Wire up run_prompt() to your model endpoint first.")
        record = {
            "prompt": str(prompt_file),
            "response": response,
            "checks": mechanical_checks(prompt_file, response),
        }
        out = RESULTS / f"{category}__{prompt_file.stem}.json"
        out.write_text(json.dumps(record, indent=2))
        print(f"saved {out}")

Then score each response 0–2 on the rubric:

Score	Meaning
0	Wrong, fabricated, or ignores constraints
1	Partially correct; usable after real editing
2	Correct and constraint-faithful; minor polish at most

Total per model: 0–16. That number is not a benchmark of the model in general — it is a measurement of this model on work that looks like yours, which is the only measurement that should drive your tooling decisions.

What to actually look for in each category

Honesty prompts (04, 05) are the most revealing. For 04_obscure_library.md, ask about a real but niche API you know well, or an older version of a popular one. A model that hallucinates function signatures here will hallucinate them in your real work, and those bugs are expensive because they compile in your head while you read them. For 05_ambiguous_requirement.md, the best response asks one or two sharp clarifying questions before coding; a model that confidently guesses is a liability on under-specified tickets.

Workflow prompts (06–08) test whether the model can operate on your artifacts — a failing test log, a diff, a partial migration — rather than generating greenfield code, which is where most real usage happens after week one.

Running it for free

Evaluation should not require a paid subscription to the thing you are evaluating — that is backwards. Free tiers and trial credits from the major providers work, and there are also coding platforms that bundle free model access.

One option I used for a run of this suite: MonkeyCode offers free model access and a free server option, which was enough to execute the eight-prompt suite end to end without touching a credit card.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness above is provider-agnostic on purpose — point run_prompt() at whatever endpoint you have, including a local model if you have the hardware for one. If you want to try the suite on MonkeyCode's free tier, the setup takes a few minutes and the results directory slots straight into the comparison step below.

Comparing runs without fooling yourself

When you have two or more results/ folders (different models, or the same model across months):

Compare per prompt, not just totals. A model that wins 14–12 but loses the two honesty prompts is the riskier daily driver.
Read the diffs on prompts where scores differ. The raw responses tell you why; totals never do.
Re-run before concluding. A single run conflates model behavior with sampling luck. Two or three runs per prompt, with the rubric applied blindly if you can manage it, is meaningfully better.

A recent DEV discussion on why sub-agent metrics are not comparable to main-thread metrics makes a related point worth internalizing: numbers only mean something within the context that produced them. Your 0–16 score is valid inside your suite, your rubric, and your domain — do not treat it as a leaderboard entry.

Limitations and who should skip this

Eight prompts is a smoke test, not a benchmark. It will reliably separate "clearly bad fit" from "worth a two-week trial", but it will not rank two good models against each other with confidence. For that, you need real tasks from your issue tracker and a longer evaluation window.
The rubric is subjective at the boundaries. The 1-vs-2 call varies between reviewers. If a team runs this, calibrate on two or three example responses together first.
Free tiers have constraints. Rate limits, queueing, or model availability can affect your runs, and a free tier may not expose the exact model you would pay for. Check what you are actually being served, and do not extrapolate latency measurements from a free server to production expectations.
If your work is highly specialized (embedded C, formal verification, regulated-industry code), a generic suite tells you little. Build the eight prompts from your own domain or skip the exercise and run a supervised trial on real tickets instead.

The takeaway

The model that demos best is rarely the model that fails best. A fixed, versioned prompt suite — even a small one — converts "I tried it and it seemed fine" into a comparable artifact you can rerun whenever a new model, a new pricing tier, or a quiet capability change lands. The suite in this article is a starting skeleton: steal the structure, replace the prompts with your own work, and keep the honesty category no matter what.

If you end up publishing your own version of the suite, I would be curious which category surprised you most.

DEV Community: Blake Yang