A Two-Model Regression Harness for Evaluating a New Low-Cost Model Release

When a new low-cost model appears, launch-day benchmarks tend to dominate the discussion. Developers rarely need a global leaderboard; they need to know whether the model breaks a critical parse function, emits an unsafe shell command, or changes behavior on a repo-specific bug fix. That answer cannot come from a vendor chart.

This article presents a small regression harness for comparing a candidate model with a baseline on a handful of private, deterministic tasks. The harness is useful when a provider offers free model access or a free server option; MonkeyCode is used here as one example of such a provider. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

No model names, quotas, or ranking numbers are asserted here. The examples use candidate and baseline because model identifiers change quickly; replace them with exact strings from the provider's documentation.

The approach has three parts: a fixed task suite, a uniform runner, and a pass/fail scorecard. A new model is not “better” in this workflow; it is either safe to try on the next task or not yet ready.

1. Define a Small Task Suite

Start with three to ten tasks that have a deterministic check. Good candidates are tasks that already broke in the past or require project-specific conventions.

1.1 Task types

Kind	Prompt	Pass condition
Code completion	Fill a missing function body.	The file compiles and passes one unit test.
Shell command	Convert a plain-language request into a shell command.	The command is non-destructive and returns the expected exit code.
Bug fix	Patch a small defect in a known snippet.	The provided regression test passes.

1.2 Task file format

Store tasks as JSON Lines. Each line contains an ID, a prompt, and the expected check.

{"id":"parse_flags","kind":"code","prompt":"Complete the Python function parse_args(argv) so it returns a namespace with input and verbose. Use argparse. Return code only.","check":"compile"}
{"id":"safe_find","kind":"command","prompt":"Write a shell command that finds Python files modified in the last 7 days and excludes the .venv directory. Return the command only.","check":"non_destructive"}
{"id":"fix_div_zero","kind":"bugfix","prompt":"Fix the function divide(a, b) so it raises ValueError when b is zero. Return the corrected function only.","check":"pytest tests/test_divide.py"}

The checks are deliberately coarse. The goal is to catch obvious regressions first, not to rank models.

2. Use a Uniform Runner

The runner sends each prompt to an OpenAI-compatible chat endpoint and saves the response under runs//.txt. Keeping all outputs on disk makes diffing and manual review straightforward.

import json
import os
import urllib.request

API_BASE = os.environ.get("EVAL_API_BASE", "http://localhost:8000/v1")
API_KEY = os.environ.get("EVAL_API_KEY", "local")
MODEL = os.environ.get("EVAL_MODEL", "candidate")

SYSTEM = (
    "Return only the requested code or command. "
    "Do not add commentary, fences, or explanatory text."
)

def chat(prompt: str) -> str:
    body = json.dumps({
        "model": MODEL,
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": prompt},
        ],
        "temperature": 0,
    }).encode()
    request = urllib.request.Request(
        f"{API_BASE}/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {API_KEY}",
        },
    )
    with urllib.request.urlopen(request, timeout=120) as response:
        payload = json.loads(response.read())
    return payload["choices"][0]["message"]["content"]

def main():
    run_dir = f"runs/{MODEL}"
    os.makedirs(run_dir, exist_ok=True)
    with open("tasks.jsonl", encoding="utf-8") as handle:
        for line in handle:
            task = json.loads(line)
            output = chat(task["prompt"])
            path = os.path.join(run_dir, task["id"] + ".txt")
            with open(path, "w", encoding="utf-8") as out:
                out.write(output)
            print(f"{task['id']}: saved")

if __name__ == "__main__":
    main()

This script uses only the Python standard library. Set EVAL_API_BASE to the provider endpoint. If the candidate model is served through a free server option, point this variable at that server; otherwise use a local endpoint or the provider's normal API.

Run it twice.

export EVAL_API_BASE="https://your-provider.example/v1"
export EVAL_API_KEY="your-key"

export EVAL_MODEL="candidate"
python eval_harness.py

export EVAL_MODEL="baseline"
python eval_harness.py

Then compare the raw outputs before looking at scores.

diff -ru runs/baseline runs/candidate || true

3. Score Pass/Fail, Not Vibes

For each task, record one of three outcomes: pass, fail, or manual_review. The manual review bucket is important for shell commands where a command can be correct but idiomatically risky.

3.1 Code completion check

For a generated function, write the context plus generated body to a temporary file and compile it under the target Python version.

python -m py_compile runs/candidate/parse_flags.txt

A better variant adds a real unit test for the generated function. The pass condition should be executable, not just “looks reasonable.”

3.2 Shell command check

Never execute a generated command directly on the host. First inspect it for destructive patterns, then run it in a container with a read-only mount and a network-disabled profile if possible.

docker run --rm --read-only --network none -v "$PWD/runs/candidate:/work:ro" python:3.12-slim sh -c 'cat /work/safe_find.txt'

This command only prints the candidate output. A destructive-flag scan can be added as a separate check for patterns such as rm -rf /, --force-remove, or shell redirections that overwrite outside the workspace. The scanner is intentionally conservative.

3.3 Bug fix check

Run the project's regression test against the patched file. If the candidate returns a complete function, place it in the expected module and execute the test.

pytest tests/test_divide.py -q

The pass condition is the same test the baseline must pass.

4. Record Results in a Decision Table

After running both models, fill one row per task.

Task	Baseline	Candidate	Notes
parse_flags	pass	pass	Candidate used argparse correctly.
safe_find	pass	manual_review	Correct find command but did not exclude .venv without extra quoting.
fix_div_zero	pass	fail	Returned a comment instead of code.

The decision table makes the next step explicit. If a candidate fails a task that the baseline handles, the migration discussion is simpler: the failing case is now reproducible and specific.

5. Limitations of the Harness

The harness is a smoke test, not a leaderboard.

1. Five to twenty tasks are too few for statistical confidence.
2. Temperature 0 reduces non-determinism but also hides variance that may appear in real use.
3. The prompt format is fixed; changing the system message can produce different results.
4. The harness measures behavior on the included tasks only, not the model's general coding ability.
5. Free server quotas or time limits may require running a smaller batch.

Do not use this approach to claim that a model is “cheaper” or “better” in general. Use it only to decide whether a specific candidate deserves a larger, project-owned evaluation.

6. Who Should Not Use This Workflow

• Teams that already have a CI evaluation suite should integrate the new model there instead of replacing it.
• Teams with no deterministic tasks or no willingness to review shell commands should not run the shell portion.
• Projects that need conversational quality rather than code or command outputs need a different rubric.

The harness adds value when the risk is replacing a code assistant or introducing a new endpoint into a build step, not when the question is broad model preference.

If the motivation is to try a free model endpoint or free server option without committing to a migration, running a ten-task smoke test is a reasonable first gate. Replace the candidate model string with whichever current release is available from the provider's documentation, keep the prompts fixed, and let the pass/fail table, not the announcement, drive the decision.