DEV Community: Avery Wang

Feed Your Worst Test to a Free Model

Avery Wang — Mon, 31 Aug 2026 12:24:11 +0000

Every repo has a test that fails at 3 AM. It fails for a reason nobody documented. I used to comment it out and move on. Then I found a better ritual.

I send that failure to a free model. MonkeyCode currently offers free models and a free server. That is enough for a serious experiment. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

But not any experiment. I don't ask for a fix directly. I ask for a root cause first. A failing test is a tiny specification. It tells the model what went wrong. The prompt should force explanation before action.

Why does that order matter? Because a patch without a diagnosis is a guess. A guess might pass one test. A guess will break a different test tomorrow. When the model explains first, it has to trace the data flow. That explanation is the part I can review.

So I built a small loop around my failing test list. The list comes from my last CI run. Each test name becomes a prompt. The prompt includes the failure output and asks for two sentences of root cause plus a diff.

while IFS= read -r test_id; do
  cat > /tmp/model_prompt.txt <<EOF
Test $test_id fails with:
$(grep -A 10 "$test_id" target/test-reports/*.txt 2>/dev/null || true)

Explain the root cause in two sentences. Propose a minimal diff.
EOF

  $MODEL_CMD < /tmp/model_prompt.txt > /tmp/model_patch.diff
  git apply --check /tmp/model_patch.diff \
    && echo "$test_id: patch applies" \
    || echo "$test_id: patch rejected"
done < failing_tests.txt

The script is deliberately boring. MODEL_CMD is the command that talks to your chosen model. I use a free model for this step, so I can run it fifty times without watching a meter. The free server option matters too, because it keeps the queue off my laptop. My laptop is already busy running the test suite.

After the loop, I don't merge anything blindly. I run a second gate. For each applied patch, I rerun the original test. Then I read the diff as if a stranger wrote it.

That is where the acceptance table comes in.

Condition	Verdict
Test passes after patch	Keep for human review
Root cause appears in the model's explanation	Trust a little more
Diff changes only one file	Read slowly, then keep
Diff changes imports or locks	Reject and ask for a second hypothesis
Model says "I cannot determine the cause"	Accept that answer. It is honest.

The table is my own rule, not a law. You should build your own. The point is to make the decision repeatable. A free model becomes useful the moment you stop treating it as an oracle and start treating it as a fast intern who needs a checklist.

This ritual also exposes the cold start problem. Free servers often sleep between requests. The first prompt of the morning can take a minute before the model wakes up. That delay is annoying. It is also informative. If you plan to use a tool in an emergency, you need to know its wake-up time before the emergency.

So I schedule the loop every Monday. The script runs against last week's failed tests. The output lands in a file I skim with coffee. No dashboard, no hype, just a list of patches and verdicts.

Now, the limits. This approach is only for deterministic tests. It is not for architecture debates or design reviews. A pass/fail test has a clear answer. A UX question does not.

Do not paste secrets into the prompt. Do not feed it proprietary failure logs unless you trust the server. A free server is a shared resource. Treat it like a shared clipboard, not a vault.

And who should skip this? Anyone who has zero failing tests. If your suite is green and stays green, you don't need a model to fix what isn't broken. Also skip it if you are debugging a year-old monolith with no test isolation. The model will produce plausible diff after diff, and you will spend the afternoon saying no.

But for a normal project, this takes less than an hour. The free models handle the boring part. The free server runs the loop at 2 AM. The only thing left for me is deciding whether to trust the patch. That decision is mine, and the evidence is in the table.

Next time a test fails, don't shrug. Feed it to the free model and see what happens. At worst, you get a wrong patch and a laugh. At best, you get a diagnosis before your coffee is cold.

Benchmark a Free AI Coding Tier on a Cold Server

Avery Wang — Sun, 30 Aug 2026 11:28:52 +0000

Benchmark a Free AI Coding Tier on a Cold Server

Benchmarking a free AI coding tier on a cold remote server produces a number that marketing screenshots rarely show. Warm local runs hide the two costs that matter most for daily coding, and those are first-token latency and variance across invocation. The workflow below uses a fixed Python harness, a small dataset, and a free server workspace so that every measurement has a reproducible home.

MonkeyCode is an open-source project that currently pairs free model access with a free server workspace, which makes it a convenient host for this method. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The exact token allowance and server conditions change over time, and the current documentation should be checked before any figure is quoted, including the operator's cited ten-million-token allowance. The value here is the methodology, not the number.

The dataset

A useful benchmark dataset is a contract, not a random collection. It should contain three prompt categories, and every prompt must have an explicit pass/fail condition so a human does not judge the output by feeling. The first category is deterministic generation, such as a function that finds the first non-repeated character. The second is regression repair, a short code snippet with a known bug and a test that must pass after the fix. The third is repository context, a recent commit from the reviewer's own project that requires matching the surrounding style.

Example dataset:

CASES = [
    {
        'name': 'first-non-repeated',
        'prompt': 'Write a Python function that returns the first non-repeated character in a string.',
        'check': 'first_non_repeated(',
        'timeout': 90
    },
    {
        'name': 'bug-repair',
        'prompt': 'Fix the bug in this function: def last(items): return items[-0]',
        'check': 'return items[-1]',
        'timeout': 90
    }
]

This is a skeleton, and the check field in a real run should call a compiler or a test runner. A substring check is enough for a smoke test, but it is not a proof of correctness.

The harness

The harness sends each prompt to an API endpoint and records two timestamps. The difference between the first streamed token and the stop event yields the two numbers that matter, first-token latency and total completion time. The script resets the conversation before every call, because a long context from an earlier prompt can silently change both numbers.

import json, os, time
from urllib.request import Request, urlopen

endpoint = os.environ['ENDPOINT']
model = os.environ['MODEL']
api_key = os.environ['API_KEY']
seed = 42

def run_case(case):
    payload = json.dumps({
        'model': model,
        'messages': [{'role': 'user', 'content': case['prompt']}],
        'temperature': 0.2,
        'seed': seed,
        'stream': True
    }).encode()
    request = Request(endpoint, data=payload, headers={'Authorization': 'Bearer ' + api_key})
    start = time.time()
    first_token = None
    total_text = ''
    for line in urlopen(request):
        line = line.decode().strip()
        if not line or not line.startswith('data:'):
            continue
        first_token = first_token or time.time() - start
        total_text += line
    return {
        'case': case['name'],
        'first_token_s': round(first_token, 3),
        'total_s': round(time.time() - start, 3),
        'pass': case['check'] in total_text
    }

This skeleton ignores retries, rate limits, and stream parsing, so treat it as an illustration and add those layers before trusting the results. The important habit is the timestamp placement, because the first token and the last token tell different stories.

Metrics and controls

One number is never enough. The harness should report median and p95 for first-token latency, total time, and a pass rate across three repeated runs. A timeout is a failure, an empty response is a failure, and a rate-limit response deserves its own label because it affects developer trust more than raw speed. Token consumption should be computed from the response body and divided by the free allowance to produce a cost per task, remembering that output tokens usually cost more than input tokens.

Every run needs the same seed, temperature, prompt order, and server session age. A seed does not guarantee identical output across every model, but it gives the report a repeatable starting point. A cold server is the correct control, because a warm server with cached context can inflate the results. The report should also state the region of the server and the date range, because free tiers change frequently.

Why a number is not marketing

Vendor benchmarks usually choose one ideal prompt, a warm connection, and an average across successes. A useful benchmark publishes the dataset, the raw timestamps, and the failure distribution, including the cold-start failures that the average hides. If a free tier fails one call out of ten, the interesting number is the failure, not the success rate, because developers will hit that failure many times per week.

Limitations

This method measures serving behavior and shallow correctness, not the quality of complex refactors. It also measures only one server endpoint from one network path, so the latency of a workspace in another region will differ. The right audience is a solo developer or a small team that wants an honest smoke test before reading another launch post. Teams with production requirements should collect real prompts from their own repositories and run the evaluation for a full week.

Developers who already maintain a golden set of prompts should use that set instead of this skeleton. The benchmark is a starting point, not a final verdict, and it should live in the repository as a script that anyone can rerun. The next time a vendor posts a screenshot, ask for the dataset and the p95, not the average. The free server from MonkeyCode makes this first run fast, and the open-source project's model access gives the benchmark a place to run without burning a personal account.

The Reviewer Is the Untested Component

Avery Wang — Sat, 29 Aug 2026 10:39:57 +0000

Every AI coding tool promotes you to reviewer overnight. Nobody hands you a manual with that promotion. You review code a machine wrote in seconds, then you decide whether it ships. Here is the question nobody asks: have you tested your own verdict? Most of us have not.

I used to review AI output like a pull request from a stranger. I checked style, ran tests, and trusted the diff. That approach has a flaw. A stranger explains intent, while an AI explains nothing. A stranger has a history, while an AI has a probability distribution. My review calibration was built on human behavior, and that calibration is now the untested component in my pipeline.

So I started treating my verdict as a function. Inputs are the prompt, the model, the budget, and the time limit. Outputs are accept, reject, or request repairs. A function that important deserves a test suite. The test suite below runs on free models through the MonkeyCode open-source project, and the free server option handles the remote execution. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two failure modes, two blind spots

False accepts feel rare until production. False rejects feel invisible because you never see the good code you sent back. Most teams notice false accepts, and only after an incident. A calibration run surfaces both on a Tuesday afternoon instead of a pager alert at 3 AM. The method is small, and that smallness is the point.

Build a calibration set first

You cannot measure a verdict without a known answer. Start with ten tasks that have a reference implementation and an executable acceptance check. Keep the tasks small and representative of your daily workload. Save them in a manifest so the prompts stay identical across runs.

{
  "task_id": "tm-014",
  "prompt": "Write a Python function clean_name(s) that strips whitespace, collapses double spaces, and capitalizes the first letter of each word. Do not use regex.",
  "reference": "clean_name('  alice   bob ') == 'Alice Bob'",
  "acceptance": ["python -m py_compile clean_name.py", "pytest test_clean.py -q"]
}

Do not reuse prompts you remember from past failures. Memory makes the experiment dishonest. Your goal is calibrated reviewing, not a trivia score.

Record the verdict, not just the output

The executable gates prove the code works. They say nothing about the code you rejected. That missing signal is your judgment, so log it.

# judgment_log.py — append one verdict per run.
import json, uuid
from datetime import datetime, timezone

def record(entry):
    entry["id"] = uuid.uuid4().hex[:8]
    entry["ts"] = datetime.now(timezone.utc).isoformat()
    with open("judgments.jsonl", "a") as f:
        f.write(json.dumps(entry) + "\n")
    return entry

record({
    "task": "tm-014",
    "model": "free-model-id",   # pin what the provider returns
    "gates_passed": True,
    "my_verdict": "accept",
    "expected": "accept",        # from the reference solution
    "alignment": "true_accept",
    "reason": "tests passed, structure looks clean"
})

Every field earns its place. expected comes from the reference solution, never from your memory. alignment names the cell in the decision table below.

Gates	Your verdict	Reference says	Label	Action
pass	accept	accept	true accept	keep the habit
pass	reject	accept	false reject	relax the rubric
fail	accept	reject	false accept	add this gate to your checklist
fail	reject	reject	true reject	record the pattern

A decision table turns vague feelings into labels. Labels turn into counts, and counts turn into a calibration curve.

Script the gates so the log is honest

Manual gates invite cheating. You will glance at the terminal and move on. Script the checks so the log entry has real signals.

#!/usr/bin/env bash
# gates.sh — run one candidate through automated checks.
set -euo pipefail
TASK="$1"; CANDIDATE="$2"
mkdir -p "out/$(dirname "$TASK")"
if python -m py_compile "$CANDIDATE"; then
  echo "gate:syntax=PASS"
else
  echo "gate:syntax=FAIL"
fi
if pytest "test_$(basename "$TASK")" -q; then
  echo "gate:unit=PASS"
else
  echo "gate:unit=FAIL"
fi

Combine the gate output with your verdict in the same JSONL file. A verdict without gates is an opinion, and gates without a verdict are logs of a machine.

Let the free server remove environment excuses

A local run carries invisible bias. Different machines have different dependencies, and a dirty environment produces failures that look like model failures. The free server option fixes that by running the agent in one remote sandbox. Your job narrows to writing prompts and recording verdicts. This separation keeps the experiment honest.

The free server has limits, and you should log them. Cold starts stretch the first call. Queue time varies during peak hours. Network latency hides inside your timestamps. Record the wall clock and the timestamp, not just the verdict. A run without a timestamp is a tweet, not data. I have not verified model identifiers or server capacity, so pin whatever the provider returns and record it verbatim.

Read the log like code

Open the JSONL file and count the four labels. False accepts and false rejects are your calibration errors. Everything else is noise. Expect the first ten runs to be ugly because you are learning a new habit. Set your threshold before you look at the data, or you will rationalize every miss. One false accept per ten tasks is a safe start, though your workload decides, not mine.

The log teaches you what to check first. Expect one recurring pattern to appear, maybe over-trusting the unit test, maybe ignoring the linter. That pattern is your personal review bias, and now it has a name.

Who should skip this

Teams under a paid service agreement still need provider guarantees. This method suits tasks that fit inside one file, not a large refactor. You also need the skill to write reference implementations, or the experiment inherits your blind spots. If the only thing you want is a quick taste, run a smoke test instead and skip the bookkeeping.

The deliverable is three files

A manifest, a logger, and a gates script. The whole method fits in one repository and costs nothing to run against the free models. Fork it, run the calibration, and publish your curve. The community needs your numbers, not another screenshot.

A Free Tier Benchmark Needs a Test Set, Not a Screenshot

Avery Wang — Fri, 28 Aug 2026 04:07:10 +0000

A free token quota is a budget, and a budget without a burn-rate measurement is just a number in a marketing email. The only honest way to evaluate a free AI coding tier is to run it against a fixed task set with controlled variables and repeatable metrics. This article defines that method and provides a harness that produces numbers worth trusting.

Recent DEV discussions about perfect harness scores and poor model scores point at the real problem, which is that evaluation quality, not model quality, is often the variable being measured. MonkeyCode, an open-source coding project, currently offers a free ten-million-token allocation and a free server option, and that offer is a useful test case for this method. A quota without a burn-rate measurement cannot be evaluated, and the free allocation makes the first benchmark run free as well.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The dataset: fixed tasks from real history

A benchmark is only as honest as its task set, and the best task set is one the model has never seen. Pull twenty small tasks from a real repository's commit history, where each task contains a prompt, a setup command, a test command, and an output filename. The prompt describes the change, the setup installs fixtures, and the test asserts the behavior, so every output either passes or fails without human judgment.

Here is the task format in JSONL, one task per line:

{"id": "fix_off_by_one", "prompt": "Fix the off-by-one error in range_sum so that range_sum(1, 5) returns 15.\n\ndef range_sum(start, end):\n    total = 0\n    for i in range(start, end):\n        total += i\n    return total", "filename": "solution.py", "setup": "true", "test": "python -c \"from solution import range_sum; assert range_sum(1,5)==15; assert range_sum(2,2)==2\"", "category": "bug_fix"}
{"id": "write_fizzbuzz", "prompt": "Write a function fizzbuzz(n) that returns 'Fizz' for multiples of 3, 'Buzz' for multiples of 5, and 'FizzBuzz' for multiples of both.", "filename": "solution.py", "setup": "true", "test": "python -c \"from solution import fizzbuzz; assert fizzbuzz(3)=='Fizz'; assert fizzbuzz(5)=='Buzz'; assert fizzbuzz(15)=='FizzBuzz'\"", "category": "feature"}

Three categories — bug fixes, features, and test writing — keep the set representative, and each task should take an experienced developer under ten minutes. Twenty tasks is the minimum viable sample because it gives a pass rate with a visible failure pattern instead of a single lucky or unlucky run. Sourcing tasks from a repository the team already maintains is the most honest option, because the commit messages describe the intended change and the test suite defines success. Public benchmark sets are convenient but risk contamination, since the model may have seen the exact prompts during training, and a small internal set of twenty tasks is worth more than a thousand public ones for this reason.

Metrics: median, not best

The metrics that matter are pass@k, median tokens per passing task, median wall time, and the failure log. Run every task three times and report the median rather than the best, because the best run is the one the marketing page will quote. Token burn on failed attempts separates an engineering number from a marketing number, because a model that fails twice before succeeding costs three times the budget. The failure log matters more than the pass rate when the tier is free, since a free quota is consumed by failed attempts as well as successful ones, and a model that passes every task on the third try has a perfect pass rate and a terrible token economy.

Controls: fix everything except the model

The harness must fix the model version, temperature, system prompt, and retry policy before the first run, then validate itself with two sanity checks. A known-broken task must fail and a known-good task must pass, because any contradiction means the harness, not the model, is what got benchmarked. Network variance is controlled by running from one region and recording latency per attempt, while a warm-up call absorbs cold-start effects before the first task. The sanity checks belong in the repository next to the task set, so a reviewer can rerun them before trusting any new result, and a harness that cannot fail on a broken task will eventually produce a perfect score for the wrong reason.

The harness

The following Python script reads the task set, calls an OpenAI-compatible endpoint, writes each model output to the target file, runs the setup and test commands, and records every attempt:

#!/usr/bin/env python3
"""Free-tier benchmark harness: fixed tasks, controlled runs, honest numbers."""
import argparse
import json
import os
import subprocess
import tempfile
import time
from pathlib import Path

from openai import OpenAI


def load_tasks(path):
    lines = Path(path).read_text().splitlines()
    return [json.loads(line) for line in lines if line.strip()]


def run_shell(script, cwd, timeout=60):
    try:
        proc = subprocess.run(
            script, shell=True, cwd=cwd,
            capture_output=True, text=True, timeout=timeout,
        )
        return proc.returncode == 0, proc.stdout[-500:] + proc.stderr[-500:]
    except subprocess.TimeoutExpired:
        return False, "timeout"


def solve(client, model, task):
    messages = [
        {"role": "system", "content": "Write only the requested code. No explanations."},
        {"role": "user", "content": task["prompt"]},
    ]
    started = time.monotonic()
    reply = client.chat.completions.create(model=model, messages=messages, temperature=0.2)
    elapsed = time.monotonic() - started
    usage = reply.usage
    return reply.choices[0].message.content, elapsed, usage.prompt_tokens, usage.completion_tokens


def evaluate(client, model, task, workdir):
    code, latency, prompt_tokens, completion_tokens = solve(client, model, task)
    task_dir = Path(workdir) / task["id"]
    task_dir.mkdir(parents=True, exist_ok=True)
    (task_dir / task.get("filename", "solution.py")).write_text(code or "")
    if task.get("setup"):
        ok, log = run_shell(task["setup"], task_dir)
        if not ok:
            return {"pass": False, "stage": "setup", "log": log}
    passed, log = run_shell(task["test"], task_dir)
    return {
        "pass": passed,
        "stage": "test",
        "latency_s": round(latency, 2),
        "prompt_tokens": prompt_tokens,
        "completion_tokens": completion_tokens,
        "log": log,
    }


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--model", default=os.getenv("MODEL", "your-model-id"))
    parser.add_argument("--base-url", default=os.getenv("BASE_URL"))
    parser.add_argument("--api-key", default=os.getenv("API_KEY"))
    parser.add_argument("--tasks", type=Path, default=Path("tasks.jsonl"))
    parser.add_argument("--attempts", type=int, default=3)
    parser.add_argument("--workdir", type=Path, default=Path(tempfile.mkdtemp()))
    args = parser.parse_args()

    client = OpenAI(base_url=args.base_url, api_key=args.api_key)
    summary = {"model": args.model, "attempts": args.attempts, "tasks": []}

    for task in load_tasks(args.tasks):
        attempts = [evaluate(client, args.model, task, args.workdir) for _ in range(args.attempts)]
        passed = sum(1 for attempt in attempts if attempt["pass"])
        tokens = sorted(a["prompt_tokens"] + a["completion_tokens"] for a in attempts)
        latency = sorted(a["latency_s"] for a in attempts)
        summary["tasks"].append({
            "id": task["id"],
            "pass_at_k": passed / args.attempts,
            "median_tokens": tokens[len(tokens) // 2],
            "median_latency_s": latency[len(latency) // 2],
        })
        print(json.dumps(summary["tasks"][-1]))

    rate = sum(t["pass_at_k"] for t in summary["tasks"]) / len(summary["tasks"])
    total = sum(t["median_tokens"] for t in summary["tasks"])
    print(json.dumps({"pass_rate": round(rate, 3), "total_median_tokens": total}))


if __name__ == "__main__":
    main()

Run it against the endpoint of the tier you are evaluating:

export BASE_URL="https://api.example.com/v1"
export API_KEY="your-key"
export MODEL="your-model-id"
python3 benchmark.py --attempts 3

The script prints one JSON line per task and a final summary with the pass rate and median token burn, which is the number the quota announcement never mentions.

Why the numbers are not marketing

A pass rate without a failure log is a claim, and a token count without a task set is a guess, so the report must include both. The output shows whether a tier passes twelve of twenty tasks with two hundred thousand median tokens or eighteen of twenty with eight hundred thousand. Those two profiles demand opposite engineering decisions, and the quota announcement cannot tell them apart. A ten-million-token allocation changes the calculation only when the burn rate is known, because a tier that burns three times the expected tokens on failed attempts consumes the quota three times faster.

Limitations and who should not use this

This benchmark measures well-scoped tasks that a developer can verify in minutes, so it says nothing about long-horizon agentic work, repo-scale refactors, or subjective code quality. Teams that need production guarantees should not route critical paths through a free tier on a twenty-task pass rate, and vendor comparisons require an identical task set to mean anything. The method is a first filter for whether a free tier deserves real project time, not a certification of production readiness.

Running this harness against MonkeyCode's free model access and free server option takes an afternoon, and the resulting JSON says more than the quota announcement ever will.

Free AI Coding Tiers Need a Protocol, Not a Press Release

Avery Wang — Wed, 26 Aug 2026 11:27:00 +0000

A benchmark is only as honest as its protocol. Most published numbers about free AI coding tiers fail that test before the first token is spent. The reason is rarely malice; it is usually a missing control, a cherry-picked task set, or a cost column that nobody kept. This article defines a minimal benchmark protocol with a frozen dataset, three behavioral metrics, and five documented controls. The protocol is deliberately small because a reproducible benchmark beats an impressive one.

The typical vendor post reports a single accuracy figure on tasks chosen after the fact. That is like timing a car on a downhill road with a tailwind. Best-of-N sampling inflates the result further because the model runs each task twenty times and the author keeps the one attempt that passed. Cost is excluded entirely, which matters enormously when the tier is free because free changes the economics of every decision that follows. A benchmark becomes trustworthy only when the dataset is frozen, the metrics are behavioral, and the controls are documented next to the results.

The dataset is the part that most people rush, and it determines whether the result means anything at all. A credible set contains twenty to thirty tasks drawn from real repositories, each with a failing test and a prompt that does not appear in any vendor's marketing material. Popular tasks from public leaderboards should be avoided because they are almost certainly in training data, which turns the exercise into a memorization check. Each task should fit in one context window but require real reasoning, such as implementing a missing function or fixing a subtle off-by-one error.

Three metrics capture the behavior that actually matters: pass rate, token cost per solved task, and wall-clock time per task. Pass rate is pass@1 with temperature zero, meaning the model gets one attempt and the hidden test decides the outcome. Token cost is measured from the prompt and completion counts, then divided by the number of solved tasks so cheap and dumb competes against expensive and smart on equal footing. Wall-clock time matters for free tiers because throttling can make a good model unusable during peak hours, and the median across runs is the right summary statistic.

Five controls separate a protocol from a press release, and each one closes a specific hole that would otherwise invalidate the result. Temperature is fixed at zero, the system prompt is identical across models, and the context window is the same size for every run. Each task is executed five times so variance is reported instead of hidden, and the entire suite runs at the same time of day to control for quota throttling. The final control is the most important one: the harness is published with the results, because an unreproducible benchmark is a story, not a measurement.

Here is the harness, kept deliberately small so that reviewing it takes less time than running it. The endpoint and key come from environment variables, which means the same script works against any OpenAI-compatible API. The three helper functions are provider-specific stubs, because the point of the protocol is the structure, not the SDK glue.

# bench_free.py — a minimal honest benchmark for free AI coding tiers
import json
import statistics
import time
from pathlib import Path

TASKS = Path("tasks.json")
TEMPERATURE = 0.0
RUNS = 5

def load_tasks():
    return json.loads(TASKS.read_text())

def run_task(model, task):
    prompt = f"{task['context']}\n\nImplement the missing function:\n{task['prompt']}"
    start = time.monotonic()
    patch = model.complete(prompt, temperature=TEMPERATURE)
    elapsed = time.monotonic() - start
    passed = run_hidden_test(patch, task["test"])
    return {
        "task": task["id"],
        "passed": passed,
        "seconds": elapsed,
        "prompt_tokens": count_tokens(prompt),
        "completion_tokens": count_tokens(patch),
    }

def summarize(results):
    rows = []
    for task_id in {r["task"] for r in results}:
        runs = [r for r in results if r["task"] == task_id]
        rows.append({
            "task": task_id,
            "pass_rate": statistics.mean(r["passed"] for r in runs),
            "median_seconds": statistics.median(r["seconds"] for r in runs),
            "median_tokens": statistics.median(
                r["prompt_tokens"] + r["completion_tokens"] for r in runs),
        })
    return rows

def main():
    model = load_model_from_env()
    results = []
    for task in load_tasks():
        for _ in range(RUNS):
            results.append(run_task(model, task))
    print(json.dumps(summarize(results), indent=2))

if __name__ == "__main__":
    main()

The tasks.json file is the frozen dataset, and it should be committed to version control before the first run. That way nobody can edit the tasks after seeing the results, which is the most common form of accidental cheating. A minimal entry looks like this, with the context, the prompt, and the hidden test that decides pass or fail.

{
  "id": "repo_007_off_by_one",
  "context": "def last_index(items, target):\n    # returns the last index where target appears, or -1",
  "prompt": "Implement last_index so it handles empty lists correctly.",
  "test": "assert last_index([1, 2, 3, 2], 2) == 3\nassert last_index([], 5) == -1"
}

Then the run is a single command, and the output is a JSON table that can be committed next to the code.

export MODEL_ENDPOINT="https://your-free-tier-endpoint/v1"
export MODEL_KEY="$YOUR_KEY"
python bench_free.py > results.json

A free tier changes the benchmark in one important way: the token allowance becomes part of the measured cost, not a discount applied after the fact. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option, and as of this writing it advertises a ten-million-token allowance that lets the harness above run without a credit card. The free server option matters for benchmarking because it removes your own infrastructure from the measurement, so the numbers reflect the model and not a noisy laptop. Running the harness against that endpoint is the fastest way to see the protocol in action, and the resulting JSON will tell you more than any screenshot. Token allowances and server limits change over time, so the honest move is to read the repository before trusting any number quoted in a blog post.

This protocol has real limits, and teams that ignore them will over-trust the output. A twenty-task dataset cannot detect regressions in long-horizon agentic workflows, where a model fails after fifty steps instead of five. The harness measures single-shot code generation, so it says nothing about interactive debugging, cross-file refactoring, or how a model behaves when the reviewer writes the tests. Teams that need production-grade evidence should run a human-reviewed subset against a baseline model they already trust, rather than treating any automated number as the final word.

The point of a benchmark is not to crown a winner but to make the next decision cheaper, and a frozen dataset with published controls does exactly that. Anyone evaluating a free coding tier can run this protocol in an afternoon and keep the results for the next release. The numbers will age, the models will change, and the protocol will still be there, which is the only property that makes a benchmark worth publishing.

Schedule the Free Model for 2 AM

Avery Wang — Tue, 25 Aug 2026 10:01:20 +0000

Liquid syntax error: Unknown tag 'endraw'

A Token Ceiling Is the Best Prompt Engineering Teacher

Avery Wang — Mon, 24 Aug 2026 20:33:30 +0000

A hard token ceiling is the most effective prompt engineering teacher because it converts vague intuition about efficiency into a measurable design constraint. Most developers write prompts as if tokens were infinite, and those habits survive only because nothing forces a reckoning. A free allowance with a visible meter changes that dynamic completely, which is why constrained environments deserve a place in every AI-assisted workflow.

The Craft of Token Frugality

The argument here is deliberately one-sided: token frugality is a craft skill, not a moral virtue. Teams that treat every prompt as a budget line item discover that shorter prompts often produce sharper results, since the model spends less context on filler and more on the actual task. The free server is the perfect training ground for this craft, because a mistake costs nothing but a few tokens and a retry.

MonkeyCode is an open-source project whose current offer includes free model access, a free server option, and a ten-million-token allowance. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The allowance and server terms can change, so readers should check the project documentation before relying on the numbers. The point of this article is not the allowance itself, but the discipline that a visible ceiling teaches.

The fitting analogy is a novelist working under a strict word count. A writer with an unlimited page budget produces sprawling drafts, while one facing a hard limit learns to compress scenes, cut adverbs, and trust the reader. Prompt engineering behaves the same way: abundance breeds laziness, and scarcity breeds architecture. A ten-million-token ceiling is generous enough for real work, yet tight enough to make waste visible.

A Two-Variant Comparison

The artifact below is a small script that measures the token cost of two prompt variants for the same task. It runs a hidden test command and reports which variant passes. It assumes an OpenAI-compatible chat completions endpoint, so the base URL and payload should be adjusted to whatever the project documents. The script is intentionally minimal, because the goal is a single comparison, not a benchmark suite.

#!/usr/bin/env bash
# frugality_check.sh - compare two prompt variants for the same task
set -euo pipefail

TASK="${1:?usage: frugality_check.sh <task-file> <test-command>}"
TEST_CMD="${2:?usage: frugality_check.sh <task-file> <test-command>}"
BASE_URL="${MONKEYCODE_BASE_URL:-https://api.example.com/v1}"
MODEL="${MONKEYCODE_MODEL:-default}"
MAX_TOKENS="${MAX_TOKENS:-2048}"

run_variant() {
  local label="$1" prompt_file="$2"
  local start end response input_tokens output_tokens result

  start="$(date +%s)"
  response="$(curl -sS "$BASE_URL/chat/completions" \
    -H "Authorization: Bearer ${MONKEYCODE_API_KEY:?set MONKEYCODE_API_KEY}" \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg p "$(cat "$prompt_file")" --arg m "$MODEL" --argjson mt "$MAX_TOKENS" \
          '{model: $m, max_tokens: $mt, messages: [{role: "user", content: $p}]}') ")"
  end="$(date +%s)"

  input_tokens="$(jq -r '.usage.prompt_tokens' <<<"$response")"
  output_tokens="$(jq -r '.usage.completion_tokens' <<<"$response")"
  jq -r '.choices[0].message.content' <<<"$response" > "${label}_output.md"

  if bash -c "$TEST_CMD" >/dev/null 2>&1; then result="PASS"; else result="FAIL"; fi
  printf 'variant=%s result=%s input=%s output=%s seconds=%s\n' \
    "$label" "$result" "$input_tokens" "$output_tokens" "$((end-start))"
}

run_variant "luxury" "${TASK}.luxury.prompt"
run_variant "frugal" "${TASK}.frugal.prompt"

The workflow is simple. Write a task description in a file, then create two prompt variants. The luxury version includes background, examples, and explicit reasoning steps, while the frugal version states only the goal, the input format, and the acceptance criteria. Run the script and compare the two lines of output. The verdict is not which variant wins, but what the token difference reveals about the task itself.

Consider a typical refactoring task. The luxury prompt might read: "You are a senior Python developer with deep experience in async patterns. The codebase below contains a synchronous HTTP client that blocks the event loop under load. Please analyze all call sites, identify every blocking operation, and rewrite the client using asyncio while preserving the existing public interface. Add type hints and explain your changes." The frugal version says: "Rewrite this HTTP client to use asyncio. Keep the public interface identical. Add type hints." Both prompts may pass the test, but the frugal version typically consumes a fraction of the input tokens. It also forces the model to rely on its own knowledge rather than the prompt's flattery.

What the Numbers Actually Say

A task that passes with both variants at similar token counts is probably well-specified, and the frugal prompt should become the default. A task that only passes with the luxury prompt needs more context, and the extra tokens are justified. A task that fails with both variants needs a different approach entirely, and no amount of prompting will fix it. That three-way classification is the real lesson of token frugality.

The token difference itself is also diagnostic. A large gap between luxury and frugal input counts suggests that the prompt contained more flattery than instruction. The model was paying for words that carried no signal. A small gap with a large absolute count suggests that the task genuinely requires detailed context, and the frugal version was simply under-specified. Both readings are useful, and neither appears when the meter is invisible.

Where the Method Stops

The method has limits, and the honest position is that token frugality is not always the goal. Complex code generation tasks often benefit from detailed context, and a frugal prompt that fails is more expensive than a luxury prompt that succeeds. Teams working with regulated data should not send proprietary code to a third-party free server at all, regardless of the allowance. The script is a teaching instrument, not a production tool.

Developers who need guaranteed latency, handle sensitive data, or run large batch jobs should skip the free tier entirely. The free server is best used for one thing: practicing the discipline of saying more with fewer tokens. That discipline transfers directly to paid infrastructure, where every wasted token has a price.

The ten-million-token allowance is not the point, and treating it as a coupon would miss the lesson entirely. The point is that a visible ceiling forces a habit that abundance never teaches. Developers who want to test this discipline can point the script at MonkeyCode's free server and run the comparison on their own task. The next prompt is a chance to practice, and the meter is the only honest judge.

Free AI Coding Credits Are a Benchmark, Not a Gift

Avery Wang — Sat, 22 Aug 2026 15:39:22 +0000

Free AI coding credits are a measurement instrument, and most developers waste them by treating the allowance as a discount. A free gym trial tells you whether the gym fits your schedule, not whether you will get fit, and free tokens work the same way. They reveal how a model behaves on your repository, your tests, and your failure modes, which is the only information that should drive a purchase decision. The correct response to a free tier is not gratitude but a benchmark run.

The common habit of burning free tokens on toy prompts like "write a Fibonacci function" produces a confident but useless verdict. A model that handles a textbook prompt gracefully can still collapse on a legacy codebase with ambiguous error messages and a flaky test suite, and the reverse is equally possible. A free tier is the cheapest way to discover which case applies to you, because the cost of failure is zero and the data is specific to your code. Treating the allowance as a benchmark changes the entire exercise from entertainment to engineering.

This is where an open-source assistant like MonkeyCode becomes a useful case study rather than a product announcement. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project's current offer includes free model access with ten million tokens and a free server option, which means a developer can run a serious evaluation without provisioning a GPU or paying for a subscription. The free server is the more interesting half of that offer because it removes the hardware excuse that usually delays honest testing.

A three-phase harness

The workflow that makes sense under this constraint has three phases: harvest real tasks from your own history, run them through a fixed prompt template, and score the answers with a deterministic script. Harvesting from git log and failing tests guarantees that the evaluation reflects your actual workload instead of the model's training data, and a fixed prompt template guarantees that differences between runs come from the model or the server rather than from your phrasing. Deterministic scoring keeps the whole exercise honest, because a human reading twenty answers will unconsciously favor the prettiest one. The script below is a template, not a finished product, and it assumes an OpenAI-compatible chat endpoint, which is the shape most assistant servers expose; the exact URL, model name, and authentication for MonkeyCode's free server are documented in the project README.

#!/usr/bin/env bash
# free-tier-harness.sh — turn free AI credits into a reproducible benchmark
# Usage: ./free-tier-harness.sh /path/to/repo
set -euo pipefail

REPO="${1:?pass a repository path}"
ENDPOINT="${ENDPOINT:-http://localhost:8080/v1/chat/completions}"
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT

# Phase 1: harvest real tasks from the repository's own history
git -C "$REPO" log --oneline -20 --grep='fix' > "$TMP/tasks.txt"

# Phase 2 and 3: fixed prompt, then a request to the assistant server
while IFS= read -r task; do
  prompt="The commit below claims to fix something. Identify the likely root cause and name the file, function, or command you would inspect first. Commit: $task"
  payload=$(jq -n --arg p "$prompt" \
    '{model:"free-tier",messages:[{role:"user",content:$p}]}')

  if ! curl -s "$ENDPOINT" \
    -H 'Content-Type: application/json' \
    -d "$payload" > "$TMP/response.json"; then
    printf 'FAIL %s -> request error\n' "$task"
    continue
  fi

  # Phase 4: deterministic scoring — concrete artifacts or nothing
  jq -r '.choices[0].message.content // empty' "$TMP/response.json" \
    | grep -E -o '([A-Za-z0-9_/-]+\.(py|js|ts|go|rs|sh))|(git|npm|pip|docker) [a-z-]+' \
    | sort -u > "$TMP/hits.txt" || true

  if [[ -s "$TMP/hits.txt" ]]; then
    printf 'PASS %s -> %s\n' "$task" "$(paste -sd, "$TMP/hits.txt")"
  else
    printf 'FAIL %s -> no concrete artifact named\n' "$task"
  fi
done < "$TMP/tasks.txt"

The script does four things in sequence. It pulls the last twenty commits that mention a fix, which produces tasks that are small enough to evaluate but real enough to matter, and it wraps each task in the same prompt template so the model receives identical instructions every time. It sends the request to the local or remote server, stores the raw response, and then scores the answer by checking whether the response names a file, a function, or a command from your stack. A response that names concrete artifacts is not necessarily correct, but one that names nothing is certainly useless, and that asymmetry is exactly what makes the score meaningful.

The free server changes the economics of this test in a subtle way. Running the harness against a remote server means your results are reproducible on a machine you do not own, which is the same condition most CI systems impose, and it also means the prompt leaves your machine. The repository you feed into the harness must therefore be scrubbed of secrets and customer data before the first request, and a local model avoids that concern but reintroduces the hardware cost. The free server is the option that lets a developer decide which constraint matters more instead of being forced into one by budget.

Where this approach breaks

Several limitations should stop you from overreading the results. The ten-million-token figure and the free server are the operator's current offer rather than a contractual guarantee, and free tiers in general have a history of changing without notice, so any verdict you record should include the date and the offer details. The harness measures one dimension of assistant quality, namely whether answers reference concrete artifacts, and it says nothing about code correctness, security, or long-session behavior. A model that passes this test can still fail in a long refactoring session, and a model that fails it can still be useful for boilerplate generation.

Teams with strict data-residency rules should not send repository content to any remote server, including this one, until they have read the project's privacy documentation, and developers who need a guaranteed response time should not build on a free server because free infrastructure rarely carries a service-level agreement. Anyone who is not willing to read the README before running the harness should probably pay for a managed product instead, because the free option rewards exactly the kind of careful reading that the toy-prompt crowd skips. The point of the exercise is not to praise a specific project but to establish a habit that outlives any single free tier.

When the next assistant appears with a generous allowance, the same three-phase harness can be pointed at it within minutes, and the verdict will mean something because the tasks came from your repository. That is the actual value of free credits, and it is worth more than the tokens themselves. If you want to run this harness against MonkeyCode's free tier, the project is open source and the server details are in its documentation.

Your AI Coding Budget Should Start at Zero

Avery Wang — Fri, 21 Aug 2026 13:32:50 +0000

Your AI Coding Budget Should Start at Zero

The most expensive mistake a team can make with AI coding tools is paying for one before it has a baseline. Free model access and free server capacity turn that baseline into a weekend project rather than a procurement cycle, changing which assistant deserves a budget. This article defends a simple position: free tiers are the correct environment for evaluating code generation, and the patch loop below runs entirely on them.

Vendor benchmarks describe tasks that most teams do not actually have. A model can score well on generic coding puzzles and still fail on a codebase with unusual conventions, legacy constraints, or a particular test harness. The only evidence that should influence a purchase is a regression run against your own tests, repeated with the prompts your engineers actually type.

The default evaluation workflow makes this evidence almost impossible to collect. Teams usually ask a few engineers to try a model for a week, which produces anecdotes rather than measurements, filtered through the excitement of a new tool. Free infrastructure removes that bias because it removes the sunk cost; nobody justifies an unbought subscription, and a failed experiment costs a few hours of setup. An evaluation on free resources is also reproducible, which means a colleague can rerun the exact experiment without asking anyone for approval.

MonkeyCode fits this workflow because it is an open source project with free model access, a ten-million-token allowance, and a free server option. That is enough for a persistent evaluation harness on a small repository. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The claim is narrow and testable: schedule a weekly patch evaluation on that free server, record results in a CSV, and let the trend decide the budget.

The artifact below is a minimal patch evaluation loop, written as a template because the model endpoint will differ for every reader. It clones a repository, sends a prompt to a model client, applies the returned diff, runs the test suite, and appends one result line per attempt.

#!/usr/bin/env bash
# patch_eval.sh — evaluate a model-generated patch against your own tests
set -euo pipefail

PROMPT_FILE="${1:-prompt.txt}"
REPO_URL="${REPO_URL:?set REPO_URL to your repository}"
WORKDIR="$(mktemp -d)"
trap 'rm -rf "$WORKDIR"' EXIT

git clone --quiet "$REPO_URL" "$WORKDIR"
cd "$WORKDIR"

# Placeholder for your model client; adapt this block to your endpoint.
python3 - "$PROMPT_FILE" <<'PY'
import sys
prompt = open(sys.argv[1]).read()
# response = your_model_client.complete(prompt)
# open("patch.diff", "w").write(response)
PY

if [ ! -s patch.diff ]; then
  echo "FAIL,no_patch,$(date -u +%F)" >> results.csv
  exit 1
fi

if git apply patch.diff && pytest -q >/dev/null 2>&1; then
  echo "PASS,$(date -u +%F)" >> results.csv
else
  echo "FAIL,test_regression,$(date -u +%F)" >> results.csv
fi

The script is deliberately boring, and that is the point, because evaluation harnesses earn their value from consistency rather than cleverness. A cron entry on the free server turns the script into a weekly experiment that accumulates evidence without any human attention. The prompt file should contain a real task from your backlog, ideally one that touches the code areas where your team feels the most pain.

0 3 * * 1 cd ~/eval && ./patch_eval.sh prompt.txt >> eval.log 2>&1

After a month of runs, the results file tells a story that no benchmark page can tell. If the pass rate stays above a pre-defined threshold, the subscription is justified by local evidence, and a downward drift appears before any contract renews. The same loop also catches provider-side changes, because a sudden drop in the pass rate becomes visible in the CSV within a week.

The threshold itself deserves the same care as the harness. Expecting a perfect pass rate will disappoint, since a model that sometimes emits a broken diff can still save hours on the tasks it handles well. A better rule compares the model's pass rate against the team's manual baseline, recording how often a human patch passes the suite on the first attempt. That comparison turns the evaluation from a pass-or-fail verdict into a productivity argument, which is the evidence a budget review actually needs.

The limitations are real and worth stating plainly. Free tiers typically carry rate limits and shared resources, so this approach suits small repositories, a handful of prompts, and weekly cadences rather than production load testing. Teams with strict data residency requirements should not send code to any external endpoint without legal review. Teams that need guaranteed response times should treat the free server as an evaluation environment rather than a runtime. The method is also only as good as the prompt file and the test suite, so a weak suite produces a confident but meaningless pass rate.

The position defended here is straightforward: the default budget for AI coding tools should be zero until local evidence says otherwise. Free model access and a free server are enough to gather that evidence, and a scheduled harness protects a team from benchmarks that never measured their code. Anyone willing to start at zero can run this experiment today, and the MonkeyCode free tier is a convenient place to begin.

Before You Trust MiniMax H3, Run This Free Baseline Harness

Avery Wang — Mon, 17 Aug 2026 12:37:17 +0000

Benchmarks are useful only when they predict behavior on a team's own code, and most public tables do not reach that bar.
A recent MiniMax H3 release has been moving through developer feeds, but a model release is not yet a reason to change a workflow.
The practical next step is to compare the new model against a free baseline on a fixed set of failing tests.
This article describes a small reproducible harness that does exactly that without vendor lock-in.

The account's earlier evaluations used a two-model regression harness, and this version adds one constraint that matters for small teams.
MonkeyCode's free model access and free server option remove the budget excuse for skipping a controlled comparison.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The free endpoint is used here only as the baseline, while the MiniMax H3 response becomes the candidate.
No benchmark score is copied from a public leaderboard, because leaderboards do not see a team's hidden edge cases.

Think of a model evaluation as a code review, not as a race.
A reviewer does not ask which candidate once won a contest; the reviewer asks which candidate passes the current patch under the project's lint rules and test suite.
The harness treats both models as command-line responders, so the evaluation can be rerun whenever a prompt changes.
Keeping the comparison in git means every conclusion has an audit trail.

The five-minute baseline

Create a project directory, install pytest, and leave two environment variables with placeholder adapter commands.
The following commands build a clean workspace and keep the baseline and candidate outputs separate.
Each model receives the same prompt through standard input, and each response is stored as a timestamped JSON line.
The shell commands do not depend on any provider SDK, which keeps the evaluation portable.

mkdir model-eval && cd model-eval
python -m venv .venv && source .venv/bin/activate
pip install pytest
mkdir -p tasks runs baseline candidate tests

Create a concrete task that can fail in ordinary code generation.
The task asks for a small parser and a test for invalid input, which makes a generated answer easy to judge.
A broad prompt would hide differences, while a narrow one exercises the same parsing edge cases that break in real projects.

cat > tasks/code_task_1.txt <<'EOF'
Write a Python function `parse_time` that accepts a string such as "1h20m"
and returns an integer number of minutes. Include a short test suite for invalid input.
EOF

The harness below reads a prompt file and invokes both adapters from environment variables.
It writes the full stdout, stderr, and return code to runs/, then stores separate text copies in baseline/ and candidate/ for easy diffing.
The script avoids hard-coding any vendor API because a model comparison should not require rewriting the runner when a provider changes.

import json
import os
import shlex
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path

def run_adapter(env_name: str, prompt: str) -> dict:
    raw = os.getenv(env_name)
    if not raw:
        raise SystemExit(f"Set {env_name} before running the harness.")
    parts = shlex.split(raw)
    proc = subprocess.run(
        parts,
        input=prompt,
        text=True,
        capture_output=True,
        check=False,
    )
    return {
        "stdout": proc.stdout,
        "stderr": proc.stderr,
        "returncode": proc.returncode,
    }

def main() -> None:
    if len(sys.argv) != 2:
        raise SystemExit("Usage: python run_pair.py tasks/code_task_1.txt")
    prompt = Path(sys.argv[1]).read_text()
    baseline = run_adapter("BASELINE_CMD", prompt)
    candidate = run_adapter("CANDIDATE_CMD", prompt)
    record = {
        "task": sys.argv[1],
        "run_at": datetime.now(timezone.utc).isoformat(),
        "baseline": baseline,
        "candidate": candidate,
    }
    runs_dir = Path("runs")
    runs_dir.mkdir(exist_ok=True)
    json_path = runs_dir / (Path(sys.argv[1]).stem + ".json")
    json_path.write_text(json.dumps(record, indent=2), encoding="utf-8")
    base_dir = Path("baseline")
    cand_dir = Path("candidate")
    base_dir.mkdir(exist_ok=True)
    cand_dir.mkdir(exist_ok=True)
    base_path = base_dir / Path(sys.argv[1]).name
    cand_path = cand_dir / Path(sys.argv[1]).name
    base_path.write_text(baseline["stdout"], encoding="utf-8")
    cand_path.write_text(candidate["stdout"], encoding="utf-8")
    print(f"wrote {json_path}, {base_path}, {cand_path}")

if __name__ == "__main__":
    main()

An adapter keeps the harness independent of a particular SDK.
When an endpoint exposes an OpenAI-compatible chat route, the same small adapter can drive both the free baseline and the candidate.
The adapter reads a prompt from standard input, sends it as a user message, and prints the assistant reply.

import argparse
import json
import sys
import urllib.request

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--base-url", required=True)
    parser.add_argument("--model", required=True)
    args = parser.parse_args()
    prompt = sys.stdin.read()
    payload = {
        "model": args.model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }
    request = urllib.request.Request(
        args.base_url.rstrip("/") + "/chat/completions",
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(request, timeout=120) as response:
        body = json.load(response)
    print(body["choices"][0]["message"]["content"])

if __name__ == "__main__":
    main()

Run both responses through the same adapter and compare the raw text before evaluating the code.
Set the endpoint and model values from the provider's current documentation, because the harness deliberately avoids embedding secrets or stale URLs.
A visible diff exposes formatting differences, code-block padding, and refusal patterns that numeric scores often bury.

export BASELINE_CMD="python adapters/http_chat.py --base-url $BASELINE_URL --model $BASELINE_MODEL"
export CANDIDATE_CMD="python adapters/http_chat.py --base-url $CANDIDATE_URL --model $CANDIDATE_MODEL"
python run_pair.py tasks/code_task_1.txt
diff -u baseline/code_task_1.txt candidate/code_task_1.txt || true

For a code task the next step is to extract the Python block from the candidate output and run a test against it.
The sample test below checks the format promised by the task, including an invalid input that should raise instead of returning a wrong total.
This is the point where a leaderboard score becomes an engineering result.

cat > tests/test_parse_time.py <<'EOF'
import pytest
from candidate_solution import parse_time

def test_basic_minutes():
    assert parse_time("1h20m") == 80

def test_minutes_only():
    assert parse_time("45m") == 45

def test_invalid_input_raises():
    with pytest.raises(ValueError):
        parse_time("tomorrow")
EOF
python - <<'PY'
from pathlib import Path
text = Path("candidate/code_task_1.txt").read_text()
code = text.split("```

python")[-1].split("

```")[0]
Path("candidate_solution.py").write_text(code)
PY
pytest -q tests/test_parse_time.py

The open-source value here is not a marketing label; it is the ability to read the diff and rerun the same input.
When a model's score comes from a private harness, readers inherit someone else's conclusion.
When the harness is a few Python files, readers can change the task, add a failing test, and repeat the comparison without paying for access.
A free baseline is therefore not merely a cost decision; it makes independent verification cheaper than trust.

This harness does not measure latency under load, long-context retrieval, safety, or license compatibility.
The results only cover the chosen prompt set, which means a different team may see different behavior.
The free baseline availability and quota are operator-supplied and should be confirmed from current MonkeyCode documentation before relying on them.
Teams that require a private on-premises model, guaranteed support, or strict output licensing should not treat a free server evaluation as a production readiness test.
Developers who want the exact endpoint values should check MonkeyCode's current documentation; the harness itself assumes nothing beyond a chat route.

A Two-Model Regression Harness for Evaluating a New Low-Cost Model Release

Avery Wang — Fri, 14 Aug 2026 11:33:02 +0000

When a new low-cost model appears, launch-day benchmarks tend to dominate the discussion. Developers rarely need a global leaderboard; they need to know whether the model breaks a critical parse function, emits an unsafe shell command, or changes behavior on a repo-specific bug fix. That answer cannot come from a vendor chart.

This article presents a small regression harness for comparing a candidate model with a baseline on a handful of private, deterministic tasks. The harness is useful when a provider offers free model access or a free server option; MonkeyCode is used here as one example of such a provider. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

No model names, quotas, or ranking numbers are asserted here. The examples use candidate and baseline because model identifiers change quickly; replace them with exact strings from the provider's documentation.

The approach has three parts: a fixed task suite, a uniform runner, and a pass/fail scorecard. A new model is not “better” in this workflow; it is either safe to try on the next task or not yet ready.

1. Define a Small Task Suite

Start with three to ten tasks that have a deterministic check. Good candidates are tasks that already broke in the past or require project-specific conventions.

1.1 Task types

Kind	Prompt	Pass condition
Code completion	Fill a missing function body.	The file compiles and passes one unit test.
Shell command	Convert a plain-language request into a shell command.	The command is non-destructive and returns the expected exit code.
Bug fix	Patch a small defect in a known snippet.	The provided regression test passes.

1.2 Task file format

Store tasks as JSON Lines. Each line contains an ID, a prompt, and the expected check.

{"id":"parse_flags","kind":"code","prompt":"Complete the Python function parse_args(argv) so it returns a namespace with input and verbose. Use argparse. Return code only.","check":"compile"}
{"id":"safe_find","kind":"command","prompt":"Write a shell command that finds Python files modified in the last 7 days and excludes the .venv directory. Return the command only.","check":"non_destructive"}
{"id":"fix_div_zero","kind":"bugfix","prompt":"Fix the function divide(a, b) so it raises ValueError when b is zero. Return the corrected function only.","check":"pytest tests/test_divide.py"}

The checks are deliberately coarse. The goal is to catch obvious regressions first, not to rank models.

2. Use a Uniform Runner

The runner sends each prompt to an OpenAI-compatible chat endpoint and saves the response under runs//.txt. Keeping all outputs on disk makes diffing and manual review straightforward.

import json
import os
import urllib.request

API_BASE = os.environ.get("EVAL_API_BASE", "http://localhost:8000/v1")
API_KEY = os.environ.get("EVAL_API_KEY", "local")
MODEL = os.environ.get("EVAL_MODEL", "candidate")

SYSTEM = (
    "Return only the requested code or command. "
    "Do not add commentary, fences, or explanatory text."
)

def chat(prompt: str) -> str:
    body = json.dumps({
        "model": MODEL,
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": prompt},
        ],
        "temperature": 0,
    }).encode()
    request = urllib.request.Request(
        f"{API_BASE}/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {API_KEY}",
        },
    )
    with urllib.request.urlopen(request, timeout=120) as response:
        payload = json.loads(response.read())
    return payload["choices"][0]["message"]["content"]

def main():
    run_dir = f"runs/{MODEL}"
    os.makedirs(run_dir, exist_ok=True)
    with open("tasks.jsonl", encoding="utf-8") as handle:
        for line in handle:
            task = json.loads(line)
            output = chat(task["prompt"])
            path = os.path.join(run_dir, task["id"] + ".txt")
            with open(path, "w", encoding="utf-8") as out:
                out.write(output)
            print(f"{task['id']}: saved")

if __name__ == "__main__":
    main()

This script uses only the Python standard library. Set EVAL_API_BASE to the provider endpoint. If the candidate model is served through a free server option, point this variable at that server; otherwise use a local endpoint or the provider's normal API.

Run it twice.

export EVAL_API_BASE="https://your-provider.example/v1"
export EVAL_API_KEY="your-key"

export EVAL_MODEL="candidate"
python eval_harness.py

export EVAL_MODEL="baseline"
python eval_harness.py

Then compare the raw outputs before looking at scores.

diff -ru runs/baseline runs/candidate || true

3. Score Pass/Fail, Not Vibes

For each task, record one of three outcomes: pass, fail, or manual_review. The manual review bucket is important for shell commands where a command can be correct but idiomatically risky.

3.1 Code completion check

For a generated function, write the context plus generated body to a temporary file and compile it under the target Python version.

python -m py_compile runs/candidate/parse_flags.txt

A better variant adds a real unit test for the generated function. The pass condition should be executable, not just “looks reasonable.”

3.2 Shell command check

Never execute a generated command directly on the host. First inspect it for destructive patterns, then run it in a container with a read-only mount and a network-disabled profile if possible.

docker run --rm --read-only --network none -v "$PWD/runs/candidate:/work:ro" python:3.12-slim sh -c 'cat /work/safe_find.txt'

This command only prints the candidate output. A destructive-flag scan can be added as a separate check for patterns such as rm -rf /, --force-remove, or shell redirections that overwrite outside the workspace. The scanner is intentionally conservative.

3.3 Bug fix check

Run the project's regression test against the patched file. If the candidate returns a complete function, place it in the expected module and execute the test.

pytest tests/test_divide.py -q

The pass condition is the same test the baseline must pass.

4. Record Results in a Decision Table

After running both models, fill one row per task.

Task	Baseline	Candidate	Notes
parse_flags	pass	pass	Candidate used argparse correctly.
safe_find	pass	manual_review	Correct find command but did not exclude `.venv` without extra quoting.
fix_div_zero	pass	fail	Returned a comment instead of code.

The decision table makes the next step explicit. If a candidate fails a task that the baseline handles, the migration discussion is simpler: the failing case is now reproducible and specific.

5. Limitations of the Harness

The harness is a smoke test, not a leaderboard.

Five to twenty tasks are too few for statistical confidence.
Temperature 0 reduces non-determinism but also hides variance that may appear in real use.
The prompt format is fixed; changing the system message can produce different results.
The harness measures behavior on the included tasks only, not the model's general coding ability.
Free server quotas or time limits may require running a smaller batch.

Do not use this approach to claim that a model is “cheaper” or “better” in general. Use it only to decide whether a specific candidate deserves a larger, project-owned evaluation.

6. Who Should Not Use This Workflow

Teams that already have a CI evaluation suite should integrate the new model there instead of replacing it.
Teams with no deterministic tasks or no willingness to review shell commands should not run the shell portion.
Projects that need conversational quality rather than code or command outputs need a different rubric.

The harness adds value when the risk is replacing a code assistant or introducing a new endpoint into a build step, not when the question is broad model preference.

If the motivation is to try a free model endpoint or free server option without committing to a migration, running a ten-task smoke test is a reasonable first gate. Replace the candidate model string with whichever current release is available from the provider's documentation, keep the prompts fixed, and let the pass/fail table, not the announcement, drive the decision.

Stop Arguing About Which Model Is Best. Build a Two-Tier Habit Instead.

Avery Wang — Thu, 13 Aug 2026 04:05:16 +0000

There's a recurring ritual in every dev community I'm part of: a model drops, someone posts benchmark screenshots, and for two days we debate whether to rewire everything. Then the next model drops and we do it again.

A few months ago I opted out of the debate. Not because model quality doesn't matter — it does — but because I realized I was optimizing the wrong variable. The question that actually changed my week wasn't which model is strongest but which of my requests are strong-model requests at all. Turns out: maybe a third. The rest were getting premium answers to trivial questions.

This post is the habit I built instead. It has three parts: sort work by what a mistake costs, keep a frozen sanity suite of my own tasks, and write the routing rules down somewhere with a conservative default.

Sort work by blast radius, not by how clever it sounds

My old instinct was to send "hard" prompts to the good model. But difficulty and danger are different axes. A tricky regex is hard to write and trivial to verify. A two-line find | xargs rm is easy to write and capable of eating a directory.

So I classify by blast radius now:

Instantly falsifiable. Formatting, renames, boilerplate, simple transformations. If it's wrong, my compiler or test suite screams within seconds. Any competent model will do — this is the natural home for free or cheap tiers.
Reviewed by me anyway. Commit messages, changelogs, Slack summaries of a diff, first-draft comments. I'm the verification layer, so a mediocre draft still saves me typing.
Verifiable only with effort. Anything touching the filesystem, package installs, CI config, database changes. Wrongness here is quiet and expensive. These get the strongest model I have and a sandboxed dry run before anything executes. I wrote a separate piece on dry-running assistant-written shell commands because this category keeps hurting people.
Judgment calls. Data model design, API shape, splitting a service. There's no test for "good architecture," so the model's role is to produce options I'll interrogate, not answers I'll apply.

The first two buckets are where the money leaks. They were easily 60% of my request volume and were getting the same expensive treatment as the fourth bucket.

Keep a frozen sanity suite, and run it before believing any release notes

When something new trends, benchmark threads tell me how it performs on curated public tasks. What I need to know is how it performs on my tasks — the ones where I already know the correct answer because I shipped it.

So I keep a small frozen suite, maybe a dozen prompts pulled from my own history: a regex I once fumbled, a pytest file with a known correct assertion set, a diff that needs a specific explanation, one nasty quoting problem. Rough sketch (adapt freely):

# Sketch only — wire `ask()` to whatever client you use.
# The critical property: SUITE is append-only. Never edit a case
# because a model failed it.

SUITE = [
    Case("quoting-bug",   prompt=P1, verify=exact_match(KNOWN_FIX)),
    Case("pytest-asserts", prompt=P2, verify=tests_still_pass_and_catch_bug),
    Case("diff-summary",  prompt=P3, verify=mentions_key_change),
]

for case in SUITE:
    answer = ask(candidate_model, case.prompt, temperature=0)
    log(case.id, model=candidate_model, ok=case.verify(answer))

The discipline matters more than the code:

Frozen means frozen. Tuning a case after a failure turns evaluation into flattery.
Failure shapes get remembered. A model that invents a plausible-but-wrong assertion in the pytest case gets banned from test-writing for me, whatever its total score says.
It has to be cheap to run. A suite you hesitate to execute is a suite you skip.

That third point is where free tiers earn their keep. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently provides free model access plus a free server option, and I've been using that slot as my low-cost tier — it's genuinely handy at evaluation time, since I can rerun the suite against a candidate without watching a token counter. I can't speak to how long that access lasts or what limits apply, so I treat it as a current convenience rather than a foundation.

Write the routing down, and make the default paranoid

Rules I hold in my head evaporate the moment I'm in a hurry. So the policy lives in a file next to my assistant config, something like:

# Routing policy — example, rename buckets to match your work
policy:
  low-stakes:      # falsifiable or self-reviewed work
    model: free-tier
  side-effects:    # shell, CI, package installs, migrations
    model: strong-tier
    gate: dry-run-in-sandbox
  judgment:        # design, cross-cutting refactors
    model: strong-tier
    gate: human-decides
unclassified: strong-tier   # <-- the line that matters most

That last line is the whole ballgame. Routing failures don't look like "cheap model flubbed a rename." They look like a dangerous task getting confidently filed under cheap. When I'm unsure what a request is, it goes up-tier; tasks only move down after the sanity suite or real usage says it's safe. Demotion is earned, never assumed.

Where this breaks down

Low volume makes it pointless. Two assistant sessions a week? Use the good model for everything and spend this energy elsewhere. The payoff scales with daily, varied usage.
Free access is weather, not climate. Build so the cheap tier disappearing means "some tasks get pricier," not "workflow stops." If MonkeyCode's free tier went away tomorrow, my setup reroutes that bucket to whatever's next cheapest — annoying, not fatal.
A dozen cases can't catch subtle drift. The suite filters out gross incompetence. Real regressions surface as real failures, at which point that failure becomes case thirteen.
Task type isn't the only axis. A "trivial" summarization over a huge context window can exceed what a weaker model holds. If your work is long-context-heavy, classify by size too.
Some people have no low-stakes bucket. If your day is auth systems, payments, and prod infrastructure, the honest output of this exercise is "everything routes strong-tier." That's a valid result, not a failed one.

The actual point

Model releases are inputs to a routing decision, not occasions for migration. Freeze a sanity suite from your own work, classify by what a mistake costs, write the rules down with a paranoid default, and let free or cheap capacity soak up the work where being wrong is boring. If you want a zero-billing environment to stand that habit up in, MonkeyCode's free models and free server are a reasonable place to start while you figure out your permanent setup.

What's in your low-stakes bucket? I'm curious whether others sort by task kind, by repo, or by whether they'd notice a mistake before it ships.