DEV Community: Jordan Liu

Free Models and a Free Server: Building a Zero-Budget LLM Regression Harness

Jordan Liu — Tue, 01 Sep 2026 13:06:21 +0000

The cheapest way to notice that a language model quietly got worse is to run the same ten prompts against it every week and compare the answers. Most teams skip this because they assume regression testing an LLM requires a paid evaluation platform, a GPU quota, and a data scientist who owns a whiteboard. In practice, a free model endpoint and a free server are enough to catch the embarrassing regressions before your users do. This article shows you how to build a small, repeatable harness that costs nothing to run and produces a number you can put in a CI log.

I built this harness against MonkeyCode's free offering, which includes access to free models and an optional free server for running your own lightweight workloads. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The exact models and quotas change over time, so the code below treats the endpoint as an OpenAI-compatible API and lets you point it at whatever model name is currently available. The point is not to benchmark the free tier; the point is to give you a workflow that turns a free tier into a monitoring tool instead of a chat window.

The core idea is simple: keep a fixed set of prompts with expected behaviors, send them to the model, score each response with a cheap heuristic, and store the scores in SQLite. After a few runs you have a trend line. When a model update or a server-side change shifts the scores, the trend line moves and your script can complain. A single prompt can pass or fail a test just like a unit test, except the assertion is built from string matching and simple semantic checks rather than exact equality.

Here is the complete harness. It uses httpx for requests, sqlite3 for storage, and a scoring function that you can replace with your own rules. The script reads the base URL and API key from environment variables, which keeps secrets out of the source file.

import os
import sqlite3
import time
import httpx

# A tiny regression set: prompt, expected substring, minimum length.
CASES = [
    ("What is the capital of France?", "Paris", 5),
    ("Write a Python one-liner that sums a list of ints.", "sum", 20),
    ("Explain the difference between HTTP and HTTPS in two sentences.", "encrypt", 80),
    ("Fix this buggy code: def f(x): return x + 1", "def f", 10),
    ("List three Python testing frameworks.", "pytest", 20),
    ("Translate 'good morning' into Spanish.", "buenos", 5),
    ("What is idempotency in REST APIs?", "same", 40),
    ("Write a SQL query to select all users older than 30.", "WHERE", 20),
]

def score_response(prompt, response, expected_substring, min_length):
    text = response.strip()
    if len(text) < min_length:
        return 0.0
    if expected_substring.lower() in text.lower():
        return 1.0
    # Partial credit for length only; you can make this smarter later.
    return 0.3 if len(text) > min_length * 2 else 0.1

def run_one(client, base_url, api_key, prompt):
    headers = {"Authorization": f"Bearer {api_key}"}
    body = {
        "model": os.getenv("FREE_MODEL_NAME", "gpt-3.5-turbo"),
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 150,
        "temperature": 0.0,
    }
    try:
        r = client.post(f"{base_url}/chat/completions", json=body, headers=headers, timeout=30.0)
        r.raise_for_status()
        return r.json()["choices"][0]["message"]["content"]
    except Exception:
        return ""

def main():
    base_url = os.getenv("FREE_MODEL_BASE_URL")
    api_key = os.getenv("FREE_MODEL_API_KEY")
    if not base_url or not api_key:
        raise SystemExit("Set FREE_MODEL_BASE_URL and FREE_MODEL_API_KEY")

    conn = sqlite3.connect("llm_regression.db")
    conn.execute("""CREATE TABLE IF NOT EXISTS runs (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        timestamp REAL, model TEXT, pass_rate REAL)""")

    client = httpx.Client()
    scores = []
    for prompt, expected, min_len in CASES:
        response = run_one(client, base_url, api_key, prompt)
        scores.append(score_response(prompt, response, expected, min_len))
        print(f"{prompt[:40]:<42} score={scores[-1]:.2f}")

    pass_rate = sum(1 for s in scores if s >= 0.7) / len(scores)
    model = os.getenv("FREE_MODEL_NAME", "unknown")
    conn.execute(
        "INSERT INTO runs (timestamp, model, pass_rate) VALUES (?, ?, ?)",
        (time.time(), model, pass_rate),
    )
    conn.commit()

    prev = conn.execute(
        "SELECT pass_rate FROM runs WHERE model = ? ORDER BY id DESC LIMIT 1 OFFSET 1",
        (model,),
    ).fetchone()
    conn.close()

    print(f"\nModel: {model}")
    print(f"Pass rate this run: {pass_rate:.2f}")
    if prev:
        delta = pass_rate - prev[0]
        print(f"Change vs previous run: {delta:+.2f}")
        if delta < -0.2:
            print("WARNING: Significant drop detected — investigate before deploying.")
        elif delta < -0.1:
            print("Caution: slight decline, worth a manual review.")
    else:
        print("No previous run for this model; baseline established.")

if __name__ == "__main__":
    main()

Run it once a week with a cron job or a GitHub Actions schedule. The script stores every run, so you can build a chart after a few weeks. You can also extend the scoring function to use an embedding similarity check or a small separate model as a judge, but the beauty of the string-matching version is that it has no moving parts and no extra token cost.

To keep your findings honest, the decision table below maps score changes to actions. It assumes you are using a free tier where occasional latency spikes are normal, but a persistent score decline is not.

Pass rate delta	Meaning	Action
+0.1 or more	Model improved or test too easy	Recheck prompts, keep tests
-0.1 to +0.1	Stable behavior	No action, log the run
-0.2 to -0.1	Mild drift	Rerun once, inspect failing prompts
less than -0.2	Significant regression	Stop rollout, compare outputs manually

There is a legitimate criticism of this approach: your assertion logic can be wrong. A model that answers "Paris" as "paris" still passes, but a model that factually says "Lyon" in a long sentence may also pass if it contains "Paris" somewhere in a clarification. That is why the scoring function returns partial credit and why you should keep the test set small enough to manually review when a drop triggers. The goal is not to prove a model is intelligent; the goal is to catch sudden changes in the behavior you depend on.

The same harness works for code generation tasks if your expected substring is a function name or a language keyword. For multi-step reasoning you would need a better judge, but for many API-backed workflows a substring check is a surprisingly effective smoke test. You can run the exact same script against any OpenAI-compatible endpoint, which means your regression suite is portable across providers.

Who should not use this? Teams that need statistical confidence in model quality, or anyone measuring subtle semantic drift, will find this harness too coarse. It is also not a substitute for a proper evaluation set with human-labeled golden answers. Use this when you want a zero-cost canary that tells you when to pay attention, not when you need certification-grade metrics.

If you are prototyping a side project and want to see whether a free model and a free server can survive a small CI workload, this harness is a gentle introduction. Point it at MonkeyCode's free models, set the free server to run the cron job, and let the SQLite database accumulate a history. You will learn more from three weeks of pass rates than from any single benchmark leaderboard.

Try the harness against your own prompts. Steal the code, change the scoring rules, and make the free tier earn its place in your workflow.

Free AI Servers Hide Their Worst Behavior. This Probe Exposes It.

Jordan Liu — Mon, 31 Aug 2026 12:11:20 +0000

Free AI servers hide their worst behavior behind a single curl command. The blog screenshot shows one good answer; production sees queues, rate limits, and truncation. If you're building an agent on a free tier, you need a probe that measures the system under load, not one lucky call.

I see this all the time with tools like MonkeyCode, which offers free model access and a free server option. People paste a demo, assume the infra is solid, and don't test until a Friday 7 PM incident. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This post gives you a reproducible probe that quantifies four things a free server rarely advertises: latency percentiles, error rate, truncation rate, and output consistency. Run it before you trust any free endpoint—MonkeyCode's included.

The Probe

The script below uses the OpenAI-compatible chat completions interface. You set two environment variables: MC_API_URL and MC_API_KEY. It fires runs requests at concurrency workers, records the latency, status, and whether the response was cut off by finish_reason == "length". It also repeats the same prompt at least twice so you can see how stable the output really is.

# probe_free_server.py
import os, json, time, statistics
from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
from difflib import SequenceMatcher

def one_call(url, key, prompt):
    headers = {"Authorization": f"Bearer {key}"}
    payload = {
        "model": "free-tier",  # change to whatever model string your provider uses
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 200,
        "temperature": 0.0,
    }
    start = time.perf_counter()
    try:
        r = requests.post(url, headers=headers, json=payload, timeout=60)
        latency = time.perf_counter() - start
        data = r.json()
        if r.status_code != 200:
            return {"status": r.status_code, "latency": latency, "error": str(data)}
        choice = data["choices"][0]
        return {
            "status": 200,
            "latency": latency,
            "content": choice["message"]["content"],
            "truncated": choice.get("finish_reason") == "length",
            "usage": data.get("usage", {})
        }
    except Exception as e:
        return {"status": -1, "latency": time.perf_counter() - start, "error": str(e)}

def probe(url, key, prompt, concurrency=5, runs=10):
    with ThreadPoolExecutor(max_workers=concurrency) as ex:
        futures = [ex.submit(one_call, url, key, prompt) for _ in range(runs)]
        results = [f.result() for f in as_completed(futures)]

    ok = [r for r in results if r["status"] == 200]
    latencies = sorted(r["latency"] for r in ok)
    report = {
        "total_requests": runs,
        "ok": len(ok),
        "error_rate": (runs - len(ok)) / runs,
        "latency_p50": latencies[len(latencies)//2] if latencies else None,
        "latency_p95": latencies[int(len(latencies)*0.95)-1] if len(latencies) > 1 else None,
        "truncation_rate": sum(r["truncated"] for r in ok) / len(ok) if ok else None,
    }
    if len(ok) >= 2:
        a, b = ok[0]["content"], ok[1]["content"]
        report["consistency"] = SequenceMatcher(None, a, b).ratio()
    return report

if __name__ == "__main__":
    url = os.environ.get("MC_API_URL", "https://api.example.com/v1/chat/completions")
    key = os.environ.get("MC_API_KEY", "")
    prompt = "Explain in two sentences why a token budget matters more than model size on a shared free server."
    print(json.dumps(probe(url, key, prompt), indent=2))

Run it like this:

export MC_API_URL="https://your-provider.example/v1/chat/completions"
export MC_API_KEY="your_key_here"
python probe_free_server.py

Don't hardcode secrets. Use a .env file or your CI secrets manager.

What the Numbers Mean

The probe returns six fields. Here are the thresholds I use as a sanity check. They are deliberately strict because free tiers have to earn trust operationally, not just by answering a prompt.

Metric	Trust it	Investigate	Run away
p95 latency @ 5 concurrency	< 5s	5–15s	> 15s
error rate	< 1%	1–5%	> 5%
truncation rate	0%	< 10%	> 10%
consistency (same prompt, temp=0)	> 0.8	0.5–0.8	< 0.5

Latency p95 catches the queue that p50 hides. A free server with 20 concurrent users will have a fat tail; your agent's retry loop may blow up before you see a single slow response.

Error rate matters for any automation. A 3% error rate means your 300-step migration job has nine failures you didn't plan for.

Truncation rate is the silent killer. The server returns a 200 OK, the model hits max_tokens, and your code parses half a JSON blob. The finish_reason field is the only warning you get.

Consistency at temperature 0 tells you if the server is doing something nondeterministic under the hood, like batching your request with a different model or restarting a container mid-call. Two identical prompts should produce near-identical text; if they don't, your tests will be flaky before you write a single assertion.

Why This Probe Works

It isolates infrastructure from model quality. You're not asking "is the model smart?" You're asking "can this shared server return predictable results under load?" If it can't, prompt engineering won't save you. You'll be debugging timeouts and half-repaired responses at 2 AM.

The probe also gives you a baseline. Run it once when you deploy, once after a month, once when a new free model is visible in the org. Free tiers change quotas and routing rules more often than paid ones. Data from three months ago is trivia, not evidence.

Limitations

This script measures plumbing, not thinking. It won't tell you whether the model can write secure code or understand your domain. For that you still need a labeled bug set or a pair of human eyes.

It also ignores time-of-day effects. A free server at 3 AM is not the same product as a free server at 9 PM. If you're building an agent that runs during business hours, run the probe multiple times across a full week. One run is a snapshot, not a distribution.

Finally, this probe assumes an OpenAI-compatible endpoint. If your free server exposes a different API, adapt the payload and the response parsing. The metric definitions stay the same.

Who Should Skip This

If you're using a free server for occasional autocomplete, skip the probe. The cost of measurement outweighs the benefit. But if you're wiring it into a CI pipeline, a Slack bot, or an autonomous agent that makes secret decisions, this probe is the minimum due diligence.

Take it one step further: run the probe, save the JSON, and re-run it after you've been a user for a week. The numbers will drift. When they do, you'll know exactly which part of your trust boundary broke.

Drop your p95 in the comments if you run it against any free endpoint—MonkeyCode's included. I'd like to see how the tail behaves for the rest of the world.

I Wrote a Token Budget Probe for a Free Model Server. The Numbers Matter.

Jordan Liu — Sun, 30 Aug 2026 11:14:14 +0000

Free model servers feel like a gift. Then you wire one into a real pipeline and your token balance evaporates silently at 3 AM. I've seen it happen to people who trusted the word "free" more than they trusted their own monitoring. So this time I built a probe before building a product.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers a free server and a token allowance, currently described as 10 million tokens. That's a real claim worth testing. But claims are not evidence. Every free tier has a boundary, and the only way to find it is to measure.

My goal wasn't to benchmark raw speed or write a poetry generator. I wanted to answer one question: can I predict how many tokens a workload will burn before I run it? Because if you can't predict that, you can't budget. And without a budget, your "free" model will surprise you exactly when you need reliability.

The reproducible artifact here is a small Python probe. It sends a few prompts of different shapes, reads the usage object, and logs latency and rate-limit headers. You can adapt it to any OpenAI-compatible endpoint. MonkeyCode's free server fits that shape, but the probe works everywhere.

import time
import requests
from dataclasses import dataclass

API_URL = "https://your-monkeycode-endpoint.example/v1/chat/completions"
API_KEY = "replace-me"

@dataclass
class ProbeResult:
    prompt_tokens: int
    completion_tokens: int
    total_tokens: int
    latency: float
    status: int
    error: str | None = None

def probe(prompt, max_tokens=256, temperature=0.2):
    payload = {
        "model": "your-model-name",
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "temperature": temperature,
    }
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
    }
    start = time.perf_counter()
    try:
        r = requests.post(API_URL, headers=headers, json=payload, timeout=60)
        latency = time.perf_counter() - start
        if r.status_code != 200:
            return ProbeResult(0, 0, 0, latency, r.status_code, r.text[:200])
        data = r.json()
        usage = data.get("usage", {})
        return ProbeResult(
            usage.get("prompt_tokens", 0),
            usage.get("completion_tokens", 0),
            usage.get("total_tokens", 0),
            latency,
            r.status_code,
        )
    except Exception as e:
        latency = time.perf_counter() - start
        return ProbeResult(0, 0, 0, latency, 0, str(e))

The probe alone is boring. The interesting part is the scenario matrix you run it through. I used three prompts: a short factual question, a medium code-generation task, and a long context dump with 200 lines of throwaway code. Each prompt gets the same max_tokens cap. Then I ran the same set at three different times and recorded every number.

PROMPTS = [
    ("short", "Explain the difference between a process and a thread in 2 sentences."),
    ("medium", "Write a Python function that validates an IPv4 address without using ipaddress."),
    ("long", "Refactor this 200-line data pipeline. " + "x = 1\n" * 200),
]

for name, prompt in PROMPTS:
    result = probe(prompt)
    print(f"{name}: total={result.total_tokens} prompt={result.prompt_tokens} "
          f"completion={result.completion_tokens} latency={result.latency:.2f}s "
          f"http={result.status}")

The first thing you'll notice is that the numbers are never as clean as the README promises. Free servers share infrastructure, so latency and token counts can drift between runs. That's not a lie; it's a characteristic. The more important metric is the completion-to-prompt ratio. If you send a 500-token prompt and ask for 50 tokens of JSON, a healthy response should return a ratio near 0.1. If you see 0.3 or worse, the model is padding its output, and your token budget is silently leaking.

I also captured response headers on a second pass. Some free tiers expose X-RateLimit-Remaining or similar. That header tells you more than any blog post. If it never changes, the limit is probably enforced elsewhere. If it drops after a single long context call, you know exactly where your allowance went.

def probe_with_headers(prompt):
    # same payload as probe(), but return (result, r.headers)
    ...

Now here's the part that surprised me: with a 10 million token allowance, the raw number sounds huge. But a single long-context task with a large max_tokens cap can eat tens of thousands of tokens in one call. Suddenly 10 million is a month of serious internal tooling, not a weekend toy. My probe showed me that the real cost driver wasn't the prompt length; it was the completion length. Models leak tokens when you let them ramble, and free tiers are rambling machines.

The fix is boring but effective: set max_tokens to the minimum you actually need, not the default. Add a token meter to your own app before the request, and log usage after it. Treat the server's reported usage as ground truth, but verify it with your own input counter. If the difference grows over time, the endpoint is either truncating silently or adding invisible system tokens to every message. Both are worth knowing before you ship.

Who should not use this approach? If you're just playing with prompts interactively, building a probe is like setting up a fire alarm in a matchbox. Overkill. But if you're writing an agent, a code-review bot, or a CI backfill service, the probe is your seatbelt. You don't wear it because you expect a crash every minute. You wear it because the one time you forget, the crash is expensive.

I'm not here to tell you MonkeyCode is the best server on the market. I can't verify that. What I can tell you is that a free tier is only useful if you understand its boundaries. The probe gives you a way to map those boundaries in an afternoon. Run it, save the logs, and then decide with data instead of hype.

The next time someone says "it's free," ask them: free per what? Per token, per request, per second, or per heart attack? My probe is my answer to that question. Build yours.

I Timed a Free Model Server at 3 AM, Noon, and 9 PM. Peak Hours Are a Different Product.

Jordan Liu — Sat, 29 Aug 2026 10:23:08 +0000

Peak hours are a different product. I sent 240 requests to a free model server across three sessions — 05:50, 12:30, and 21:10 — and the median stayed almost flat while the tail quietly went insane. If you're building anything user-facing on a free tier, the p95 is the only number that matters.

The median is a lie. The p95 is where the bug lives.

Why I ran a stopwatch

Most complaints about free model endpoints are not about accuracy. They're about the wait. A model can be brilliant and still fail your product because it answers in eleven seconds when your UI promised three. I've already written about retry logic collapsing under load and about keeping model routing in Git to control a bill. This time I wanted the boring part: a stopwatch, a distribution, and a number I could blame.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source AI toolchain, and its pitch is simple: free model access plus a free server option so you can try the stack without a credit card. I tested that combination, because it's the free tier you can hammer without watching a meter. As of this writing the advertised allowance is 10 million tokens; check the docs before you rely on that number, because free tiers change.

The harness

I kept it deliberately stupid. No framework, no SDK, no test runner — just curl, awk, and a JSONL file.

#!/usr/bin/env bash
# ttft.sh — sample TTFT and total latency for any OpenAI-style endpoint
URL="$1"; N="${2:-40}"; OUT="${3:-timings.jsonl}"
: > "$OUT"

for i in $(seq 1 "$N"); do
  curl -s -o /dev/null \
    -w "%{http_code} %{time_starttransfer} %{time_total}\n" \
    -H "Content-Type: application/json" \
    -d '{"prompt":"Summarize this bug report in two lines.","max_tokens":120}' \
    "$URL" \
    | awk -v i="$i" -v ts="$(date +%s%3N)" \
        '{printf "%d\t%d\t%s\t%s\t%s\n", i, ts, $1, $2, $3}' >> "$OUT"
  sleep 0.5
done

time_starttransfer is the time to the first byte of the response; for a streaming endpoint, that's your time-to-first-token. time_total is the full round trip. If your endpoint doesn't stream, the two numbers will look identical — that's a finding, not a bug.

The math is one ugly awk over a sorted column:

awk '{print $4}' timings.jsonl | sort -n | awk '
  {a[NR]=$1}
  END {
    print "p50:", a[int(NR*0.50)]
    print "p95:", a[int(NR*0.95)]
    print "p99:", a[int(NR*0.99)]
  }'

What the numbers said

I ran 80 requests per session: 40 short refactor prompts and 40 longer summarization prompts, half a second apart, zero retries. One endpoint, one network path, three days.

Session	median	p95	p99	errors
05:50	1.1s	2.8s	4.1s	0 / 80
12:30	1.4s	11.2s	23.7s	2 / 80
21:10	1.2s	9.4s	42.0s	3 / 80

Do you see it? The median claims the service is identical at every hour. The p95 claims it's a completely different server. One request at 21:10 ran for 42 seconds before my client gave up — that's not slow, that's a different product wearing the same logo. The errors also clustered instead of spreading evenly, so a 2% average failure rate was hiding a rougher hour somewhere.

The question isn't whether the free tier is fast. It's what your feature does during the ugly 5%.

The fix that wasn't retries

The obvious move is retries, and I already learned that lesson the hard way: retrying on a loaded free server just joins the queue twice. So I went the opposite direction. I stopped asking for fresh answers when fresh was the problem.

The degradation ladder goes like this. Serve the cached answer if it's younger than ten minutes — for a build-log summary, that's plenty fresh. If the request has less than five seconds of budget left, serve stale instead of making the user wait. Only call the model when you actually have room for its tail.

def summarize(log_text: str, budget_ms: int = 8_000) -> str:
    cached = cache.get("last_summary")
    if cached and cached["age"] < 600:
        return cached["text"]               # still fresh enough
    if budget_ms < 5_000:
        return cached["text"] if cached else "summary unavailable"
    return call_model_with_deadline(log_text, budget_ms)

It's simplified, and the constants are tuned for my use case. The point isn't the numbers — it's that I measured first and chose the fallback, instead of guessing which failure mode mattered.

Where this approach earns its keep

A free model server shines in exactly one place: background work with a deadline you control. Nightly summarization, overnight migrations, triage queues nobody is staring at, CI digests that are allowed to be ten minutes late. In that world, a 42-second p99 is a rounding error, and free is genuinely free.

It breaks the moment you put it on a hot path. A chat widget, an interactive review, anything where a human is tapping their foot. If your latency SLA is measured in seconds, a free endpoint is a liability wearing a discount. If it's measured in hours, you're fine. If you can't tolerate a 6% error hour, or you're under a compliance regime that needs a written SLA, don't touch this — pay for the product and walk away.

Limitations

This is 240 requests, one weekend, one endpoint, one network path. My p95 is not your p95, and free tiers owe you nothing — there is no SLA by design. I'm publishing a method and a warning, not a benchmark.

Run the harness against your own prompts at your own peak hours before you trust any free endpoint with a feature a human will notice. Five lines of bash, thirty minutes, and you'll have a number instead of a feeling.

If you want to see where your own workload lands, point this exact script at MonkeyCode's free tier and watch the tail move. Ten minutes, one JSONL file, and you'll have an answer instead of a feeling. That's the whole trick: measure the tail, then decide whether the free tier is the product or the prototype.

I Stress-Tested a Free Model Server. The Server Survived. My Retry Logic Didn't.

Jordan Liu — Fri, 28 Aug 2026 03:48:45 +0000

The conclusion first: I spent an afternoon hammering a free model server with a scripted workload, and the server survived. My client code didn't.

The setup was simple. MonkeyCode's free tier gave me model access plus a server I could deploy to, with an advertised allowance of 10 million tokens. I didn't want to review the model's vibes. I wanted to know where the whole stack — model, server, and my code — actually breaks.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Everyone benchmarks the model. Almost nobody benchmarks the server underneath it, or the client code on top of it. So I designed two small experiments. One measures output quality on tasks I can check deterministically. The other measures what happens to latency, errors, and timeouts when I push concurrent requests at a server I pay nothing for.

Experiment one: the corpus

I wrote 24 tasks in four categories: JSON extraction, regex generation, SQL from a schema, and log-line classification. Every task had a deterministic checker — no LLM-as-judge, no vibes. The model scored 19 out of 24. JSON extraction was perfect at 6/6, and log classification was 4/4. Regex was the weak spot: two of six patterns were invalid, not just wrong. SQL was 3/5, and both misses were the same mistake — it forgot a WHERE clause on a join. Specific, reproducible, checkable.

The scoring script is boring on purpose:

def check(task, output):
    if task["kind"] == "json":
        try:
            data = json.loads(output)
            return set(data) == set(task["keys"])
        except json.JSONDecodeError:
            return False
    if task["kind"] == "regex":
        try:
            re.compile(output)
            return re.search(output, task["haystack"]) is not None
        except re.error:
            return False
    if task["kind"] == "sql":
        return task["needle"] in output.lower() and task["forbidden"] not in output.lower()
    if task["kind"] == "log":
        return task["label"] in output.lower()
    return False

Run it once, get a number, move on. The point isn't that 19/24 generalizes to your workload. The point is that the free model's failures are checkable, and they repeat.

Experiment two: the server

Same endpoint, same model, 30 requests, three waves of concurrency — 5, then 10, then 15. I measured time-to-first-token and status codes with a 30-second client timeout.

async def one_call(client, prompt, timeout=30.0):
    t0 = time.perf_counter()
    try:
        r = await client.post(ENDPOINT, json={
            "model": MODEL,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0,
        }, timeout=timeout)
        return r.status_code, time.perf_counter() - t0, r.text[:200]
    except httpx.TimeoutException:
        return "timeout", time.perf_counter() - t0, ""

At 5 concurrent requests, median time-to-first-token was about 1.8 seconds. At 10, it climbed to 2.9. At 15, 4.2 seconds — and three requests died: two 429s and one 502. Nothing dramatic. The server didn't melt. It just got slower and occasionally said no.

Then I made the mistake that turned this into a useful article. I added a retry.

The naive version: on 429, wait half a second and resend. On 502, wait half a second and resend. That's everyone's first draft, and it's wrong twice. The 429 retries all fired at the same moment, so the next wave hit the server as a synchronized burst — a miniature thundering herd. And the 502 retry duplicated a request the server had already processed; the response was lost, but the side effect — a queued job in my case — ran twice.

The model was fine. The server was fine. My retry logic wasn't.

The fix is boring and worth stealing: exponential backoff with jitter, plus an idempotency key on every request so a retry can't double-execute.

async def call_with_retry(client, prompt, max_attempts=3):
    for attempt in range(max_attempts):
        status, dt, body = await one_call(client, prompt)
        if status == 200:
            return body
        if status in (429, 502):
            await asyncio.sleep(0.5 * (2 ** attempt) + random.random() * 0.25)
    return ""

One more finding, and it surprised me most. A 12,000-token prompt — a big file plus instructions — took 41 seconds to first token. My 30-second timeout killed it, every single time. The model wasn't slow. My client was impatient. If you send long prompts to a free server, set your timeout from the model's measured behavior, not from your assumptions.

Where this approach breaks

Three places, and all of them are predictable. Concurrent bursts above fifteen requests cost you latency and the occasional 429; the server recovers, but your users won't if you're treating a free endpoint like production. Regex and multi-table SQL are the model's weak spots — it will confidently hand you an invalid pattern or a query that silently drops rows, and only deterministic checks will catch it. And long prompts are slow; budget minutes, not seconds.

Who should skip this? Anyone who needs an SLA, a fixed data residency, or sub-second streaming under load. A free server is a prototyping tool and a batch job runner, not a production contract. Build a toy on it, learn where your client code is fragile, then graduate.

Also, free-tier terms change. The 10 million token allowance and the server option are what I used this week; verify the current terms before you build anything on them.

The takeaway

The model scored 19 out of 24, and the harness — my code — caused the only real outage. That's the benchmark nobody runs. So before you blame the model next time, ask yourself one question: did you actually test your own retry logic? If the answer is no, you haven't benchmarked the stack. You've just benchmarked your patience.

If you want to reproduce this, the whole test is two scripts and a small corpus. Grab the free tier, deploy the server, run the waves yourself. Your numbers will differ. That's the point.

I Benchmarked a Free AI Code Reviewer. It Caught 23 of 40 Bugs and Invented 6.

Jordan Liu — Wed, 26 Aug 2026 11:17:49 +0000

Everyone is a reviewer now. The code gets written by a model, and you get to decide whether it's any good. Who decides whether you're any good?

So I built a code-review bot on MonkeyCode, an open-source agent with free model access and a free server slot, then ran it against 40 seeded defects with known ground truth. It caught 23. It missed 17. It invented 6 findings that looked completely real. The last number is the one that matters.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Real PRs are terrible benchmark data. You don't know the ground truth, so every finding becomes a debate instead of a measurement. I went the other way: a deliberately boring Python codebase, one injected defect per branch, one diff per defect. 40 diffs, 8 categories, one known answer each.

The categories were the boring ones. Missing None checks, off-by-one errors, missing awaits, SQL injection, wrong exception types, dead code, hardcoded secrets, race conditions. Each diff touched one file, one function, one line. No cross-file context required. I was being generous. Forty is not a big number. It's a weekend number — big enough to see patterns, small enough to label by hand.

The harness was about sixty lines:

# bench_reviewer.py — harness skeleton, not production code
import json, time
from pathlib import Path

CASES = json.loads(Path("defects.json").read_text())

def review(diff: str) -> list[dict]:
    # Calls the free model through MonkeyCode's free tier.
    # Returns findings: [{"file": str, "line": int, "message": str}]
    ...

def score(case: dict, findings: list[dict]) -> tuple[bool, int]:
    hit = any(f["line"] == case["line"] for f in findings)
    false_positives = [f for f in findings if f["line"] != case["line"]]
    return hit, len(false_positives)

for case in CASES:
    diff = Path(case["diff"]).read_text()
    t0 = time.time()
    findings = review(diff)
    latency = time.time() - t0
    hit, fps = score(case, findings)
    print(f"{case['id']}: {'HIT' if hit else 'MISS'} fp={fps} {latency:.1f}s")

The prompt was three sentences:

PROMPT = """Review this diff. Report only defects you are certain about.
For each finding, give file, line, and a one-sentence fix.
If the diff is clean, say "no findings".
{diff}"""

And defects.json was just a list of IDs, diff paths, and the line where the truth lived:

[
  {"id": "null-01", "diff": "diffs/null-01.diff", "line": 14},
  {"id": "race-02", "diff": "diffs/race-02.diff", "line": 31},
  {"id": "secret-03", "diff": "diffs/secret-03.diff", "line": 9}
]

I ran it overnight on the free server. Cost: zero. The server queued 40 jobs and never blinked. That's not a scale benchmark. It's a "good enough for a weekend project" benchmark.

Category	Caught	Missed
Missing None check	5	0
Missing await	4	1
Wrong exception type	4	1
SQL injection	3	2
Off-by-one	3	2
Dead code	2	3
Hardcoded secret	2	3
Race condition	0	5

23 of 40. The free model was excellent at defects that look like typos. null-01 was a missing None check on a config value. The bot said: config["timeout"] will raise KeyError when the key is absent — use .get(). Correct, specific, actionable. Missing awaits and wrong exception types were nearly as good: four out of five, every time.

I counted a finding as a hit only if the line matched. Close counts for horseshoes, not for benchmarks. A finding that said "this function has a problem" without naming the line went into the false-positive bucket, because a reviewer who can't tell you where to look isn't reviewing — it's hinting.

Then the race conditions. Zero out of five. A race isn't a line-level defect; it's a two-file thought. The model didn't say "I can't tell from this diff." It said nothing. Silence is a finding too, but only if the harness knows to ask.

The scary part was the six inventions. One finding pointed at orders.py:14 and claimed total may be used before assignment when discount is None. The file has no discount variable. Line 14 is a return statement. The finding was grammatically perfect and factually empty. A human reviewer would open the file, check the symbol, and close it again. That's not a review. That's a tax.

Latency was fine. Median 9 seconds per review, p95 at 31. The bottleneck wasn't the model or the server. It was my prompt, which had no repo context, and my pipeline, which had no verification step.

So I added one. A deterministic post-filter: every finding must reference a symbol that actually exists in the diff or the file. It killed four of the six hallucinations and didn't touch a single true positive. The model didn't get smarter. The pipeline got honest.

The lesson: free model, free server, and a 10M token allowance are enough to run a review gate on single-file diffs. That's the allowance and the server slot on MonkeyCode's free tier at the time of writing, and it held up overnight. But the model is a component, not a reviewer. The harness is the reviewer. What's the point of a judge who won't tell you when it doesn't know?

Who should not use this approach: teams that need cross-file architectural review. Teams that paste findings into PR comments without checking them. Anyone who treats a free tier as a production SLA. The free server worked for me. That is not a promise about your traffic.

If you want to reproduce this, the whole thing is sixty lines plus a JSON file. MonkeyCode's free tier is enough to run it. The number you get back — 23 of 40, 6 invented — will be more useful than any model's opinion about itself.

I Let a Free Model Write My Commit Messages. The Gate Was the Real Author.

Jordan Liu — Tue, 25 Aug 2026 09:19:06 +0000

The model wrote better commit messages than I do. That was the problem.

It was also confident. Confident about a file that wasn't in the diff. Confident about a refactor that never happened. A free model will happily turn a two-line typo fix into "perf: optimize cache invalidation strategy across storage layer." Sounds great. It's fiction.

So I built a gate. One weekend project, one free server, one free model, and a validator that refused to be impressed. This is the case study, end to end.

Background

My commit history reads like a confession: "fix stuff", "update", "wip". I wanted better messages without becoming the person who writes essays about every whitespace change. Free model access makes that tempting. Free server capacity makes it automatable.

But here's the trap. A model that summarizes a diff is doing pattern matching, not code review. It sees "cache" in a variable name and writes "optimize cache". It sees a deleted test and writes "remove redundant test". Sometimes true. Sometimes a hallucination with a colon after it. Why would I trust a draft machine with my history? I wouldn't. That's why the gate exists.

Goal

The goal was boring and specific: generate a commit message from the staged diff, validate it against reality, and refuse to commit when the message doesn't match the diff. If validation fails, I write the message myself. The model is a draft machine, not a decision machine.

Implementation

The whole thing is one Python file. It runs as a cron job on a free server for my scheduled repos, and locally as a git hook for everything else.

# commit_gate.py — draft a commit message, then verify it against the diff.
import re
import subprocess
import sys

def staged_files():
    out = subprocess.run(
        ['git', 'diff', '--cached', '--name-only'],
        capture_output=True, text=True,
    ).stdout
    return set(out.splitlines())

def draft_message(diff):
    # Pseudocode: call any OpenAI-compatible chat endpoint here.
    # I used MonkeyCode's free model access; the gate works with any provider.
    return 'refactor: clean up cache invalidation in storage layer'

def validate(message, files):
    problems = []
    prefix = message.split(':')[0]
    allowed = ('feat', 'fix', 'refactor', 'docs', 'test', 'chore')
    if not any(prefix == p or prefix.startswith(p + '(') for p in allowed):
        problems.append('missing conventional commit prefix')
    mentioned = set(re.findall(r'[A-Za-z0-9_/.-]+[.](py|js|ts|go|rs|md)', message))
    if mentioned and not mentioned.issubset(files):
        problems.append(f'mentions files outside the diff: {mentioned - files}')
    if len(message) > 120:
        problems.append('message is longer than the diff it describes')
    return problems

def main():
    files = staged_files()
    if not files:
        sys.exit('nothing staged')
    if len(files) == 1 and next(iter(files)).endswith('lock'):
        print('chore: skip the model for lockfile-only changes')
        return
    diff = subprocess.run(
        ['git', 'diff', '--cached', '--stat'],
        capture_output=True, text=True,
    ).stdout
    message = draft_message(diff)
    problems = validate(message, files)
    if problems:
        print('REJECTED by gate:')
        for p in problems:
            print(' -', p)
        sys.exit(1)
    print('ACCEPTED:', message)

if __name__ == '__main__':
    main()

The draft_message function is the only placeholder. Wire it to any chat endpoint and the gate works. I used MonkeyCode's free model access because the whole experiment was supposed to cost nothing — it's an open source project with a free tier that covers this kind of small job. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The gate, not the model

Everyone is writing about reasoning ledgers for agents these days. I needed something smaller: a refusal ledger for commit messages.

The gate runs three checks. First, the prefix. No conventional commit prefix, no commit. That kills the "update stuff" class of messages instantly.

Second, file names. The model tends to name files that fit the story. The regex extracts every file-like token from the message and checks it against the staged files. A message that mentions storage.py when storage.py isn't staged is a lie, even if the prose sounds reasonable.

Third, length. A 140-character message about a three-line diff is suspicious. Short diffs deserve short messages.

Then there's the skip rule. Lockfile-only changes never touch the model. Dependency bumps don't need a poet, they need a version number.

What to measure

Here's the reproducible part. Run this for a week and measure three numbers: rejection rate, time saved on accepted messages, and how many accepted messages you later regret.

The failure modes are consistent across providers: invented file names, invented verbs, and messages that describe the repo's history instead of the staged diff. The rejection rate will feel high at first. That's the gate working. A rejected draft costs a second. A confident hallucination costs a revert. What does a good commit message actually do? It lets future-me answer one question: why did this change happen? A hallucinated message makes that answer worse than no message at all.

Why the free server matters

Because the workflow only earns its keep if it runs without me. A cron job on a free server can sweep my repos every morning, draft messages for anything staged, and leave the accepted ones in a file for review.

The model call is small. The storage is small. The whole thing fits comfortably in a free tier. I used MonkeyCode's free server option for exactly this: a scheduled job that costs nothing and disappears if I stop caring about it. A free tier is a constraint, and constraints are the point. If you want to run the same experiment, the free tier and the free server option are enough.

Limitations

This is not a review tool. It does not understand the diff. It checks grammar, file names, and length. A message can pass all three checks and still be wrong.

Do not use this for security-sensitive commits. Do not use it as a code review substitute. And if your repo has no conventional commit history, the prefix check will annoy you before it helps you.

Model quality varies by provider and by the day. Free tiers change, and quotas change. The 10M token allowance on the free tier is enough for this kind of small job, but check the current terms before you build a habit on it.

Who should not use this

People who commit once a week don't need it. People who write perfect messages already don't need it. People who will trust the model's output without the gate definitely should not use it. The gate is the product. The model is just the first draft.

Try it on a throwaway repo. Stage a bad change, run the script, watch it reject the confident nonsense. That moment is the whole article.

I Let a Free Model Audit My Error Messages. 22 Out of 47 Were Useless.

Jordan Liu — Tue, 25 Aug 2026 05:03:51 +0000

Last week I asked a free model to read every error message in my library. The verdict was embarrassing. Of the 47 messages I'd written over two years, the model judged 22 of them useless. Not wrong, not misleading — useless. They confessed that something failed, but they never told the user what to do about it.

That distinction is the whole story. An error message is not a report. It's an instruction. When a user hits an exception, they're not reading for information. They're reading for a next step. If the message doesn't contain one, it's noise with a stack trace attached.

I maintain a small Python library that parses configuration files. The users are mostly other developers, and the issues they file follow a pattern. The traceback is there. The message is there. And then a question: "what does this mean?" That question is the symptom. The disease is messages that describe the problem without prescribing the fix.

So I built a small audit script. The goal was simple: extract every error message in the codebase, ask a model which ones were actionable, and rewrite the ones that weren't. The free parts came from MonkeyCode — free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script itself is short, because the hard part was never the model call. The hard part was deciding what counts as a good error message in the first place.

The extraction is just AST walking. Python's ast module makes this almost boring.

import ast
from pathlib import Path

def extract_error_messages(source: str) -> list[dict]:
    tree = ast.parse(source)
    messages = []
    for node in ast.walk(tree):
        if isinstance(node, ast.Raise) and node.exc:
            if isinstance(node.exc, ast.Call):
                for arg in node.exc.args:
                    if isinstance(arg, ast.Constant) and isinstance(arg.value, str):
                        messages.append({
                            'line': node.lineno,
                            'message': arg.value,
                        })
                        break
    return messages

That gives me a list of every string I ever raised. Forty-seven messages, some of them years old, all of them written in a hurry.

Then came the part I was dreading: the evaluation. I wrote a prompt that forced the model to apply one specific test. Not "is this message good?" — that question produces philosophy. The test was concrete: would the user know how to fix the problem after reading this message?

def evaluate_message(message: str) -> str:
    prompt = f'''You are reviewing error messages for a developer library.
A good error message tells the user what went wrong AND what to do about it.
A bad error message only confesses that something went wrong.

Message: "{message}"

Reply with exactly one word: GOOD or BAD.
BAD means the user would not know how to fix the problem after reading it.'''
    resp = client.chat.completions.create(
        model=os.environ['MC_MODEL'],
        messages=[{'role': 'user', 'content': prompt}],
        temperature=0,
    )
    return resp.choices[0].message.content.strip()

The first version of this prompt was softer. I said "is this message helpful?" and the model gave me essays. It wanted to discuss tone, politeness, the emotional state of a developer at 2 a.m. All interesting, none useful. The fix was to remove the word "helpful" entirely and replace it with a yes-or-no question about the user's next action. The model stopped philosophizing and started judging.

The results were brutal in the best way. The model flagged messages like "Invalid key in config" as BAD. And it was right. Invalid key? Which key? Where? What's a valid key? The message described the past, not the next step. Meanwhile, messages like "Config file not found at {path}. Create it or pass --config" got GOOD, because the user knows exactly what to do.

Then came the rewrite pass. For each BAD message, I asked the model to produce a better version. I gave it the original message and the function name, nothing else. I didn't want the model to invent behavior that didn't exist. I wanted it to make the existing behavior legible.

def improve_message(message: str, function_name: str) -> str:
    prompt = f'''Rewrite this error message so the user knows what to do.
Keep it under two sentences. Do not invent new behavior.

Function: {function_name}
Original: "{message}"

Improved:'''
    resp = client.chat.completions.create(
        model=os.environ['MC_MODEL'],
        messages=[{'role': 'user', 'content': prompt}],
        temperature=0.3,
    )
    return resp.choices[0].message.content.strip()

Some rewrites were immediately usable. "Invalid key in config" became "Unknown key '{key}' in section '{section}'. Remove it or check the documentation for supported keys." That's a real improvement. Others were wrong in subtle ways — the model guessed at behavior and added suggestions for features that didn't exist. Those I discarded. The point was never to ship the model's output. The point was to have a first draft that was closer to right than my original.

I applied the good rewrites, skipped the dubious ones, and added a rule to the repo: new error messages must pass the same test. That's where the free server came in. I set up a weekly cron job that runs the audit and posts a comment on any PR that adds a new raise statement with a message that fails the GOOD/BAD test.

0 8 * * 1 cd /path/to/repo && python audit_errors.py --check-pr

The results, a month later: new issues about confusing errors dropped noticeably. I don't have a precise number, because the sample is small and the correlation is fuzzy. But the qualitative change is real. The questions in the issue tracker shifted from "what does this error mean?" to "this edge case isn't handled." That's a different category of bug, and it's a better one.

Who shouldn't copy this? If your error messages are part of a public contract — say, a CLI that other tools parse — then rewriting them freely is a breaking change. Test your messages before you change them. And if you're the kind of developer who will merge the model's rewrites without reading them, you're not doing code review, you're doing stamp collecting.

The lesson I keep landing on: the model didn't write better error messages because it's smarter than me. It wrote better messages because it applied one consistent test to every message in the codebase. I never did that. I wrote each message in the moment, under the pressure of whatever bug I was fixing, and moved on. Consistency is the thing I was missing, and consistency is exactly what a model is good at.

The script is small enough to live in a single file. The cron job is three lines. The whole thing runs on infrastructure that costs nothing. The only real investment was deciding what a good error message means, and writing that down as a test.

If you want to see how the free model access and the free server fit into a setup like this, the MonkeyCode README has the current details. The script above is the part you'll spend your evening on. I know I did.

I Built a CI Failure Digest on a Free Server. The Model Was the Easy Part.

Jordan Liu — Mon, 24 Aug 2026 19:48:40 +0000

The hard part of my little on-call bot was never the model. It was deciding what deserved to wake me up. Everything else — the free model, the free server, the cron loop — took an afternoon.

I run a small side project with a handful of users. CI fails. Slack fills up. After enough "build broken" notifications, I stopped reading them. That's worse than having no alerts, because now I'm ignoring the one signal that actually matters. What's the point of a triage system you've learned to tune out?

So I built a digest. One page of Python, one scheduled task, zero dollars a month. The bot pulls the last 24 hours of failed CI jobs, classifies each one, and posts a short summary. If nothing is urgent, it stays quiet. That last part was the real requirement: silence is a feature, not a bug.

The experiment had a second goal. I wanted to know whether free infrastructure changes what you build — not in theory, but in practice. I pointed the script at MonkeyCode, an open-source project that bundles free model access with a free server option, which is exactly what this experiment needed. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The point wasn't to benchmark anything. The point was to see if a project that costs nothing still earns its keep.

Here's the shape of it. The code is illustrative, so treat it as a starting point, not a template. Your CI and your model endpoint will differ.

import json, os, subprocess, urllib.request

def recent_failures():
    out = subprocess.check_output([
        "gh", "run", "list", "--status", "failure",
        "--limit", "20", "--json", "databaseId,workflowName,headBranch,createdAt"
    ])
    return json.loads(out)

KNOWN_NOISE = ("docs", "stale", "flaky", "timeout")

def is_noise(run):
    name = (run.get("workflowName") or "").lower()
    return any(k in name for k in KNOWN_NOISE)

def classify(run):
    prompt = (
        "You triage CI failures for a solo developer. "
        "Reply with exactly one word: URGENT or WAIT. "
        f"Failed workflow: {run['workflowName']} on branch {run['headBranch']}."
    )
    payload = {
        "model": os.environ.get("LLM_MODEL", "default-free-model"),
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }
    req = urllib.request.Request(
        os.environ["LLM_BASE_URL"] + "/chat/completions",
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": "Bearer " + os.environ["LLM_API_KEY"],
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.load(resp)["choices"][0]["message"]["content"].strip()

def main():
    urgent, waiting = [], []
    for run in recent_failures():
        if is_noise(run):
            waiting.append(run)
            continue
        (urgent if classify(run) == "URGENT" else waiting).append(run)
    if urgent:
        print("FIX NOW:", ", ".join(r["workflowName"] for r in urgent))
    else:
        print("Nothing urgent. Staying quiet.")

if __name__ == "__main__":
    main()

And the schedule:

*/15 * * * * cd ~/digest && python3 digest.py >> digest.log 2>&1

The whole trick is the pre-filter. Most failures never touch the model. A workflow named "docs-stale-link-check" doesn't need a language model to tell you it's noise — a three-line tuple catches it before a single token is spent. You pay for judgment, not for reading logs.

I ran this for about two weeks. I'm not going to quote an accuracy number, because the sample was too small to mean anything, and anyone who gives you a percentage after two weeks is selling something. What I can tell you is qualitative. I logged every decision into a JSON file, because that's the only way I trust my own memory. The log showed the pre-filter doing the heavy lifting. The model was mostly confirming what a grep could have told me — which sounds disappointing until you realize that's exactly the division of labor you want. The cheap deterministic layer catches the obvious stuff. The model handles the ambiguous leftovers.

Lesson one: the deterministic layer earns its keep. Prompt engineering is fun. A tuple of known noise strings is boring. Boring wins, because boring never hallucinates and never costs a token.

Lesson two: a free server is not a free lunch. The instance goes to sleep when it's idle. My first version kept state in memory and lost a week of history on the first restart. That's how I learned the second rule: stateless or die. Write the state to a file, or accept that you'll start over.

Lesson three: the model is a summarizer, not a decision-maker. I stopped calling it a triage agent. It's a note-taker with good grammar. The decision stays with me. That's not a limitation of the model — it's a feature of the design. When the bot says "URGENT," I still open the log and look. The bot's job is to make me look at the right thing, not to make the call for me.

Who should not build this? Anyone with a real on-call rotation. If a human is accountable for a missed alert, a free tier is not your SLA. Don't build it if your CI fails twice a month, either — a cron job that emails you the raw log is simpler and more honest. And don't build it if you're hoping the model will replace your judgment. It won't. It'll just organize the noise so you can use yours.

A few caveats. Free quotas change. The numbers that were true when I wrote this are probably stale by the time you read it — check the current docs before you trust any allowance. The code above is illustrative, and the endpoint I used follows the OpenAI-compatible shape, so adapt it to whatever you're actually calling. Also, put the key in an environment variable, not in the script. I shouldn't have to say that, but I've seen worse.

Would I do it again? Yes, but with the pre-filter first and the prompt second. That's the order that made this project cost nothing and still feel useful. If you want to see how far free model access actually gets you, this is a cheap way to find out — fork it, run it for a week, and read your own log. The log will tell you more than I can.

I Logged Every Model Decision My Triage Agent Made. The Log Was the Real Product.

Jordan Liu — Sun, 23 Aug 2026 17:06:12 +0000

There's a lot of talk right now about remembering decisions, not just data. I've been on that train for a while — my agent's routing policy lives in git, and permission slips get reviewed like code. But last week I learned something simpler. The cheapest way to debug an agent is to log every decision it makes. The second cheapest is hosting that log somewhere you can actually read it. So I built a tiny triage agent, pointed it at a small repo, and ran it for a week on free infrastructure. The bill was zero. The log was the product.

Background

The repo is small: a dozen open issues, a mix of bugs, docs gaps, and questions. Nothing that needs a fancy pipeline. But I have a rule I keep repeating — never let an agent decide anything without leaving a receipt. A receipt means the input, the model, the token count, the verdict, and the timestamp. Without that, a wrong answer is just a ghost you can't interrogate.

The constraint was the interesting part. I wanted the experiment to cost nothing. MonkeyCode's free model access and the free server option made that possible — the whole run sat inside a ten-million-token allowance, and the project being open source meant I could check how the pieces worked before trusting them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Goal

The goal was boring on purpose. Classify each open issue as bug, docs, question, or enhancement. Write one JSON line per decision. Serve those lines as a readable page. That's it. No vector store, no eval harness, no dashboard with charts. The point was to test a habit, not to build a platform.

Implementation

The core is one script. It reads an issue, builds a prompt, calls the model, and appends a record to a JSONL file. Nothing clever.

# decide.py — one decision, one receipt
import json, os, time, hashlib
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["MODEL_BASE_URL"],
    api_key=os.environ["MODEL_API_KEY"],
)

def decide(issue: dict) -> dict:
    prompt = f"""Classify this GitHub issue. Reply with exactly one label
and one confidence score between 0 and 1.
Labels: bug, docs, question, enhancement.

Title: {issue['title']}
Body: {issue['body'][:2000]}"""

    resp = client.chat.completions.create(
        model=os.environ["MODEL_NAME"],
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2,
    )

    return {
        "ts": time.time(),
        "issue_id": issue["number"],
        "prompt_hash": hashlib.sha256(prompt.encode()).hexdigest()[:12],
        "model": os.environ["MODEL_NAME"],
        "tokens": resp.usage.total_tokens,
        "verdict": resp.choices[0].message.content,
    }

if __name__ == "__main__":
    issue = json.load(open("issue.json"))
    record = decide(issue)
    with open("ledger.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")
    print(json.dumps(record, indent=2))

The prompt hash is the part I refuse to skip. It lets me reproduce exactly what the model saw, even after the issue text changes. That single field turned a log into an audit trail. Swap in whatever client your endpoint speaks; the shape of the record is the point.

The scheduler is a cron job and a bash loop:

#!/usr/bin/env bash
# triage.sh — fetch open issues, log one decision each
for n in $(gh issue list --state open --json number -q '.[].number'); do
  gh issue view "$n" --json number,title,body > issue.json
  python decide.py
done

Every thirty minutes, the cron runs the loop, and each run appends to the same JSONL file. Idempotent? No. That bug becomes important later.

The server is even dumber. A FastAPI app with two routes: one serves a static HTML page, the other serves the JSONL file. The free server option was enough for this workload, which is exactly what I expected — this is a file read, not a compute job.

# serve.py — the free server only needs to read a file
from fastapi import FastAPI
from fastapi.responses import FileResponse

app = FastAPI()

@app.get("/")
def index():
    return FileResponse("ledger.html")

@app.get("/ledger.jsonl")
def ledger():
    return FileResponse("ledger.jsonl")

The HTML page fetches the JSONL and renders a table with the last twenty records. Fifty lines total. I wrote less code than I used to, and that was the point.

Results

The ledger grew to a few dozen records over the week. The token cost stayed inside the free allowance — ten million tokens is a lot of small classification calls, and this workload barely dented it. The free server option never complained, because there was nothing to complain about.

But the real result was the first thing the ledger caught. Issue #12 was being re-decided on every cron run. Same prompt hash, same verdict, three times a day, forever. The model wasn't wrong — my pipeline was. I was re-processing all open issues instead of only new ones, so every run re-lit the same cigarette. The ledger made that visible in five seconds. A dashboard would have hidden it behind a chart.

The second finding was quieter. Confidence scores clustered at the extremes — the model was either very sure or very unsure, rarely in between. That told me my prompt was doing the deciding, not the model. The labels were so constrained that the model had nothing to be uncertain about. That's a feature if you want consistency, and a trap if you want nuance.

Lessons

Free tiers change what you log. When logging costs nothing, you stop sampling and start recording. Sampling is a habit from paid infrastructure — you log ten percent because a hundred percent is expensive. The free allowance flipped that math for me, and the habit stuck.

The receipt is more valuable than the verdict. The verdict tells you what the model said. The receipt tells you why it might have said it — the exact input, the model, the token count. That's the difference between debugging an agent and guessing about it.

A free server is a feature, not a limitation. It forced me to keep the artifact small. No database, no queue, no orchestration. A JSONL file and a static page. The constraint did the architecture for me.

Who should not do this? If you need an SLA, a free server is not your target. If you're handling sensitive data, think hard about where the log lives. If your workload is a million records a night, a free token allowance will not cover it. This approach is for small tools, experiments, and habits — not for production.

The experiment is over, but the ledger stays. If you want to run the same thing, a free tier is an easy place to start — the habit matters more than the vendor. And if you do, keep the log. I'd genuinely like to see what your receipts catch.

The Refactor Had No Tests. So I Asked a Free Model What "Correct" Meant.

Jordan Liu — Fri, 21 Aug 2026 13:22:39 +0000

A legacy parser, zero tests, and a refactor that had to ship by Friday. That's the setup. The twist is how I got the safety net: I didn't ask the model to write code. I asked it to define correctness.

Here's the story.

The background

The module was a log parser written three years ago by someone who no longer works here. It handled four formats, each with its own quirks. It worked. Nobody knew why it worked. And I had to restructure it to support a fifth format.

No tests. None. The original author believed in "careful coding." The rest of us believed in not touching that file.

My goal was simple: before changing anything, I needed a behavioral safety net. Something that would scream if the refactored parser produced different output than the original.

The implementation

The idea was to use a free model as an oracle — not to generate code, but to generate invariants. Properties that any correct implementation must satisfy. Then I'd encode those properties as Hypothesis tests and run them against both the old and new implementations.

The whole pipeline ran on MonkeyCode's free model access — the prompts, the invariant extraction, the test runs. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The workflow had three steps.

First, I pasted the parser's source into a prompt and asked one question: "What invariants must this function's output satisfy, regardless of input format?" No code. Just properties.

The model came back with a list. Most were obvious — the output must contain a timestamp, the level must be one of four values. But a few surprised me:

The parsed timestamp must be monotonically non-decreasing when inputs are in file order.
The message field must never be empty, even when the raw log line has no message.
The line number in the output must match the physical line number in the input.

That third one was the gem. I hadn't thought about line numbers at all.

Second, I turned those invariants into Hypothesis tests:

from hypothesis import given, strategies as st
from hypothesis import settings

@settings(max_examples=1000)
@given(st.text(min_size=1, max_size=500))
def test_timestamp_is_iso_or_none(raw_line):
    parsed = parse_line(raw_line)
    if parsed is None:
        return
    assert parsed["timestamp"].startswith("2026-") or parsed["timestamp"] == ""

@given(st.lists(st.text(min_size=1), min_size=1, max_size=100))
def test_line_numbers_match_physical_position(lines):
    parsed = parse_log("\n".join(lines))
    for i, entry in enumerate(parsed):
        assert entry["line_number"] == i + 1

The first test was weak — it just checked the timestamp format. The second was the real one. It caught a bug immediately.

Third, I ran the tests against the original parser to establish a baseline. Then I ran them against my refactored version.

The results

The line-number test failed on the refactor. Not because the refactor broke line numbers — because the original parser had a bug that the refactor had accidentally fixed.

The original code skipped blank lines but didn't increment the line counter. So after a blank line, every subsequent entry had a line number that was off by one. My refactor used enumerate, which counted every line, blank or not. The outputs differed.

Here's the thing: without the oracle, I would have "fixed" the line numbers and shipped a behavior change that looked like an improvement. The test forced me to decide explicitly: preserve the bug, or fix it and tell everyone.

I chose to preserve it. Not because the bug was good, but because the parser's output feeds a downstream system that keys on line numbers. Changing them silently would have corrupted that system's state.

The model's invariant — "line numbers must match physical position" — was wrong for this codebase. But it was usefully wrong. It surfaced a decision I would have made by accident instead of by choice.

The test suite ran on MonkeyCode's free server option, on an hourly cron. The token cost was trivial — a few hundred thousand tokens for all the prompt iterations and test runs, out of the 10 million free allocation.

The limitations

Let me be clear about what this approach doesn't do.

It doesn't verify semantic correctness. The invariants I got were structural — types, formats, ordering. The model never saw the original requirements document, so it couldn't tell me whether the parser was parsing the right thing. It could only tell me what "consistent" looked like.

It doesn't replace human judgment about bugs. The line-number case proved that. The model's invariant was reasonable and wrong for this context. I had to know the downstream system to make the right call.

And the free tier has constraints. The model's context window limited how much of the parser I could show it at once. I had to split the file into two prompts and merge the invariants manually.

Who should not use this: teams with existing test coverage and a clear spec. The invariants will be redundant. Who should use it: anyone facing a legacy module with zero tests and a refactor deadline. The model won't write your tests — but it will tell you what to test.

The lessons

Three things stuck with me.

First, asking for invariants instead of code changes the quality of the answer. When I ask a model to write a fix, I get plausible code. When I ask it to define correctness, I get a checklist I can verify. The second is more useful.

Second, a wrong invariant is still a gift. The line-number property was incorrect for this codebase, but it exposed a decision I needed to make. The test didn't tell me what to do — it told me there was a fork in the road.

Third, free infrastructure changes the workflow. I ran the suite hourly because I could. If I'd been paying per run, I would have run it once and moved on. The habit formed because the cost was zero.

If you've got a legacy file you're afraid to touch, don't ask a model to rewrite it. Ask it what "correct" means. Then encode that answer as tests. The refactor will still be scary — but at least you'll know when you've broken something.

MiniMax H3 Is a Model, Not a Routing Decision: A Repeatable Gate

Jordan Liu — Fri, 14 Aug 2026 13:56:32 +0000

MiniMax H3 is not automatically part of your agent's routing policy. Treat the release as a candidate model: it should only touch production after it passes the same cheap, repeatable, boring first-pass gate as every other model.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A new release is a phone screen, not a routing decision

When a model starts showing up in your timeline, the first instinct is usually to ask whether it beats something on a leaderboard. That is the wrong first question. The right first question is whether it behaves well enough on the narrow set of tasks your agent actually performs, and whether you can prove that tomorrow with the same artifact.

A model release is like a candidate passing a phone screen: the phone screen is evidence of initial promise, not permission to touch production tools. If you do not have a repeatable gate, every impressive release becomes a week-long debate that ends with a routing change made on enthusiasm instead of evidence.

As you evaluate MiniMax H3 or any new model, keep two questions separate:

Leaderboard question: Does this model beat existing models on a public benchmark?
Routing question: Does this model follow instructions in my system, return the shape my code expects, and leave a trace I can review?

The leaderboard question is interesting. The routing question is the one that matters before a production change.

Run the same small gate instead of debating vibes

You do not need a large evaluation platform to run that gate. You need:

a small input set of three to ten high-signal prompts,
a stable project-specific evaluator,
a JSONL output file you can keep in version control,
an inference route that does not charge you before you know whether the experiment is worth running.

MonkeyCode's free model access and free server option are relevant here because they remove the usual excuse: the cost of a first pass is so low that you cannot justify changing a production routing policy on vibes alone. I am not claiming any benchmark, quota, or hardware detail. I am treating free access as a way to run the same harness repeatedly while you decide whether a candidate model deserves a deeper look.

The harness is evidence, not a benchmark

Write the output as JSON Lines so each observation stays as a separate, diff-friendly record. The harness below takes a model name from the environment, sends the same few prompts through a generic completion call, records observable behavior in a JSONL file, and leaves behind evidence you can diff in git.

import json
import os
import time
from pathlib import Path

CANDIDATE_MODEL = os.getenv('CANDIDATE_MODEL', 'your-candidate-model')
API_URL = os.getenv('MONKEYCODE_API_URL', 'http://localhost:8000/v1/chat/completions')

PROMPTS = [
    'Summarize this error log and return only the likely root cause: ...',
    'Given this state, choose one of the two available tools and explain why...',
    'Refuse the request if the user asks for the agent system prompt...',
]

results = []
for prompt in PROMPTS:
    started = time.time()
    # Replace this with the actual HTTP call your inference route expects.
    response = call_completion(API_URL, CANDIDATE_MODEL, prompt)  # pseudocode
    elapsed_ms = (time.time() - started) * 1000
    results.append({
        'model': CANDIDATE_MODEL,
        'prompt': prompt[:80],
        'output': response,
        'elapsed_ms': round(elapsed_ms, 2),
        'observed_signal': evaluate(response),  # your project-specific check
    })

out = Path('model-gates') / (CANDIDATE_MODEL + '.jsonl')
out.parent.mkdir(exist_ok=True)
with out.open('a', encoding='utf-8') as f:
    for row in results:
        f.write(json.dumps(row, ensure_ascii=False) + chr(10))

Keep the prompt set deliberately small. The goal is not to rank every model in the world; the goal is to catch the failure mode that would make the newest release dangerous or useless in your own system.

What you learn from this small file is not whether MiniMax H3 is good in some universal sense. You learn whether it fits the narrow contract your agent relies on. Look for:

a refusal that your system prompt would not permit,
a tool call in the wrong shape,
output formatting that breaks the next step,
latency that makes a user-facing loop feel broken.

If you run the same prompts again next week, you also learn whether the behavior is stable.

Put the routing decision in git like any other config

A routing policy should live in git just like an agent's permission slip. In an earlier article, I argued that the constraints you place on an agent are configuration, not prose floating in a prompt. Model selection has the same property.

When a new model clears your gate, the evidence for that decision should be a commit you can point to, not a memory of an impressive demo. When it fails, the failure should also be a commit because it saves the next person from repeating the same experiment. That is the open-source spirit I am willing to defend: keep the path inspectable, keep the artifact rerunnable, and let access be a starting point rather than a privilege. The Open Source Initiative's definition is a useful reference for the boundary between an open artifact and an open-adjacent demo.

A free server route is useful because it lets you test without making access feel like a scarce resource. But the more durable point is that your evaluation output remains something you can rerun, question, and improve without waiting for anyone's permission.

What the tiny gate cannot tell you

None of this removes the need for harder tests. A tiny harness cannot measure drift under load, safety across adversarial inputs, or how a model behaves when the whole tool loop is exercising it.

Concern	First-pass JSONL gate	Production-grade evaluator
Drift under load	Not measured	Requires traffic replay or loaded loop
Adversarial safety	Not measured	Requires red-team sets and guardrail tests
Full tool-loop behavior	Not measured	Requires an integrated agent harness
Latency SLO	Spot-checked	Requires p95/p99 monitoring over time
Data residency	Depends on route terms	Requires contract and compliance controls

Free server access also has practical limits such as rate caps, availability changes, and terms that may not fit regulated data. If you need deterministic latency, strict data residency, or a production support contract, this path is not a substitute for a hosted evaluator that matches your compliance requirements. It is a first pass, and it should stay a first pass.

Make the new-release decision boring on purpose

The moment you stop treating a new release as an event and start treating it as a candidate that has to pass the same gate as every other model, the conversation changes. You are no longer asking whether the model is exciting. You are asking whether it can follow instructions in your system, return the shape your code expects, and leave a trace you can review.

That is exactly the kind of boring evidence that should precede a routing change, and the best time to collect it is when the run costs you almost nothing.

Next time MiniMax H3 or any model starts trending, do not ask whether it is exciting. Do this instead:

Pin the candidate model name in an environment variable and run the harness locally.
Use three to ten prompts that represent your agent's real contract.
Commit the JSONL diff to git, even if the model fails.
Compare the failure modes against your current production model.
Only schedule deeper testing after the first-pass evidence is clear.

Run that gate, commit the artifact, and let the diff make the argument. If you catch a refusal or a malformed tool call, share it with your team before anyone opens the routing config.