DEV Community: Blake Yang

The Retry Loop That Ate a Free Tier: A Rate-Limit Postmortem

Blake Yang — Tue, 25 Aug 2026 08:30:10 +0000

The webhook handler passed every local test, deployed cleanly, and then took down the free server at 2:14 AM. The symptom looked like a provider quota problem, but the root cause was a single AI-generated loop that retried too eagerly. This post walks through the debugging path from the first 429 to the actual fix, because the same failure pattern is hiding in a lot of generated code.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Symptom: 429s That Appeared Out of Nowhere

The setup was deliberately small: a webhook endpoint generated with MonkeyCode's free model access, hosted on its free server, and connected to a third-party API that occasionally returned 503. MonkeyCode is an open-source project, and its free tier includes a 10-million-token allowance plus a free server instance, which is plenty for a small integration but tight enough to expose amplification bugs. The handler worked for two days, and then legitimate requests started failing with 429 Too Many Requests even though the app was nowhere near its monthly token budget.

The first debugging instinct was to blame the third-party provider, since 429 is a rate-limit status code and that provider had been flaky all week. A quick check of the provider's status page showed no incident, which shifted the suspicion back to the free server. The server's own logs showed nothing unusual, because the requests that failed were outgoing retries, not incoming traffic.

The turning point came when the developer replayed the exact failing request sequence locally against the same API and got 200 responses. The code was identical, the payload was identical, and the only difference was the server environment. That local success strongly suggested an environmental cause, so the investigation moved to the server's outgoing traffic.

The Investigation: Following the Retry Trail

The access logs on the free server revealed the pattern: every time the downstream API returned a 503, the handler fired off a new request immediately, then another, then another, with no delay between attempts. A single downstream hiccup of three seconds produced forty outgoing requests in under a minute, and the third-party API responded by rate-limiting the server's IP. The logs made the amplification visible in one command:

grep "POST /api" access.log | awk '{print $4}' | cut -d: -f2 | uniq -c

The AI-generated retry loop looked harmless at first glance because it was short and readable. Here is the simplified version that caused the incident:

def call_api(payload):
    while True:
        response = requests.post(API_URL, json=payload)
        if response.status_code < 500:
            return response.json()
        # keep retrying until the API accepts the request

The loop retries forever, it retries instantly, and it treats every 5xx as worth another attempt. On a free tier with a per-minute request cap, that behavior converts a transient downstream error into a self-inflicted outage, because the retries themselves consume the quota that legitimate requests need. The second problem was that the loop ignored the Retry-After header that the API included in its 429 responses; the API was explicitly saying "wait 30 seconds," and the handler was responding with "how about right now."

The Root Cause: The Happy Path Is Generated, the Failure Path Is Not

The real lesson is not that the AI model wrote a bad loop, because plenty of human-written loops have the same flaw. The lesson is that generated code tends to be optimized for the happy path, and the failure path is where the assumptions live: retries need backoff, backoff needs jitter, and jitter needs a maximum ceiling. This pattern matters more now that AI coding tools produce more of the codebase, because the happy path is exactly what models generate confidently and the failure path is exactly what they gloss over.

The fix for this incident had three parts:

Respect the Retry-After header whenever the downstream service provides one.
Use exponential backoff with jitter so retries spread out instead of clustering.
Cap the total retry budget so a stuck service cannot consume the entire free tier.

import random
import time

MAX_RETRIES = 5
BASE_DELAY = 1.0

def call_api(payload):
    for attempt in range(MAX_RETRIES):
        response = requests.post(API_URL, json=payload)
        if response.status_code < 500:
            return response.json()
        retry_after = response.headers.get("Retry-After")
        if retry_after is not None:
            delay = float(retry_after)
        else:
            delay = BASE_DELAY * (2 ** attempt) + random.uniform(0, 0.5)
        time.sleep(min(delay, 30))
    raise RuntimeError("downstream API still failing after retries")

The decision table that now lives next to this code is simple: retry on 429 with the header's delay, retry on 5xx with exponential backoff, and never retry on 4xx validation errors because the request itself is the problem. That last rule matters more than it looks, because retrying a bad payload only multiplies the damage.

The Reusable Debugging Workflow

The debugging path that found this bug is worth keeping as a checklist for any AI-generated code that talks to external services. Start with the symptom and verify whether it reproduces outside the failing environment, because a local success strongly suggests an environmental cause. Then trace the outgoing traffic rather than the incoming traffic, since the failing requests were the ones the server initiated, not the ones it received.

The next step is to look for amplification loops, which are any code path where one external failure produces multiple internal attempts. A quick grep for while True or retry in the generated code usually reveals the loop, and counting the outgoing requests per minute confirms whether the loop is amplifying the problem. The final step is to apply the retry decision table and re-run the exact failure scenario, which in this case meant simulating a downstream 503 and watching the outgoing request rate drop from forty per minute to five.

Limitations and Who Should Skip This Approach

This retry strategy is not a universal fix, and there are setups where it is the wrong tool entirely. If the downstream service requires exactly-once delivery, retries alone are insufficient because a retry can duplicate a side effect, so the handler needs idempotency keys or a deduplication layer on top of the backoff logic. The approach also assumes the caller controls the retry budget, which is not true when the client is a third party that retries aggressively on its own; in that case the server needs rate limiting on the receiving side, not just polite retries on the sending side.

Teams with a generous paid tier might never see this bug, because the quota is large enough to absorb the amplification, which is exactly why the free tier is a better place to learn the lesson. The constraint exposes the flaw early, and fixing it on a constrained budget produces code that behaves well when the traffic eventually grows.

The Practical Takeaway

AI-generated code is fast to produce and slow to trust, and this incident is a concrete example of where that trust needs to be earned. The fix was not more code but more restraint: a retry cap, a delay that respects the server's instructions, and a rule that validation errors never retry. The constrained setup that produced the bug also made the debugging cheap, because the free server's logs and quota told the whole story without any paid observability tooling.

For anyone who wants to test this pattern without risking a production account, the workflow is to generate a webhook handler, point it at a flaky endpoint, and watch what happens under a per-minute limit. The free 10-million-token allowance and free server make that experiment cost nothing to run, and the failure mode is educational precisely because it is cheap to trigger.

The 3 AM Job That Never Ran: Taming Sleep Mode on a Free Server

Blake Yang — Mon, 24 Aug 2026 19:10:13 +0000

At 3:00 AM, the log file should have shown a new entry, but it did not. The next night, the same silence appeared again. I had scheduled an AI-powered summary job to run on a free server, and after a few hours of inactivity, the server had quietly gone to sleep, taking my cron job with it. This is the story of how I found the problem, why it happens, and the wake-up pattern that fixed it.

The server was the free tier of MonkeyCode, an open-source project that offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The fix, however, is not specific to MonkeyCode; it applies to any free server with a sleep policy.

The Symptom: A Cron Job That Skips

The scheduled job was a simple Python script that summarized the previous day's logs and sent the result to a webhook. It was configured to run at 3:00 AM via cron, and it worked for the first two days. On the third day, the webhook received nothing, and the cron log showed no entries for the entire night. The server process was still running, and the system clock was correct, which ruled out the most obvious causes.

The First Suspect: Timezone and Syntax

The first step was to verify the cron configuration itself. The crontab entry used the correct server timezone, and the syntax matched the standard five-field format. A manual test of the command worked without errors. This eliminated timezone confusion and syntax mistakes, leaving the cron daemon or the server environment as the remaining suspects.

The Real Culprit: Sleep Mode

The system logs revealed the truth: the server had entered a sleep state after roughly an hour of inactivity. Many free tiers spin down idle processes to conserve resources, and the MonkeyCode free server appears to follow the same pattern. When the server sleeps, the cron daemon is suspended, so scheduled jobs simply do not fire until the server wakes up again.

The Wake-Up Pattern

The solution is to keep the server awake by sending periodic HTTP requests from an external service. A cloud-based cron service such as cron-job.org can hit a lightweight endpoint on your server every few minutes. Each request wakes the server if it is sleeping, and the endpoint can also check whether the scheduled job is due and trigger it immediately.

A Minimal Flask Wake-Up Endpoint

The following Flask application exposes a /wake endpoint and runs the scheduled job in a background thread. The external cron service calls this endpoint every five minutes, which prevents the server from sleeping and also provides a reliable trigger for the 3 AM job.

from flask import Flask
import time
import threading

app = Flask(__name__)

job_due = False
JOB_HOUR = 3

def run_scheduled_job():
    # Replace with your AI summarization logic
    print("Running the scheduled job at", time.strftime("%Y-%m-%d %H:%M:%S"))

@app.route("/wake")
def wake():
    global job_due
    job_due = True
    return "OK"

def worker():
    global job_due
    last_run_day = None
    while True:
        if job_due:
            now = time.localtime()
            if now.tm_hour == JOB_HOUR and now.tm_min < 10 and last_run_day != now.tm_yday:
                run_scheduled_job()
                last_run_day = now.tm_yday
            job_due = False
        time.sleep(1)

if __name__ == "__main__":
    threading.Thread(target=worker, daemon=True).start()
    app.run(host="0.0.0.0", port=3000)

The external cron service should be configured to send a GET request to https://your-server.example.com/wake every five minutes. The endpoint marks the job as due, and the background thread runs it exactly once per day by tracking the last run day.

The Idempotency Trap

A naive implementation can run the job twice if the external cron request arrives while the job is still executing. The fix is to add a lock that prevents concurrent runs. A simple file lock works for a single-process server, but a database-based lock is safer if you scale to multiple workers.

import os

LOCK_FILE = "/tmp/scheduled_job.lock"

def run_with_lock():
    if os.path.exists(LOCK_FILE):
        return
    open(LOCK_FILE, "w").close()
    try:
        run_scheduled_job()
    finally:
        os.remove(LOCK_FILE)

This lock is not perfect, because a crash can leave a stale lock file. A more robust approach uses an atomic filesystem operation or a database row with a unique constraint. For a free-tier experiment, the simple lock is usually enough.

Testing the Wake-Up Pattern

To verify the pattern, you can simulate sleep by stopping the Flask app, then start it again and send a wake request. The logs should show the job running if the request arrives within the scheduled window. A more thorough test uses a fake clock to confirm that the job runs exactly once even when the external cron service fires multiple times.

Limitations and Who Should Skip This

The wake-up pattern depends on an external cron service, which adds a third-party dependency and a small amount of latency. Free servers may also limit the number of requests per minute, so a five-minute interval is usually safe, but a shorter interval could trigger rate limits. If the server completely stops rather than sleeps, the wake request may not be enough to start it, and a manual restart might be required. This approach is best for jobs that can tolerate a few minutes of delay and do not need second-level precision.

The Bottom Line

Free tiers are a great way to experiment with AI workloads, but they come with infrastructure quirks that are rarely documented. Sleep mode is one of those quirks, and the wake-up pattern is a simple, server-agnostic workaround. MonkeyCode's free tier includes a 10-million-token allowance and a free server option, which makes it a convenient place to test this pattern. If you are running scheduled AI jobs on a budget, the wake-up endpoint is a small piece of code that can save you a lot of missed runs.

The Exit Code That Lied: Debugging a Silent Failure on a Free Server

Blake Yang — Sat, 22 Aug 2026 14:21:36 +0000

A scheduled sync job ran every hour on a small free server, and its logs claimed success after every single run. The target database, however, was quietly missing records that the upstream API clearly contained, which made the logs look like a deliberate lie. This article walks through that failure from the first symptom to the actual root cause, and it highlights three debugging techniques that matter more than any single fix.

The job was deliberately simple in its design. A Python script pulled new records from an upstream endpoint, inserted them into a local SQLite database, and wrote a summary line to stdout for the log. Cron invoked the script through a pipeline that appended the output to a log file, and the whole thing lived on a free server with a strict time budget and a process manager that could reclaim the job at any moment.

The first sign of trouble appeared when a comparison query showed a gap between the upstream record count and the local one. The log file ended with "sync completed" on every run, so the initial assumption was a data mismatch rather than a crash in the job itself. The operator asked an AI model, accessed through MonkeyCode's free model access, to review the script. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The model suggested three plausible improvements: add a retry loop with exponential backoff, print more diagnostic lines, and wrap per-record processing in a try/except so one bad record could not stop the batch.

All three suggestions were reasonable in isolation, and all three made the failure harder to see in practice. The retry loop re-fetched the same broken record, the extra prints vanished into the void, and the try/except converted a crash into a silent skip. With AI-assisted debugging becoming a default workflow for many teams, the lesson here is that a model's plausible suggestion is still a hypothesis, not a verified root cause. This is the moment when the debugging retrospective really begins, because the fix was never going to come from more logging or more retries.

Technique 1: Audit the real exit code

The cron entry looked like this:

0 * * * * cd /srv/sync && python3 sync.py 2>&1 | tee -a sync.log

A pipeline returns the exit status of its last command, so the reported status always came from tee rather than from Python itself. The shell keeps the real statuses in the PIPESTATUS array, and reproducing the pipeline by hand exposed the lie immediately:

cd /srv/sync
python3 sync.py 2>&1 | tee -a sync.log
echo "pipeline exit: $?"                 # always 0
echo "python exit:   ${PIPESTATUS[0]}"   # 1

The Python process had been failing for days, and the pipeline had been reporting success the entire time. Adding set -o pipefail to the wrapper script would have caught this on the very first run.

Technique 2: Force unbuffered output

The next mystery was the missing diagnostic output from the script. Python buffers stdout when it is not attached to a terminal, so prints that happened just before a crash stayed in the buffer and disappeared when the process was killed. Running the script with python3 -u or setting PYTHONUNBUFFERED=1 revealed exactly where the job stopped, which turned out to be the first record that lacked an external_id field.

Technique 3: Build a minimal reproducer

The full script was too noisy for bisection, so the operator stripped it down to the smallest failing case: one upstream call, one insert, and one deliberate raise.

# repro.py
import sys, time

print("connecting to upstream...", flush=True)
time.sleep(1)
raise KeyError("missing field: external_id")

Running this tiny script under the same pipeline and the same unbuffered mode confirmed the root cause in under a minute. The broad except Exception that the AI review had added was swallowing the KeyError, logging a single "skipped" line, and moving on as if nothing had happened.

The root cause, in three layers

The failure was not one bug but three compounding layers that reinforced each other. The pipeline masked the exit code, stdout buffering hid the last prints, and the broad exception handler converted a real error into a silent skip. The AI-suggested retry loop made the situation worse because every retry re-fetched the same broken record, and the "skipped" log line looked like progress to anyone reading the output.

The fix

The corrected shell wrapper checks the real exit status and fails loudly instead of hiding it:

#!/usr/bin/env bash
set -euo pipefail
cd /srv/sync
python3 -u sync.py 2>&1 | tee -a sync.log

The corrected Python side logs the traceback and re-raises the error instead of swallowing it:

import logging

logging.basicConfig(
    filename="sync.log",
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)

def process(record: dict) -> None:
    try:
        insert(record)
    except KeyError as exc:
        logging.exception("record %s is missing a field", record.get("id"))
        raise

A heartbeat file gave the scheduler a way to detect stalls even when the process vanished without a trace:

#!/usr/bin/env bash
# heartbeat.sh — run from cron every five minutes
last=$(stat -c %Y /srv/sync/heartbeat 2>/dev/null || echo 0)
now=$(date +%s)
if (( now - last > 300 )); then
  echo "sync job heartbeat expired" >&2
  exit 1
fi

The job itself touches the heartbeat file after every batch of records, so a stalled or killed run becomes visible within five minutes instead of days.

The reusable checklist

Check the real exit status of every pipeline with PIPESTATUS or set -o pipefail.
Run Python with -u or PYTHONUNBUFFERED=1 whenever output feeds a log file.
Never wrap a loop body in a bare except Exception that only logs a line; log the traceback and re-raise.
Add a heartbeat file for long-running jobs on servers that can reclaim processes at any time.
Make the job idempotent with a watermark so a re-run after a crash is safe.
When an AI model suggests a fix, apply the smallest change first and confirm the failure is still visible before adding retries.

Limitations and who should skip this approach

The free server is an experiment environment, not a production platform, and the constraints that made this bug visible are the same constraints that make it unsuitable for critical workloads. Teams with strict compliance or availability requirements should keep their jobs on infrastructure they control. The 10-million-token free allowance and the free server option are useful for reproductions like this one, but limits and conditions can change, so checking the current documentation before building a workflow around them is the responsible move.

The debugging techniques themselves transfer everywhere, and they are the real takeaway from this retrospective. MonkeyCode is an open-source project that pairs free model access with a free server option, which makes it a practical place to run constrained-environment experiments like this one. If that kind of debugging is a regular part of your week, the project is worth a look; the free allowance makes it easy to test a hypothesis without opening a wallet.

The Upload Endpoint Passed Every Test. Then the Security Checklist Found a Path Traversal.

Blake Yang — Fri, 21 Aug 2026 13:02:39 +0000

A file upload endpoint generated by an AI coding assistant passed every functional test in the suite. It accepted valid files, rejected oversized ones, and returned the correct status codes for each scenario. Then a fifteen-minute security checklist found a path traversal vulnerability that would have allowed an attacker to write files anywhere on the server. The endpoint was functionally correct and completely unsafe at the same time.

This gap is exactly what most AI code review workflows miss. Functional tests verify that code does what it is supposed to do, and security review verifies that code does not do what it is not supposed to do. AI models are trained on patterns that look correct, and those patterns rarely include adversarial input.

Why Functional Tests Are Not Security Tests

A test suite that checks the happy path and a few error cases tells a developer nothing about whether an endpoint is safe. The upload endpoint had tests for valid uploads, empty files, and oversized files, and all of them passed. None of the tests used a filename like ../../etc/cron.d/evil, because the test author was thinking about functionality, not about what an attacker could do.

Security issues live in the space between what the code does and what the code should be allowed to do. A functional test asks whether the code works, and a security check asks whether the code can be abused. These are different questions, and they require different review techniques.

A Security Review Checklist for AI-Generated Code

The following checklist takes about fifteen minutes to run on a typical endpoint. It is not a comprehensive security audit, but it catches the most common vulnerability classes in AI-generated code.

Input validation: Is every user-supplied value validated against an allowlist?
Path handling: Can any user input influence a file path?
SQL queries: Is every query parameterized, with no string concatenation?
Command execution: Can any user input reach a shell command?
Authentication: Is the endpoint protected by an auth check?
Authorization: Does the code verify that the user owns the resource?
Error messages: Do failures leak stack traces or internal paths?
Dependencies: Are all packages pinned to known versions?
Rate limiting: Can an attacker hammer the endpoint without consequences?
Secrets: Are any credentials hardcoded in the source?

Each item on this list maps to a concrete code pattern. The path handling check, for example, looks for open(), os.path.join(), or Path() calls that include a variable derived from request data.

The Vulnerability in Practice

The AI-generated upload endpoint stored files with a user-supplied filename, and the code was compact enough to look innocent:

@app.post("/upload")
def upload_file(request):
    filename = request.form["filename"]
    content = request.files["file"].read()
    with open(f"/uploads/{filename}", "wb") as f:
        f.write(content)
    return {"status": "ok"}

A request with filename=../../tmp/pwned would escape the uploads directory and write a file anywhere the process could write. A request with filename=../../app/main.py could overwrite the application itself, depending on file permissions.

The fix is to sanitize the filename and validate the extension against an allowlist:

from pathlib import Path

ALLOWED_EXTENSIONS = {".jpg", ".png", ".pdf"}

@app.post("/upload")
def upload_file(request):
    filename = Path(request.form["filename"]).name
    if Path(filename).suffix not in ALLOWED_EXTENSIONS:
        return {"error": "invalid extension"}, 400
    content = request.files["file"].read()
    with open(f"/uploads/{filename}", "wb") as f:
        f.write(content)
    return {"status": "ok"}

The Path().name call strips any directory components, and the allowlist rejects anything that is not an image or a PDF. The same endpoint now handles the original test cases and the adversarial ones.

Automating the Triage

A simple shell script can catch the most common vulnerability patterns in seconds, and it can run in CI on every pull request:

#!/usr/bin/env bash
# security-triage.sh - quick pattern checks for AI-generated code

echo "== SQL string concatenation =="
grep -rnE "SELECT.*(\+|f\")" --include="*.py" . || echo "OK"

echo "== open() with request-derived variables =="
grep -rnE "open\(.*(request|form|args|json)" --include="*.py" . || echo "OK"

echo "== subprocess or os.system with variables =="
grep -rnE "(subprocess|os\.system).*(\+|f\")" --include="*.py" . || echo "OK"

echo "== hardcoded secrets =="
grep -rnE "(password|secret|api_key)\s*=\s*['\"]" --include="*.py" . \
  | grep -vE "os\.environ|getenv" || echo "OK"

This script is a triage tool, not a security review. It produces false positives, and it misses anything that does not match a pattern. What it does well is catching the obvious mistakes that AI models make with surprising regularity.

Where the Free Tier Fits

MonkeyCode's free model access was used to generate the original upload endpoint, and its free server option deployed it for testing. The same free model access can review the code with a security-focused prompt, and the review is useful as a second pair of eyes. The model flagged the unsanitized filename as a potential issue, though it did not explain the full attack chain, and the confirmation came from the checklist and a manual trace of the request path.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server also makes the CI triage script practical for side projects and small teams. A scan that runs in seconds and costs nothing is a scan that runs on every pull request, and a scan that runs on every pull request catches issues before they reach production.

Limitations

This checklist covers common web application vulnerabilities, and it does not cover business logic flaws, race conditions, or issues that require deep domain knowledge. The automated script is pattern-based, which means it misses anything that does not match a known pattern and flags some things that are not vulnerabilities. A checklist and a grep script are a floor, not a ceiling.

The approach also assumes that the reviewer understands the vulnerability classes well enough to evaluate the checklist results. Someone who does not know what path traversal is will not be able to judge whether the code is vulnerable, and a checklist cannot teach that knowledge.

Who Should Not Use This Approach

Teams building payment systems, healthcare platforms, or other high-risk applications should not rely on a checklist and a pattern scan. Those systems need professional security review, penetration testing, and threat modeling, and no amount of free-tier tooling replaces that. The checklist is for developers who currently do no security review at all, and it gives them a starting point that is better than nothing.

The upload endpoint is still in production, and the security scan now runs on every pull request. The vulnerability was found by a fifteen-minute checklist, and the fix was three lines of code. That is the pattern that matters: most AI-generated code has common, predictable vulnerabilities, and common vulnerabilities can be caught with common checks.

MiniMax H3 Buzz? I'd Rather Keep a Free Model on a Short Leash

Blake Yang — Fri, 14 Aug 2026 12:03:29 +0000

Last Tuesday, my feed was full of MiniMax H3 takes. Hot takes, cold takes, screenshot takes. I caught myself scrolling for the one magic benchmark before fixing a flaky rename script. Again.

I've been burned this way before. Shiny model, zero evidence, and a half-finished eval harness from the last shiny model sitting in a folder called evals_do_not_delete.

So I did something different this time. I ignored the MiniMax H3 buzz long enough to ask: what would actually change my mind?

If you've read my earlier posts, you know I don't trust vibes. I want a small, reproducible loop where the model is free to run and the evidence is mine. That's where MonkeyCode's free model access and free server option became useful to me.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The loop I keep using is boring on purpose. One task file in. One evidence file out. No dashboard, no thumbs-up button, no “look at this one impressive generation” screen capture.

Here is the tiny harness I keep in the repo.

#!/usr/bin/env python3
'''Run one coding task through a free model endpoint and keep the evidence.'''
import json
import subprocess
import sys
import time
from pathlib import Path

TASK = Path('tasks/rename_symbol.json')
PROMPT = TASK.read_text()
MODEL_CMD = sys.argv[1:]

start = time.time()
proc = subprocess.run(MODEL_CMD, input=PROMPT, text=True, capture_output=True)
elapsed = time.time() - start

record = {
    'task': TASK.name,
    'command': MODEL_CMD,
    'elapsed_seconds': round(elapsed, 2),
    'stdout': proc.stdout,
    'stderr': proc.stderr,
    'returncode': proc.returncode,
}
Path('runs').mkdir(exist_ok=True)
Path(f'runs/{int(start)}.json').write_text(json.dumps(record, indent=2))
print(f'Saved run {int(start)}.json, rc={proc.returncode}')

I use it like this:

python eval_one_task.py monkeycode-free-runner --free

monkeycode-free-runner is just a stand-in for whatever command points at my free endpoint. The point is not the wrapper. The point is that every run lands in runs/ as plain JSON.

My task file looks like this:

{
  "id": "rename_symbol_001",
  "repo": "sample_python",
  "instruction": "Rename helper_function to parse_id in parser.py and update call sites.",
  "expected_files": ["parser.py", "tests/test_parser.py"]
}

Notice what the harness does not do. It does not call the model good or bad. It does not parse the output. It does not take a screenshot. It just keeps evidence.

The review step stays manual. I diff the files the model touched, run the tests, and check whether anything outside expected_files changed. If it did, that is a finding, not a vibe.

For a lot of tasks, the free run is enough. For other tasks, it is not. I keep the decision table short.

Task	Use free loop?	Why
Rename a symbol across a small repo	Yes	Small blast radius, easy to diff
Fix a flaky test in an isolated module	Yes	Clear pass/fail signal
Refactor auth code with secrets	No	Don't send secrets to a third-party server
Generate a one-off SQL report	Maybe	Run only against local mock data

The free server option matters to me because reruns are cheap. I can run the same task again after a model update without burning a paid quota or my laptop battery. That is where the loop stops being a one-off demo and starts being a habit.

Now about the open spirit. I keep hearing open source thrown at every model release. If a maintainer wants to show me a license, great. I am not going to call MonkeyCode open source unless that license says so. What I care about is the workflow staying open: a command I can run, a free server option I can point it at, and a plain JSON trail I can diff. That is the part I do not want to give up when the MiniMax H3 threads disappear.

This loop is not magic. It will not tell you that one model is better than another. It will not stop a bad suggestion from entering your editor. It only removes the easiest excuses for not checking.

If your code cannot leave your machine, do not send it to any free server. If you need per-request audit logs or a compliance reviewer, build that separately. If you are hoping for a leaderboard from one run, this is not the tool for you. Free access can change, so I keep the task files portable and do not let a single vendor own the format.

The MiniMax H3 wave will pass. The evidence folder will still be there.

If you already have a free endpoint that lets you keep your evidence outside a walled dashboard, try running the same boring task twice: once today, once after the next model drop. The diff between those two runs is the only benchmark I trust.

My Free-Server Loop for Reviewing AI Refactors (and the First Thing It Broke)

Blake Yang — Fri, 14 Aug 2026 10:43:45 +0000

I keep catching myself doing the same AI review loop by hand. Ask a model to refactor a function, read the diff, run the tests, notice the patch does not apply, ask again, run the tests again. The model is not the slow part. I am the slow part, because I never wrote the loop down.

So I made it boring.

The point is not to trust the model more. The point is to create a cheap repeatable signal before I open a real PR. If a proposed refactor passes a baseline test suite, I will read it carefully. If it fails, I skip it and stop turning my evening into a vibes check.

A lot of AI coding advice right now feels like either 'trust the agent' or 'never trust the agent.' Both are feelings. I wanted a tiny referee instead.

The free-tier setup

The two MonkeyCode pieces I used for this experiment are the free model access and the free server option. (Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am treating those availability claims as operator-supplied, not a permanent guarantee.) The model proposes a diff. The server does the unglamorous work: apply the patch, run tests, and report whether the repo still passes.

I am deliberately not calling this a benchmark. One refactor is one sample, and a green test run does not prove a change is correct. It only proves the change did not break the checks I already had.

The runner I copied

This is a proposal script, not a production harness. I used a generic OpenAI-style client because many low-cost and free model paths expose one. Replace the base URL, model name, and repo path with your own values.

import json
import os
import subprocess
import sys
from pathlib import Path

JOBS = Path('jobs.json')
PROJECT_DIR = Path('./repo')


def run_tests():
    result = subprocess.run(
        ['pytest', '-q', '--disable-warnings'],
        cwd=PROJECT_DIR,
        capture_output=True,
        text=True,
        timeout=120,
    )
    return result.returncode == 0, result.stdout + result.stderr


def ask_model(prompt):
    from openai import OpenAI

    client = OpenAI(
        base_url=os.environ['MODEL_BASE_URL'],
        api_key=os.environ['MODEL_API_KEY'],
    )
    response = client.chat.completions.create(
        model=os.environ.get('MODEL_NAME', 'free-model'),
        messages=[{'role': 'user', 'content': prompt}],
        temperature=0.2,
    )
    return response.choices[0].message.content


def main():
    baseline_ok, baseline_log = run_tests()
    print('baseline ok:', baseline_ok)
    if not baseline_ok:
        print(baseline_log)
        sys.exit(1)

    jobs = json.loads(JOBS.read_text())
    for job in jobs:
        proposed_diff = ask_model(job['prompt'])
        patch_path = Path('candidate.patch')
        patch_path.write_text(proposed_diff)

        check = subprocess.run(
            ['git', 'apply', '--check', str(patch_path)],
            cwd=PROJECT_DIR,
            capture_output=True,
            text=True,
        )
        if check.returncode != 0:
            print(job['id'], '-> SKIP: patch did not apply')
            continue

        subprocess.run(['git', 'apply', str(patch_path)], cwd=PROJECT_DIR, check=True)
        ok, log = run_tests()
        subprocess.run(['git', 'apply', '-R', str(patch_path)], cwd=PROJECT_DIR, check=True)

        print(job['id'], '->', 'PASS' if ok else 'FAIL')
        if not ok:
            print(log[-800:])


if __name__ == '__main__':
    main()

Sample jobs.json:

[
  {
    "id": "extract-validator",
    "prompt": "Return only a git patch for the validate_email function in src/validators.py. Replace the nested conditionals with a clearer guard clause. Do not change public behavior."
  }
]

The baseline check matters more than the model. If the repo does not pass before the model runs, the loop is meaningless. My first dry run failed because pytest discovered zero tests and exited successfully. That was not a model failure; it was a harness failure. Now I make the suite catch a deliberate failing test before I add any model work.

A routing table instead of a robot

Risk	Example change	Route	Why
Low	Variable rename, formatting	Free model plus local tests	Cheap and easy to revert
Medium	Multi-file behavior change	Free server runner plus targeted tests	The longer run can happen off my laptop
High	Auth, payment, or security-sensitive path	Human review first	A free model should not be the final gate

Where the free server actually helps

The free server option matters for the long-running part. I can leave the runner pointed at a small repo, schedule it against a small queue, and let it churn through low-risk mechanical patches while I do other work. The server does not make the model smarter. It removes the excuse I usually have for not running the loop.

Limitations and when to skip this

A free server is not a CI system. It may not have an SLA, durable storage, or enough time for large suites.
Free model quotas can change. Do not send proprietary or regulated code into this loop.
Passing tests is a floor, not proof. A bad suite will happily bless a bad refactor.
This is triage, not model evaluation. Do not rank models from one or two patches.

Who should not use this: anyone handling production secrets, compliance-heavy code, or repositories where a mistaken apply could be expensive. Start with a toy repo or a branch you can delete.

If you try this, start with one mechanical refactor and deliberately make the baseline test fail once. If your runner catches it, the loop is working. Then you can go back to arguing with the model about naming, but at least the boring part is already handled.

Route AI Coding Tasks by Risk: A Free-Tier-First Workflow You Can Actually Measure

Blake Yang — Thu, 13 Aug 2026 03:21:46 +0000

Most discussions about AI coding tools start with "which model is best?" I've found that's the wrong first question. The better question is: which of my tasks actually need the strongest model, and which ones don't?

In my earlier posts I wrote about building a small evaluation suite for AI coding models and a falsification loop for reviewing AI-generated refactors. This post is the missing piece between them: a routing layer that decides, per task, whether a free-tier model is good enough — and a way to measure whether that decision was right, instead of trusting vibes.

The problem: paying frontier prices for boilerplate work

When every prompt goes to the most expensive model by default, two things happen:

You burn budget on tasks a weaker model handles fine (renaming, boilerplate, docstrings, simple test generation).
You never build intuition for where the strong model genuinely matters, because you never see the failure distribution of the cheap one.

The fix isn't a blog-post benchmark. It's a per-task routing rule plus a log you can audit weekly.

Step 1: Classify tasks by blast radius, not difficulty

Difficulty is subjective. Blast radius — what breaks if the output is wrong and you don't catch it — is not. I use three tiers:

Tier	Task examples	Failure cost	Default route
Low	Rename/refactor with compiler backing, boilerplate, doc comments, unit test scaffolding, commit message drafts	Caught by compiler/CI in seconds	Free/cheap model
Medium	New function in an existing module, bug fix with a clear reproducer, small migration script	Caught by code review or tests, costs an hour	Free model first, escalate on failure
High	Concurrency changes, auth/payment logic, schema migrations on live data, security-sensitive parsing	May reach production silently	Strongest available model + mandatory human review

Two rules make this table work:

Escalation is cheap, so bias toward the free tier. If the free model's output fails your checks, you escalate that one task. You lose minutes, not money.
Tier High is non-negotiable. Anything whose failure mode is "silent wrongness in production" never starts on the free tier, no matter how confident you feel.

Step 2: A gate that every AI output must pass

Routing only works if each tier has an objective accept/reject gate. Mine is:

#!/usr/bin/env bash
# gate.sh — run after applying any AI-generated change.
# Exit 0 = accept, non-zero = escalate to a stronger model or do it by hand.
set -e

echo "== typecheck =="
npx tsc --noEmit          # swap for: mypy, go build, cargo check...

echo "== existing tests =="
npm test -- --silent      # must pass with zero new failures

echo "== diff sanity =="
# Reject diffs that touch files outside the task's declared scope.
# I pass the allowed path prefix as $1, e.g. ./gate.sh src/billing/
git diff --name-only | grep -v "^$1" && {
  echo "FAIL: change escaped declared scope"; exit 1;
} || true

echo "PASS"

The scope check matters more than it looks. In my experience the most common free-tier failure isn't wrong logic — it's the model "helpfully" editing files you didn't ask about. A one-line git diff filter catches that class entirely.

Step 3: Log every routed task, review weekly

This is the part that turns routing from a superstition into a measurement. One line of JSON per task:

{"date":"2026-08-11","tier":"low","route":"free","gate":"pass","escalated":false,"minutes":6}
{"date":"2026-08-11","tier":"medium","route":"free","gate":"fail","escalated":true,"minutes":19}
{"date":"2026-08-11","tier":"high","route":"strong","gate":"pass","escalated":false,"minutes":31}

After two weeks, answer three questions from the log:

What percentage of free-routed tasks passed the gate on the first attempt? (My threshold: if it drops below ~60% for a tier, that tier's routing rule is wrong.)
When tasks escalated, did the strong model actually fix it, or was the task misclassified as Medium when it was really High?
Are High-tier tasks sneaking into the free route? (Any "yes" here is a process bug, fix the table, not the model.)

This is deliberately the same philosophy as my earlier evaluation-suite post: small, runnable, and honest about failure counts instead of average-case impressions.

Where the free tier comes from

Routing toward a free tier only helps if you actually have one. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option, which is what makes a free-first routing rule practical to run as an individual — the "free route" in the table above is a real default rather than a hypothetical one, and the free server means the logging/gate scripts can run somewhere other than your laptop. I won't quote specific model names, quotas, or performance numbers here, because those change and you should verify them yourself against the current offering; the workflow in this post is deliberately provider-agnostic, and the gate + log will tell you within two weeks whether the free tier is pulling its weight for your codebase.

Limitations, and who shouldn't do this

Tiny codebases and solo weekend projects: if you write ten AI-assisted tasks a week, the log overhead exceeds the savings. Just use whatever model and move on.
Domains where correctness is unverifiable by tests (e.g., ML feature engineering without ground truth, UX copy): the gate script can't catch silent wrongness, so routing by gate results gives false confidence.
Regulated or security-critical code: the High tier in my table should probably be "no AI generation at all, AI-assisted review only." A routing table is not a compliance story.
The classification itself is a judgment call. Expect to misclassify for the first two weeks; the weekly log review exists precisely to correct that.

The takeaway

"Which model is best" is a benchmark question. "Which model is sufficient for this task, and how would I know if it wasn't" is an engineering question. A blast-radius table, an objective gate, and a one-line-per-task log will answer it for your own workflow in about two weeks — and whatever free tier you route to, you'll know exactly how much it's earning its place. If you try this, I'd genuinely like to hear what your pass-rate numbers look like; that's the dataset nobody publishes.

How to Review AI-Generated Refactors: A Repeatable Falsification Loop

Blake Yang — Fri, 07 Aug 2026 08:04:47 +0000

Can you trust an AI-generated refactor without re-doing the work yourself? Yes — if you stop treating the diff as a patch to approve and start treating it as a hypothesis to falsify. The workflow below is a repeatable, scripted loop (snapshot behavior → generate in isolation → mechanical falsification → human review) that shrinks a 400-line AI diff into something your judgment can actually handle.

Last month I wrote about building a free evaluation suite for comparing AI coding models. That suite answers which model to trust. This article is about the next question, which turned out to be harder in practice: once you've picked a model, how do you review its output on real refactoring work without either rubber-stamping it or re-doing the work yourself?

The failure mode I kept hitting: an AI-generated refactor looks clean, passes a quick skim, and then breaks an edge case three files away. Manual review of a 400-line diff is exactly where attention fails. So I built a small, repeatable loop that treats every AI refactor as a hypothesis to falsify, not a patch to approve.

The loop in one picture

Snapshot behavior before the refactor. Capture the current behavior as executable evidence (tests + a few golden outputs), not as a vague memory of "it worked."
Generate the refactor in isolation. One concern per request, with an explicit diff scope.
Run mechanical falsification. Tests, type checks, and a diff-shape check that flags changes outside the declared scope.
Only then do human review. By this point the diff is smaller and already survived the cheap checks.

The key idea is that steps 1–3 are scripted and identical every time, so the human judgment in step 4 is spent where it actually matters. This mirrors the classic advice behind characterization tests: pin down what the code observably does before you restructure it, because a refactor by definition must not change behavior.

Artifact: the falsification script

This is the core of the workflow. It's deliberately boring shell — no framework to maintain:

#!/usr/bin/env bash
# falsify-refactor.sh — run after applying an AI-generated refactor
set -euo pipefail

SCOPE_FILE="${1:-.refactor-scope}"
BASE_BRANCH="${2:-main}"

echo "== 1. Test suite =="
npm test -- --silent || { echo "FAIL: tests"; exit 1; }

echo "== 2. Type check =="
npx tsc --noEmit || { echo "FAIL: types"; exit 1; }

echo "== 3. Scope check: did the diff stay inside the declared files? =="
CHANGED=$(git diff --name-only "$BASE_BRANCH"...HEAD)
VIOLATIONS=0
while read -r f; do
  if ! grep -qxF "$f" "$SCOPE_FILE"; then
    echo "  OUT OF SCOPE: $f"
    VIOLATIONS=1
  fi
done <<< "$CHANGED"
[ "$VIOLATIONS" -eq 0 ] || { echo "FAIL: scope"; exit 1; }

echo "== 4. Behavior snapshot =="
# Golden outputs captured BEFORE the refactor (see below)
node scripts/capture-behavior.js | diff -u .golden/behavior.txt - \
  || { echo "FAIL: behavior drift"; exit 1; }

echo "PASS: refactor survived falsification"

Before requesting the refactor, I capture the golden behavior once:

git checkout main
node scripts/capture-behavior.js > .golden/behavior.txt

capture-behavior.js is just a script that exercises the module's public API with representative inputs and prints normalized output — a poor man's characterization test for code that doesn't have full test coverage yet.

The scope file is the part people skip, and it's the part that catches the most real damage. When I ask a model to "extract the validation logic from checkout.ts into a pure module," I write down exactly which files may change:

src/checkout.ts
src/validation.ts
src/validation.test.ts

If the model also "helpfully" tweaks an unrelated utility, the script fails loudly instead of that change slipping through inside a big diff.

Where the free model access fits

This loop is generation-heavy: each refactor request is cheap to check but not cheap to produce, and I often want two or three candidate diffs for the same task so I can compare approaches. Running that volume through a paid API adds up fast for what is essentially iterative drafting.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

In my current setup I run the generation step through MonkeyCode, which offers free access to coding models and a free server option for hosting the workflow. That combination matters here for a specific reason: the falsification script is fully automated, so I can point it at the server, batch several candidate refactors, and only look at the ones that survive. The economics of "generate three, discard two" stop mattering when the generation step is free.

The important thing is that the loop doesn't depend on any particular model. If the free tier disappears tomorrow, the script doesn't change — only the generation source does.

Decision table: when a candidate diff is worth human review

After running this loop for a few weeks, I noticed my review decisions collapsed into a small table:

Tests	Types	Scope	Behavior	Verdict
✅	✅	✅	✅	Review properly — worth your time
❌ any	—	—	—	Discard, don't debug the model's diff
✅	✅	❌	✅	Re-scope: ask again with a tighter prompt
✅	✅	✅	❌	Interesting: either the refactor changed semantics, or your golden snapshot was wrong. Check the snapshot first.

The last row is the underrated one. About a third of my "behavior drift" failures were actually stale golden outputs, which means the loop doubles as a cheap audit of how well I understand my own code's observable behavior.

Limitations, honestly

The golden snapshot is only as good as the inputs you thought of. This is characterization testing, not proof. A refactor can pass and still break an input you never captured.
It doesn't work for behavior-changing work. This loop is for refactors — semantics-preserving changes. Feature work needs actual tests, not diffed output.
Shell + git diff is fragile on monorepos with generated files. You'd want path filters and to exclude lockfiles from the scope check.
Free model access has practical limits. Throughput, context size, and availability can vary, so I keep prompts narrow (one concern per request) rather than shipping whole subsystems at once.

Who should not use this

If your codebase already has fast, high-coverage tests, steps 1 and 4 are mostly redundant — your CI is already the falsification loop, and you just need the scope check. And if your changes are usually behavior-changing features rather than refactors, this whole framing will slow you down.

Try it this week

If, like me, you maintain code with patchy coverage and you're increasingly letting models do the mechanical restructuring, a scriptable falsification step is the difference between "AI-assisted refactoring" and "AI-generated risk." Here's how to start today:

Copy the falsify-refactor.sh script above into your repo and adapt the test/type commands to your stack.
Pick one low-risk module and write a capture-behavior.js that exercises its public API with 5–10 representative inputs.
Run your next AI refactor through the loop — with a scope file, even if it feels bureaucratic. Watch what it catches.
If you want the batch-generation part without paying per candidate, MonkeyCode's free model access and free server are one low-friction way to run it — the script above works regardless of what generates the diffs.

What does your review process for AI-generated diffs look like? Drop a comment — I'm especially curious how people handle the scope-creep problem on larger refactors, and I'll fold the best answers into a follow-up post.

A Repeatable Loop for Reviewing AI-Generated Refactors Before You Merge Them

Blake Yang — Fri, 07 Aug 2026 03:03:08 +0000

The loop in one picture

Snapshot behavior before the refactor. Capture the current behavior as executable evidence (tests + a few golden outputs), not as a vague memory of "it worked."
Generate the refactor in isolation. One concern per request, with an explicit diff scope.
Run mechanical falsification. Tests, type checks, and a diff-shape check that flags changes outside the declared scope.
Only then do human review. By this point the diff is smaller and already survived the cheap checks.

The key idea is that steps 1–3 are scripted and identical every time, so the human judgment in step 4 is spent where it actually matters.

Artifact: the falsification script

This is the core of the workflow. It's deliberately boring shell — no framework to maintain:

#!/usr/bin/env bash
# falsify-refactor.sh — run after applying an AI-generated refactor
set -euo pipefail

SCOPE_FILE="${1:-.refactor-scope}"
BASE_BRANCH="${2:-main}"

echo "== 1. Test suite =="
npm test -- --silent || { echo "FAIL: tests"; exit 1; }

echo "== 2. Type check =="
npx tsc --noEmit || { echo "FAIL: types"; exit 1; }

echo "== 3. Scope check: did the diff stay inside the declared files? =="
CHANGED=$(git diff --name-only "$BASE_BRANCH"...HEAD)
VIOLATIONS=0
while read -r f; do
  if ! grep -qxF "$f" "$SCOPE_FILE"; then
    echo "  OUT OF SCOPE: $f"
    VIOLATIONS=1
  fi
done <<< "$CHANGED"
[ "$VIOLATIONS" -eq 0 ] || { echo "FAIL: scope"; exit 1; }

echo "== 4. Behavior snapshot =="
# Golden outputs captured BEFORE the refactor (see below)
node scripts/capture-behavior.js | diff -u .golden/behavior.txt - \
  || { echo "FAIL: behavior drift"; exit 1; }

echo "PASS: refactor survived falsification"

Before requesting the refactor, I capture the golden behavior once:

git checkout main
node scripts/capture-behavior.js > .golden/behavior.txt

src/checkout.ts
src/validation.ts
src/validation.test.ts

If the model also "helpfully" tweaks an unrelated utility, the script fails loudly instead of that change slipping through inside a big diff.

Where the free model access fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The important thing is that the loop doesn't depend on any particular model. If the free tier disappears tomorrow, the script doesn't change — only the generation source does.

Decision table: when a candidate diff is worth human review

After running this loop for a few weeks, I noticed my review decisions collapsed into a small table:

Tests	Types	Scope	Behavior	Verdict
✅	✅	✅	✅	Review properly — worth your time
❌ any	—	—	—	Discard, don't debug the model's diff
✅	✅	❌	✅	Re-scope: ask again with a tighter prompt
✅	✅	✅	❌	Interesting: either the refactor changed semantics, or your golden snapshot was wrong. Check the snapshot first.

Limitations, honestly

The golden snapshot is only as good as the inputs you thought of. This is characterization testing, not proof. A refactor can pass and still break an input you never captured.
It doesn't work for behavior-changing work. This loop is for refactors — semantics-preserving changes. Feature work needs actual tests, not diffed output.
Shell + git diff is fragile on monorepos with generated files. You'd want path filters and to exclude lockfiles from the scope check.
Free model access has practical limits. Throughput, context size, and availability can vary, so I keep prompts narrow (one concern per request) rather than shipping whole subsystems at once.

Who should not use this

But if, like me, you maintain code with patchy coverage and you're increasingly letting models do the mechanical restructuring, a scriptable falsification step is the difference between "AI-assisted refactoring" and "AI-generated risk." If you want to try the batch-generation part without paying per candidate, MonkeyCode's free model access and free server are one low-friction way to run it — the script above works regardless of what generates the diffs.

What does your review process for AI-generated diffs look like? I'm especially curious how people handle the scope-creep problem on larger refactors.

Stop Vibes-Testing AI Coding Models: A Repeatable Evaluation Suite You Can Run for Free

Blake Yang — Wed, 05 Aug 2026 06:55:44 +0000

Most developers evaluate a new AI coding model the same way: open a chat, type "write a REST API", nod at the output, and either subscribe or move on. I have done this too, and it is a terrible method. The output always looks competent on the first prompt, and the model's real weaknesses only show up on the fifth refactor, the ambiguous requirement, or the codebase question it answers with confident fiction.

This article is a small, repeatable alternative: a fixed suite of eight prompts, a scoring rubric, and a harness that records everything so you can compare models (or the same model a month later) on evidence instead of vibes. You can run the whole thing on free tiers — I will note one option below — so cost is not an excuse to skip it.

Why one-shot prompts mislead you

A single prompt conflates three different capabilities:

Fluency — producing syntactically plausible code. Nearly every current model passes this.
Specification fidelity — doing what you actually asked, including the boring constraints ("no external dependencies", "must handle empty input").
Honesty under uncertainty — saying "I don't know this API" instead of inventing one.

Fluency is what a demo measures. The other two are what determine whether the model saves you time or creates debugging debt. Your evaluation suite should probe all three separately.

The suite: eight prompts, three categories

Store these as files so the suite is versioned and re-runnable. Adjust the domain to your actual work — a frontend developer should swap the systems prompts for component tasks — but keep the categories:

evals/
  spec_fidelity/
    01_constrained_api.md      # implement X with explicit constraints
    02_refactor_with_tests.md  # change behavior, keep tests green
    03_boring_edge_cases.md    # empty input, unicode, large input
  honesty/
    04_obscure_library.md      # asks about a niche/deprecated API
    05_ambiguous_requirement.md# under-specified task; does it ask or guess?
  workflow/
    06_debug_this.md           # broken code + failing test output
    07_explain_diff.md         # explain a non-obvious diff
    08_multi_step_plan.md      # plan a migration, then execute step 1

Example prompt, 01_constrained_api.md:

Write a Python function `retry_with_backoff(fn, retries, base_delay)`.
Constraints:
- Standard library only.
- Exponential backoff with full jitter.
- Raise the last exception after retries are exhausted.
- Include type hints and one usage example.
Do not explain the code; output code only.

The constraints are the point. A model that adds tenacity as a dependency or skips jitter has failed specification fidelity even if the code runs.

The harness

The harness is deliberately boring: run each prompt, save the raw response, and fill in a scorecard. Automation can check mechanical constraints (did it import a banned package? does the code parse?); judgment calls (did it ask a clarifying question?) stay manual. Here is a minimal Python sketch — label it as a starting point, adapt it to whatever API you are testing:

# eval_runner.py — adapt endpoint/auth to your provider
import json, subprocess, sys, time
from pathlib import Path

SUITE = Path("evals")
RESULTS = Path("results") / time.strftime("%Y-%m-%d_%H%M")
RESULTS.mkdir(parents=True, exist_ok=True)

def run_prompt(text: str) -> str:
    """Send `text` to the model under test, return raw response.
    Replace the body with your provider's SDK or HTTP call."""
    raise NotImplementedError  # wire up your model endpoint here

def mechanical_checks(prompt_path: Path, response: str) -> dict:
    checks = {"response_nonempty": bool(response.strip())}
    if "constrained_api" in prompt_path.name:
        checks["no_external_import"] = "import tenacity" not in response
        checks["mentions_jitter"] = "jitter" in response.lower() or "random" in response
    return checks

for category in ("spec_fidelity", "honesty", "workflow"):
    for prompt_file in sorted((SUITE / category).glob("*.md")):
        prompt = prompt_file.read_text()
        try:
            response = run_prompt(prompt)
        except NotImplementedError:
            sys.exit("Wire up run_prompt() to your model endpoint first.")
        record = {
            "prompt": str(prompt_file),
            "response": response,
            "checks": mechanical_checks(prompt_file, response),
        }
        out = RESULTS / f"{category}__{prompt_file.stem}.json"
        out.write_text(json.dumps(record, indent=2))
        print(f"saved {out}")

Then score each response 0–2 on the rubric:

Score	Meaning
0	Wrong, fabricated, or ignores constraints
1	Partially correct; usable after real editing
2	Correct and constraint-faithful; minor polish at most

Total per model: 0–16. That number is not a benchmark of the model in general — it is a measurement of this model on work that looks like yours, which is the only measurement that should drive your tooling decisions.

What to actually look for in each category

Honesty prompts (04, 05) are the most revealing. For 04_obscure_library.md, ask about a real but niche API you know well, or an older version of a popular one. A model that hallucinates function signatures here will hallucinate them in your real work, and those bugs are expensive because they compile in your head while you read them. For 05_ambiguous_requirement.md, the best response asks one or two sharp clarifying questions before coding; a model that confidently guesses is a liability on under-specified tickets.

Workflow prompts (06–08) test whether the model can operate on your artifacts — a failing test log, a diff, a partial migration — rather than generating greenfield code, which is where most real usage happens after week one.

Running it for free

Evaluation should not require a paid subscription to the thing you are evaluating — that is backwards. Free tiers and trial credits from the major providers work, and there are also coding platforms that bundle free model access.

One option I used for a run of this suite: MonkeyCode offers free model access and a free server option, which was enough to execute the eight-prompt suite end to end without touching a credit card.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness above is provider-agnostic on purpose — point run_prompt() at whatever endpoint you have, including a local model if you have the hardware for one. If you want to try the suite on MonkeyCode's free tier, the setup takes a few minutes and the results directory slots straight into the comparison step below.

Comparing runs without fooling yourself

When you have two or more results/ folders (different models, or the same model across months):

Compare per prompt, not just totals. A model that wins 14–12 but loses the two honesty prompts is the riskier daily driver.
Read the diffs on prompts where scores differ. The raw responses tell you why; totals never do.
Re-run before concluding. A single run conflates model behavior with sampling luck. Two or three runs per prompt, with the rubric applied blindly if you can manage it, is meaningfully better.

A recent DEV discussion on why sub-agent metrics are not comparable to main-thread metrics makes a related point worth internalizing: numbers only mean something within the context that produced them. Your 0–16 score is valid inside your suite, your rubric, and your domain — do not treat it as a leaderboard entry.

Limitations and who should skip this

Eight prompts is a smoke test, not a benchmark. It will reliably separate "clearly bad fit" from "worth a two-week trial", but it will not rank two good models against each other with confidence. For that, you need real tasks from your issue tracker and a longer evaluation window.
The rubric is subjective at the boundaries. The 1-vs-2 call varies between reviewers. If a team runs this, calibrate on two or three example responses together first.
Free tiers have constraints. Rate limits, queueing, or model availability can affect your runs, and a free tier may not expose the exact model you would pay for. Check what you are actually being served, and do not extrapolate latency measurements from a free server to production expectations.
If your work is highly specialized (embedded C, formal verification, regulated-industry code), a generic suite tells you little. Build the eight prompts from your own domain or skip the exercise and run a supervised trial on real tickets instead.

The takeaway

The model that demos best is rarely the model that fails best. A fixed, versioned prompt suite — even a small one — converts "I tried it and it seemed fine" into a comparable artifact you can rerun whenever a new model, a new pricing tier, or a quiet capability change lands. The suite in this article is a starting skeleton: steal the structure, replace the prompts with your own work, and keep the honesty category no matter what.

If you end up publishing your own version of the suite, I would be curious which category surprised you most.