DEV Community: Finley Sun

Mutations Catch What Your Agent Patch Misses

Finley Sun — Wed, 02 Sep 2026 13:47:33 +0000

Mutations Catch What Your Agent Patch Misses

A date parser broke in production. The agent patch looked correct. All unit tests passed. Two weeks later, a daylight-saving edge case reappeared. The tests never checked the patch's blind spots.

This is the quiet failure mode of AI-generated code. The patch changes behavior silently. The suite stays green. You merge. Then drift shows up months later. The standard remedy is more tests. But tests only cover what the developer knows to check. Agents produce code with unknown unknowns.

Mutation testing attacks that gap. It injects small defects into the source. Then it runs the suite against each mutant. If a test fails, the mutant is killed. If a mutant survives, the suite has a blind spot. Every survivor marks an untested assumption.

I combined mutation testing with property-based checks and a flaky-test freeze. I ran the whole loop on a free server. The setup is reproducible. You can copy it.

Start with a function that parses ISO dates.

from datetime import datetime

def parse_iso(value: str) -> datetime:
    return datetime.fromisoformat(value.replace('Z', '+00:00'))

An agent patch adds support for ISO week dates. It passes the existing tests. But does the new branch have real protection? A plain unit test checks one example.

def test_week_date():
    assert parse_iso('2026-W01-1').year == 2026

That is a single point. Property-based testing explores the space. Hypothesis generates many strings matching the pattern.

from hypothesis import given, strategies as st

@given(st.from_regex(r'[0-9]{4}-W(?:[0-4][0-9]|5[0-3])-[0-9]'))
def test_week_dates_parse_without_error(value):
    parsed = parse_iso(value)
    assert parsed.year >= 2000

This catches format errors. It does not validate calendar correctness. Week 53 of some years does not exist. The property test may miss it.

Now run mutation testing. I used mutmut against the patched file.

$ mutmut run --paths-to-mutate parse.py

The report shows surviving mutants. Each survivor is a place where tests do not care. One survivor changed the week boundary from 53 to 52. No test noticed. The property test generated a range, but the valid week count depends on the year.

I added explicit fixtures for the known boundaries.

FIXED_BOUNDARIES = [
    ('2020-W53-7', True),
    ('2021-W53-7', True),
    ('2026-W53-1', False),
]

The principle is simple. Property tests generate broad input. Fixtures pin rare known values. Mutation testing tells you where the two still miss together.

Flaky tests create another layer of risk. A test that passes sometimes and fails sometimes gives false confidence. I froze flaky tests with a rerun policy.

@pytest.mark.flaky(reruns=3, only_rerun=['AssertionError'])
def test_week_boundary():
    ...

The freeze is explicit. A flaky test does not block the merge. It also does not lie silently. The rerun policy records the evidence.

The whole loop runs every night on a free server. MonkeyCode provides free model access and a free server option. The agent uses the model access to generate candidate patches. The server runs the test loop. I do not provision my own CI hardware for this verification step.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Here is the full loop. First the agent generates a patch. Then the existing suite runs. Property tests follow. Mutation testing comes next. Inspect any surviving mutants. Add fixtures for the missing boundaries. Mark flaky tests with a rerun policy. Repeat until no high-value mutant survives.

The order matters. Property tests find blasts. Mutation tests find blind spots. Fixtures turn blind spots into fixed landmarks. The flaky freeze prevents noise from hiding real failures.

The result is a measurable gate for agent patches. You still need a human to review intent. But mutation testing converts vague trust into a repeatable number.

Who should not do this? Throwaway scripts and short-lived demos do not need the overhead. Mutation testing consumes CPU time. Free servers have resource limits. Use it for code that will be maintained for quarters, not days.

Try this workflow on your next agent patch. Your future self will thank you when the mutation that would have slipped through fails loudly.

Catch Silent Behavior Drift in Agent Patches with Golden Snapshots

Finley Sun — Tue, 01 Sep 2026 12:58:17 +0000

Last Tuesday, an agent patch turned every 404 into a 403. All tests passed. The patch changed the error handler after the assertions ran. That is not a test failure. That is a blind spot.

Most test suites only assert what you remembered. They cannot see the shadows. Agent patches love shadows. They move code. They rename symbols. They reorder branches. The behavior stays the same until it does not.

I use behavioral snapshots to catch that drift. The idea is painfully boring. Record the output of fixed probes. Run the same probes after the patch. Compare the fingerprints. If they differ, the patch changed something unintended. You get a diff. No guesses. No flaky tests.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The infrastructure is intentionally cheap. I run the harness on MonkeyCode's free server. I use the free model token pool of 10 million tokens to generate probe inputs. That pool pays for edge cases. No GPU is needed. No paid CI runner.

The core harness is a single Python script. It reads fixture files. It executes your function in a subprocess. It hashes the JSON output. Here is the code.

import json
import hashlib
import subprocess
import sys

def run_probe(func_file, input_data):
    proc = subprocess.run(
        [sys.executable, func_file],
        input=json.dumps(input_data),
        capture_output=True,
        text=True
    )
    if proc.returncode != 0:
        return {"crashed": proc.stderr[-200:]}
    return json.loads(proc.stdout)

def snapshot(func_file, fixtures):
    out = {}
    for name, data in fixtures.items():
        result = run_probe(func_file, data)
        raw = json.dumps(result, sort_keys=True)
        out[name] = hashlib.sha256(raw.encode()).hexdigest()
    return out

That is the whole engine. You might want to filter volatile keys first. Add a cleaner function.

def scrub(obj, volatile):
    if isinstance(obj, dict):
        return {k: scrub(v, volatile) for k, v in obj.items() if k not in volatile}
    if isinstance(obj, list):
        return [scrub(i, volatile) for i in obj]
    return obj

Run snapshot against the base commit. Store the JSON file. Check out the agent patch. Run it again. Diff the two files. The exit code is nonzero when anything moves.

Let me show you a concrete wrapper. Suppose your code has a payment function.

# probe_payment.py
import sys, json
from payments import process

def main():
    payload = json.load(sys.stdin)
    result = process(payload["amount"], payload["currency"])
    print(json.dumps({
        "status": result.status,
        "code": result.code
    }))

if __name__ == "__main__":
    main()

Your fixtures file contains extreme values.

{
  "zero": {"amount": 0, "currency": "USD"},
  "negative": {"amount": -5, "currency": "USD"},
  "huge": {"amount": 10000000, "currency": "EUR"}
}

The model-generated probes follow a simple prompt. I ask it to list five inputs that would break the function. The free token pool covers that request. I paste the JSON into the fixture file. The model suggests weird values I would never pick. I do not run it at test time. That would be non-deterministic. I run it once, offline, and commit the fixtures.

Now add the cron job. On the free server, I set up a ten-minute loop.

*/10 * * * * /opt/snapshot/run.sh

The shell script pulls the candidate branch, rebuilds the fingerprint, and compares. If the diff is not empty, it logs the failure and exits with code 1.

#!/bin/bash
cd /opt/snapshot
git fetch origin agent-patch
git checkout agent-patch
python build_snapshot.py > candidate.json
diff baseline.json candidate.json || exit 1

That is it. No test framework. No dependencies beyond Python and git.

Why does this work? Because agent patches usually preserve the happy path. They break the quiet corners. The happy path has assertions. The quiet corners have nothing. Snapshots give those corners a witness.

The 404-to-403 case would have produced a different status code. The fingerprint would change. The merge would stop. A human would look at the diff. The bug would never ship.

There are limits. Deterministic code only. If your function returns the current time or a random UUID, the snapshot will say \"everything changed.\" You need to scrub those fields. The scrub function above works for most cases.

Do not use this for exploratory prototypes. Do not use it for frontend components with heavy state. The probe costs will eat your token pool. The signal-to-noise ratio will be terrible.

Also, this is not property-based testing. Property tests assert invariants. Snapshots assert continuity. You want both. The earlier article on agent patches covered property checks. This one covers the missing layer.

The free tier from MonkeyCode makes experimentation cheap. I can spin up a new probe set without thinking about cost. The 10-million-token allowance is enough for a month of edge-case generation. The free server keeps the loop running.

I am going to keep this harness in my stack. Next I want the model to generate probes automatically. I will ask for pathological inputs: empty strings, null bytes, Unicode snowmen. Then the snapshot grows with the codebase.

Try it on your next agent patch. Build one probe. Add two fixtures. Run it before and after. See what your model changed underneath. The diff will tell you a story.

That is the whole trick. No magic. Just a fingerprint of behavior, taken twice.

Quarantine Agent Patches on a Free Ephemeral Server

Finley Sun — Mon, 31 Aug 2026 12:02:35 +0000

Yesterday, my coding agent fixed a crash. The patch passed my laptop. CI failed on a race condition. I spent an hour debugging. Then I realized: my local environment had cached state. The agent's patch was fine in theory. It failed because the test ran in a different universe.

A trending post this week said AI promoted every developer to reviewer. Nobody tested the reviewer. True. But we can at least test the patches those reviewers approve. That requires a clean room. Not a philosophical one. A literal throwaway server.

MonkeyCode is an open-source platform for AI-assisted development. It offers two things I use constantly: free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The server is small, but that is the point. You want the minimal environment where your patch either works or not.

Here is my quarantine workflow. It takes five minutes. It costs nothing but a few tokens and some patience.

First, save the agent's patch as a branch. If you use GitHub, open a pull request. If you use GitLab, same idea. The important thing is that the diff is isolated from your main work.

Second, start a free server on MonkeyCode. You will get a shell. Treat it as disposable. Do not install your dotfiles. Do not configure your editor. Keep it as close to a blank machine as you can.

Third, clone your repository on that blank machine. Use a shallow clone to save time. git clone --depth 1 gives you just the current snapshot. That is all you need for tests. Full history is ancient archaeology now.

Check out the agent's branch. Run a focused test suite. Not the whole suite. Just the files the patch touched.

Property-based tests are my first choice here. They hammer a function with random inputs. They expose off-by-one errors and hidden assumptions. Example: your agent changed a timestamp parser. Write this in test_parser.py:

from hypothesis import given, strategies as st, settings
from mypkg.parser import parse_timestamp

@given(st.integers(min_value=0, max_value=2**31))
@settings(max_examples=200)
def test_round_trip(epoch):
    original = parse_timestamp(epoch)
    assert parse_timestamp(original.timestamp()) == original

Then run it on the free server with a quiet terminal:

git clone --depth 1 https://github.com/you/repo.git /tmp/quarantine
cd /tmp/quarantine
git checkout agent-patch
pip install -e .
pytest -q test_parser.py

If a test fails, read the traceback carefully. Then paste it into an LLM. MonkeyCode's free model access lets you do this without spending your own API credits. The model can propose a hypothesis: "The parser assumes local time, but the server is UTC." That is a valuable second opinion.

Fixtures also matter. Agent patches often leak state. You can pin environment variables inside a fixture:

import pytest
import time

@pytest.fixture
def fixed_tz(monkeypatch):
    monkeypatch.setenv("TZ", "UTC")
    time.tzset()

Attach it to any test that touches dates, times, or sorting. This removes a whole class of heisenbugs.

Now the hard part: flaky tests. A flaky test ruins a quarantine. It gives you false confidence. The patch looks broken when it's not. Or worse, it looks fine when it's actually fragile. My rule: freeze flaky tests. Mark them so they do not run by default:

@pytest.mark.skipif(
    not os.getenv("RUN_FLAKY"),
    reason="Flaky; run manually after triage"
)

Run RUN_FLAKY=1 only when you are ready to chase ghosts. Otherwise ignore them. This keeps the quarantine signal clean.

You can wrap all of this in one script. Name it quarantine.sh. It should ssh into the free server, run the commands, and exit. Something like this:

#!/bin/bash
set -euo pipefail
HOST=$(monkeycode host --new)  # placeholder for your server address
ssh "$HOST" "
  git clone --depth 1 $REPO_URL /tmp/q &&
  cd /tmp/q &&
  git checkout $PATCH_BRANCH &&
  pip install -e . &&
  pytest -q test_$AREA.py
"

The exact command for spawning a server depends on MonkeyCode's current CLI. Check the official repository for syntax. The idea is the same: run the test on a host you do not care about.

The whole loop is short. Create branch, clone, run tests, ask for a second opinion. You can do it for every agent patch. I now do it for every suspicious one. It caught two real bugs in the last week. Both were timezone issues. Both would have shipped if I trusted the local pass.

What are the limits? The free server is not a full CI runner. It has modest CPU and memory. A microservices integration test will not fit. Use it for focused unit and property tests. Also, the token allowance is generous but finite. Reserve model calls for actual failures. Do not pipe every passing test log into the chat.

Who should skip this? Teams with robust CI already. If you have a pipeline that rebuilds the world, you don't need a quarantine. This is for solo developers and small projects. It is for the gap between your laptop and the real environment.

Think of it as a pre-flight check. Pilots do not take off based on a quick glance at the sky. They check instruments in a controlled way. Agent patches deserve the same discipline.

The free server is your instrument. Use it. Your main branch will thank you.

Fuzz the Patch, Not the Promise: A Hypothesis Harness for Agent-Generated Code

Finley Sun — Sun, 30 Aug 2026 11:07:22 +0000

The patch passed 47 tests. It still broke the build. The agent smiled in the PR thread. Nobody knew what to trust.

Unit tests confirm examples. They do not confirm behavior. Property tests generate thousands of inputs. They verify invariants hold. This is the missing layer for agent patches.

This article shows a compact harness. It fuzzes every modified function. It freezes the failing inputs. It runs for free on MonkeyCode's server. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why Agent Patches Need Fuzzing

Agents optimize for plausible code. They copy patterns from training data. They rarely reason about edge cases. A patch can look perfect and still mishandle zero values, null strings, or NaN.

Property tests catch these gaps. You define an invariant. The library generates adversarial inputs. The test fails with a minimal counterexample. That counterexample becomes a regression test.

Good invariants come from the spec. For example, a checkout total must never be negative. A parser must never return an empty node set for valid input. A rate limiter must never allow more than N requests. Write these as pure boolean asserts.

The Harness in Action

Here is the core loop. It tests a function from a fresh agent patch.

from hypothesis import given, settings, strategies as st
from my_agent_patch import compute_total

@given(
    price=st.floats(min_value=0, max_value=1e6, allow_nan=False, allow_infinity=False),
    qty=st.integers(min_value=0, max_value=1000),
)
@settings(max_examples=1000, deadline=None)
def test_total_never_exceeds_expected(price, qty):
    total = compute_total(price, qty)
    assert 0 <= total <= price * qty

This test runs one thousand times. It uses random prices and quantities. If the agent introduced a sign error, the first failure appears fast.

Hypothesis shrinks the failure. It finds the smallest input that breaks the invariant. That input is worth more than a hundred hand-written cases.

Freezing the Flaky

Hypothesis sometimes finds an input. The input only fails intermittently. This usually indicates state leakage or time dependence. You cannot ignore it. You must freeze it.

Keep a pool of known counterexamples. On failure, the harness appends the input to a JSON file. Then it replays that input as a deterministic test.

import json
from hypothesis import given, strategies as st

failing_inputs = []

def record_failure(price, qty):
    failing_inputs.append({"price": price, "qty": qty})
    with open("failing_inputs.json", "w") as fq:
        json.dump(failing_inputs, fq)

@given(
    price=st.floats(min_value=0, max_value=1e6),
    qty=st.integers(min_value=0, max_value=1000),
)
def test_total_with_freeze(price, qty):
    if is_frozen(price, qty):
        return  # already covered by deterministic test
    total = compute_total(price, qty)
    assert 0 <= total <= price * qty

The freeze preserves the failure. The property test keeps hunting. The build stays honest.

Store the seed with each input. Hypothesis can reproduce any failure from a seed. This removes heisenbugs for good.

Running It on Free Infrastructure

Schedule this harness on MonkeyCode's free server. The setup takes one command. The server runs nightly. It costs zero dollars. The agent that produced the patch can also be called via MonkeyCode's free model access. This keeps the whole loop free.

The harness outputs a simple report. It shows pass/fail counts and the frozen inputs. The report becomes a PR comment. Reviewers see exactly what the fuzzer found.

Limitations

Property tests are only as good as your invariants. They cannot detect missing features. They cannot judge code style. They fail when the system has external side effects. Do not use them on UI code or database migrations.

Free server tiers have resource limits. Large suites may time out. Keep the harness scoped to pure functions. Run heavy workloads on paid infrastructure if needed.

Who Should Not Use This

Skip this if your project has no stable interface. Skip it if you cannot articulate any invariant. Skip it if your codebase is legacy spaghetti with global state. Property testing will drown in noise.

Use it when you have pure functions and a clear spec. Use it when agent patches change logic frequently. Use it when you want to trust the patch, not the promise.

Try It Against Your Next Patch

Take your last agent patch. Write two properties for a modified function. Run them on MonkeyCode's free tier. See what fails.

That failure is your next check-in.

A Zero-Token Triage for Agent-Patch Failures

Finley Sun — Sat, 29 Aug 2026 10:08:09 +0000

An agent opens a pull request at 2 AM. The summary says small refactor, all tests pass; nobody ran the tests. The reviewer clicks approve anyway.

This is the new review loop. AI turned every developer into a reviewer. The review tooling never arrived. The fix is not a better prompt. The fix is a cheaper gate.

Most agent-patch failures have three causes. The model changed behavior. The server was unavailable. The test itself was flaky. A pull request cannot tell these apart. A triage pipeline can. It runs before any LLM review. It spends zero tokens.

Run fixtures first

Every accepted patch leaves a trace. Save its inputs and outputs as JSON files under tests/fixtures. A new patch must reproduce those outputs exactly. The check runs in seconds. Most regressions die here. No model call happens.

The fixture corpus is your project memory. Capture new fixtures automatically on merge. Delete or edit them only with a human decision.

Run properties second

Fixtures cover known cases. Properties cover unknown ones. State invariants about the changed functions. A parser should satisfy parse(format(x)) == x. A time parser should reject impossible dates. A deduplicator should preserve order.

Hypothesis generates the edge cases for you. The checks are deterministic and fast. They cost nothing. They catch what fixtures miss.

Freeze flaky tests third

A random failure destroys trust in the pipeline. Give a suspect test three attempts. Three failures in a row mean quarantine. A quarantined test never blocks a review.

Review the quarantine weekly. Fix the test or delete it. Freezing is not forgetting. It moves noise off the critical path.

Review only survivors

The patch passed fixtures, properties, and reruns. Now a model may look. A free model endpoint writes the summary. The summary joins the classified report. No flaky noise reaches the model. The human reads a clean map and makes the final call.

The artifact

The script below implements that order. It classifies failures as passed, failed, or flaky. It prints a JSON report you can attach to the pull request. This is a sketch, not a package. Adapt it to your repository.

#!/usr/bin/env python3
"""Zero-token triage for agent patches. A sketch, not a package."""
import json
import subprocess
import sys
from pathlib import Path

FIXTURE_DIR = Path("tests/fixtures")
RERUNS = 3

def run(cmd, timeout=300):
    try:
        result = subprocess.run(
            cmd, shell=True, capture_output=True, text=True, timeout=timeout
        )
        return result.returncode, (result.stdout + result.stderr)[-400:]
    except subprocess.TimeoutExpired:
        return 124, "timeout"

def triage(patch_dir):
    report = {"passed": [], "failed": [], "flaky": []}

    for fixture in sorted(FIXTURE_DIR.glob("*.json")):
        code, out = run(f"python fixture_runner.py {fixture} {patch_dir}")
        entry = {"fixture": fixture.name, "exit": code, "out": out}
        target = report["passed"] if code == 0 else report["failed"]
        target.append(entry)

    code, out = run("pytest tests/properties -q", timeout=600)
    entry = {"suite": "properties", "exit": code, "out": out}
    target = report["passed"] if code == 0 else report["failed"]
    target.append(entry)

    code = 0
    for attempt in range(RERUNS):
        code, _ = run("pytest tests/flaky_candidates -q", timeout=900)
        if code == 0:
            break
    if code != 0:
        report["flaky"].append({"quarantined": True, "attempts": RERUNS})

    return report

if __name__ == "__main__":
    print(json.dumps(triage(sys.argv[1]), indent=2))

The sketch assumes two pytest directories exist. tests/properties holds the invariant checks. tests/flaky_candidates holds the repeat offenders. Both need a placeholder test. Pytest exits nonzero on an empty suite. The fixture_runner.py script is your responsibility. Load the fixture, apply the patch, compare outputs.

A sample report looks like this:

{
  "passed": [{"fixture": "accept_01.json", "exit": 0}],
  "failed": [{"suite": "properties", "exit": 1}],
  "flaky": [{"quarantined": true, "attempts": 3}]
}

Anything in failed blocks the review. Anything in flaky enters quarantine quietly. Everything in passed moves to the model summary. That split is the point. You learn what to distrust before you pay for a verdict.

Why a free tier fits

This loop was designed for small budgets. Free servers sleep and throttle. Free model access has rate limits. None of that matters. The triage rarely calls a model. The expensive step sees only survivors.

MonkeyCode is an open-source project for agent workflows. It fits at exactly this point. Its free model access can generate the summary. Its free server option can host the triage run. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

As of late August 2026, the free tier includes a ten-million-token grant and a free server option. Free-tier terms change often. Confirm the current limits in the MonkeyCode repository before relying on them.

Limits

This triage cannot catch semantic drift. Fixture replay checks recorded behavior only. Property checks need an invariant worth stating. A quarantine without weekly review hides real defects. Free tiers can throttle at the worst moment. Use paid infrastructure for strict SLAs. Use this triage for deterministic code and a tiny budget.

Do not treat the model summary as approval. Treat it as a map for a human. Run this script before your next review. If the budget fits, try MonkeyCode's free tier as a starting point. Confirm today's numbers in the repository. Run it once. Read the report. Then decide if the patch ships.

A Token-Efficient Gate for Agent Patches

Finley Sun — Fri, 28 Aug 2026 03:39:33 +0000

An agent burned most of a token grant last week. The patch was small and looked clean. The test output was enormous. The agent kept reading unrelated failures. It kept revising the wrong code.

That pattern is becoming the default. Agent patches are generated to look plausible. They satisfy the tests in front of them. They rarely satisfy the invariants behind them. And the verification loop wastes tokens on noise.

The fix is a gate that spends tokens carefully. It selects tests by blast radius. It runs property checks and fixtures. It freezes flaky failures. It prints a short failure summary for the agent to read. Every part of the gate exists to save tokens.

Start with test selection. A parser patch does not need the billing suite. A schema change does not need the parser property tests. Map changed files to test directories. Run only what the patch can break.

import subprocess
import sys
from pathlib import Path

def select_tests(diff_path="HEAD"):
    """Map changed files to test paths. Cheap blast-radius selection."""
    changed = subprocess.run(
        ["git", "diff", "--name-only", diff_path],
        capture_output=True, text=True,
    ).stdout.splitlines()
    selected = set()
    for path in changed:
        if "parser" in path:
            selected.add("tests/property/test_parser.py")
        if "schema" in path:
            selected.add("tests/fixtures/")
        if "api" in path:
            selected.add("tests/integration/")
    return list(selected) or ["tests/unit/"]

def run_selected(selected):
    cmd = ["pytest", *selected, "-q"]
    return subprocess.run(cmd, capture_output=True, text=True)

The mapping is crude. It is also honest. You extend it as your repo teaches you which files break together.

Then freeze the flaky noise. Every repo has flaky tests. They fail for reasons unrelated to the patch. Without a freeze, they block the gate with noise. The freeze is a file that lists known-flaky tests. The gate ignores listed failures. It blocks unlisted ones.

FLAKY_FREEZE = Path("flaky_freeze.txt")

def load_freeze():
    if not FLAKY_FREEZE.exists():
        return set()
    return {line.strip() for line in FLAKY_FREEZE.read_text().splitlines() if line.strip()}

def new_failures(output, frozen):
    failed = [line for line in output.splitlines() if "FAILED" in line]
    return [f for f in failed if f not in frozen]

Then compress the output. The agent reads the gate output. Every token it reads is a token it cannot spend on the next revision. A wall of tracebacks is expensive. A short summary is cheap.

def summarize(failures):
    """One line per failure. Enough for the agent to act."""
    if not failures:
        return "Gate passed."
    lines = ["Gate blocked. New failures:"]
    lines.extend(failures)
    lines.append("Fix these, then rerun the gate.")
    return "\n".join(lines)

The full gate ties it together.

def main():
    frozen = load_freeze()
    selected = select_tests()
    result = run_selected(selected)
    if result.returncode != 0:
        failures = new_failures(result.stdout, frozen)
        print(summarize(failures))
        sys.exit(1)
    print("Gate passed.")

That is the whole loop. The gate selects, runs, freezes, and summarizes. The agent gets a short, actionable failure list. The token cost per iteration stays low.

Where to run it. A gate needs a server. Agent workflows need tokens. Both cost money. This is where the economics change.

MonkeyCode is an open-source AI coding tool. It offers free model access with a 10 million token grant and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The loop becomes simple. The agent proposes a patch using the free token grant. The gate runs on the free server. The gate fails. The agent reads the short summary and revises. The revision also comes from the free grant. The loop costs zero dollars.

A cron job keeps the gate honest.

0 2 * * * cd /srv/agent-patches && ./patch_gate.py >> gate.log 2>&1

Track the budget. A 10 million token grant sounds large. It disappears faster than you expect. A single revision can consume thousands of tokens. The test run itself consumes none. The agent reading the output does. Track the budget like you track disk space.

def estimate_tokens(text):
    # Rough heuristic: one token per four characters.
    return len(text) // 4

The heuristic is not exact. Tokenizers are not that simple. But the number gives you a trend. Trends matter more than precision when a grant is finite.

Who should not use this. The gate checks behavior, not design. A patch can pass every check and still be the wrong abstraction. It is not a substitute for review.

The gate is also not for regulated environments. If you need audit trails and signed approvals, a cron job on a free server will not satisfy compliance. Use the pattern, not the infrastructure.

And if your tests are already slow, the gate inherits that slowness. Fix the slow tests first. A gate that nobody runs is a decoration.

The pattern. Generate patches with free tokens. Verify them with a free server. Freeze the flaky noise. Compress the feedback. The agent does the writing. The gate does the judging. The reviewer makes the design call.

That division is the real win. Agents write more code than any human can review. Gates verify more behavior than any human can test. Humans make the few decisions that matter.

Try the loop once with a small patch. The free grant is there for exactly that. The difference is visible between a patch that passes and a patch that survives.

A Testing Strategy for Agent Patches: Property Checks, Fixtures, and a Flaky Freeze

Finley Sun — Thu, 27 Aug 2026 03:04:41 +0000

An agent opened a pull request at 2:47 AM. All checks passed. The merge took eleven seconds.

Three days later, an edge case broke in production. The unit tests stayed green the whole time.

AI agents turned every developer into a reviewer. Nobody built a test strategy for the patches themselves. That gap now costs more than the code the agents save.

Most agent patches fail the same way. They pass the examples in the ticket. They break the invariants nobody wrote down. A reviewer staring at a diff cannot catch what the tests never check.

Unit tests assert one input and one output. Agent patches need a stronger contract. They need properties that hold across the whole input space.

This article describes a three-layer strategy: property checks, frozen fixtures, and a flaky-test quarantine. It is a concrete gate, not a philosophy.

Layer one: property checks

Property-based testing flips the question. Instead of "does this example pass?", ask "does this invariant hold for a thousand generated cases?". The generator explores the edges a human reviewer skips.

Here is a minimal property test for a discount function an agent just wrote:

from decimal import Decimal
from hypothesis import given, strategies as st

def apply_discount(price: Decimal, rate: Decimal) -> Decimal:
    # agent-generated code under test
    return price - (price * rate)

@given(
    st.decimals(min_value=0, max_value=100_000),
    st.decimals(min_value=0, max_value=1),
)
def test_discount_never_exceeds_price(price, rate):
    result = apply_discount(price, rate)
    assert result >= 0
    assert result <= price

Run this against an agent patch and the obvious bugs die fast. Negative prices, rounding drift, and overflow die first. The generator finds them before a reviewer finishes the first file.

The trick is writing the right properties. Good candidates come from the domain: totals stay non-negative, balances stay consistent, IDs stay unique, timestamps stay monotonic. Bad candidates are the ticket examples rewritten as assertions. Those are already covered.

Layer two: frozen fixtures

Property checks cover the input space. Fixtures cover the real world. Agent patches often break on data shapes the ticket never mentioned.

A fixture freeze is a contract. You snapshot a realistic dataset and hash it. Every agent patch runs against the same frozen world. When a test changes behavior, the patch caused it, not the data.

# scripts/freeze_fixtures.py
import hashlib
from pathlib import Path

FIXTURE_DIR = Path("tests/fixtures/data")

def freeze() -> None:
    for path in sorted(FIXTURE_DIR.rglob("*")):
        if path.is_file():
            digest = hashlib.sha256(path.read_bytes()).hexdigest()
            print(f"{path.relative_to(FIXTURE_DIR)}: {digest}")

if __name__ == "__main__":
    freeze()

Commit the output. Compare it in CI. If a fixture changes, the diff must be explicit and reviewed. Otherwise an agent can "fix" a test by editing the data instead of the code. That is the most common silent cheat in agent-generated patches.

Layer three: flaky freeze

The third layer is discipline. Agent patches multiply the number of test runs. A flaky test that passed once now fails randomly at 3 AM. The worst response is rerunning until green.

A flaky freeze is simple: quarantine, don't retry. When a test fails without a code change, move it to a quarantine suite. It stops blocking the pipeline. It also stops hiding real regressions.

# scripts/detect_flaky.py
"""Run a test file three times. Report any test that flips."""
import subprocess
import sys

TEST_FILE = sys.argv[1]
seen: dict[str, list[bool]] = {}

for run in range(3):
    result = subprocess.run(
        ["pytest", TEST_FILE, "--tb=no", "-q"],
        capture_output=True, text=True,
    )
    failed = {
        line.split("::")[-1]
        for line in result.stdout.splitlines()
        if "FAILED" in line
    }
    for test in set(seen) | failed:
        seen.setdefault(test, []).append(test in failed)

for test, results in seen.items():
    if len(set(results)) > 1:
        print(f"FLAKY: {test} -> {results}")

Run this on the agent's branch before review. A test that fails once in three runs is noise, not a signal. Quarantine it. Make the agent fix the flake separately, or reject the patch.

The gate, in order

The full gate runs in three steps. Property checks first, because they are fast and brutal. Fixture hashes second, because they catch silent data edits. The flaky detector last, because it protects the first two from noise.

Only then does a human reviewer read the diff. The reviewer's job shrinks to judgment. Does the approach fit the codebase? Do the properties match the product's real invariants?

Where free infrastructure fits

Running this gate continuously needs compute and model calls. MonkeyCode is an open-source project that pairs free model access with a free server option. The server runs the gate on a schedule. The model access drafts candidate properties and summarizes failure clusters for the reviewer.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The gate does not require a specific vendor. Any CI runner and any test framework will do. The free tier just lowers the cost of starting.

Limitations

This strategy has real limits. Property checks only test the properties you write. If the invariant is wrong, the generator happily validates the wrong behavior. Fixtures rot when real data drifts. The freeze protects stability, not relevance.

Who should not use this? Small throwaway scripts. One-off migrations. Demos that die next week. The setup cost is real. If the patch touches code with no invariants worth protecting, the gate is overhead, not safety.

A green gate is not a proof of correctness. It is a filter. It raises the cost of a bad merge. It does not make bad merges impossible. The human reviewer still owns the final call.

The next time an agent opens a PR at 2:47 AM, let the properties argue with it first. If you want to see this gate running on free infrastructure, MonkeyCode's open-source project and free server option are a reasonable place to start.

A Nightly Model-Regression Harness on Free Models and a Free Server

Finley Sun — Wed, 26 Aug 2026 11:13:29 +0000

Last month, a prompt that returned clean JSON for six months started returning markdown. No code changed. No prompt changed. The model changed.

Model drift is the silent production bug. You cannot pin a model version forever, and you cannot trust any provider's "we will let you know." What you can do is run a nightly regression harness that compares today's model output against yesterday's baseline. The catch is cost. A real regression suite burns tokens fast.

Free models change that math. MonkeyCode, an open-source AI coding assistant, offers free model access for experimentation and a free server option for running scheduled jobs. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This article builds a nightly model-regression harness with three parts: golden prompts, a comparison script, and a scheduler. You will get a decision table for choosing a comparison method, a working Python script, and a list of teams that should not use this approach.

The problem: models change under you

Providers update models without a changelog. They deprecate versions. They tweak system prompts. Your prompt was tuned against a model that no longer exists.

The failure mode is nasty. Your unit tests pass because they mock the model. Your integration tests pass because they use the same model. Production fails because production uses the real model, which changed last Tuesday.

A nightly regression harness catches the drift within 24 hours. It runs a fixed set of prompts, records the outputs, and compares them against a stored baseline. When the diff exceeds a threshold, you get a report before your users get an error.

Part one: golden prompts

Golden prompts are the inputs that matter most. Not your whole prompt library. The twenty prompts that drive revenue-critical paths.

Store them as a JSON file. Each entry has an ID, a prompt, and a baseline output.

[
  {
    "id": "extract-invoice-total",
    "prompt": "Extract the total amount from this invoice. Return JSON with keys: currency, amount.",
    "baseline": "{\"currency\": \"USD\", \"amount\": 1234.56}"
  },
  {
    "id": "classify-ticket",
    "prompt": "Classify this support ticket as billing, technical, or other. Reply with one word.",
    "baseline": "billing"
  }
]

The baseline is captured once, manually, and reviewed. After that, it is frozen. A baseline change is a code change. It needs a PR, not a silent update.

Part two: the comparison script

The script reads the golden prompts, calls the model, and compares each output against the baseline. It writes a report and exits non-zero when the failure rate is too high.

import json
import sys
from difflib import SequenceMatcher

def load_golden(path):
    with open(path) as f:
        return json.load(f)

def call_model(prompt):
    # Replace with your provider's client.
    # MonkeyCode's free model access works here.
    return "{\"currency\": \"USD\", \"amount\": 1234.56}"

def similarity(a, b):
    return SequenceMatcher(None, a, b).ratio()

def main(golden_path, threshold=0.9):
    golden = load_golden(golden_path)
    failures = []

    for case in golden:
        actual = call_model(case["prompt"])
        score = similarity(actual, case["baseline"])
        status = "PASS" if score >= threshold else "FAIL"
        if status == "FAIL":
            failures.append({"id": case["id"], "score": score, "actual": actual})
        print(f"{status} {case['id']}: {score:.2f}")

    if failures:
        with open("regression-report.json", "w") as f:
            json.dump(failures, f, indent=2)
        print(f"{len(failures)} regression(s) detected.")
        sys.exit(1)

if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "golden.json")

The script is deliberately simple. No framework, no dependencies beyond the standard library. You can run it anywhere, including the free server option that MonkeyCode provides for scheduled jobs.

Part three: the nightly schedule

A regression harness only works if it runs. Schedule it nightly. A cron job on a free server is enough.

0 2 * * * cd /opt/model-regression && python compare.py golden.json >> nightly.log 2>&1

The free server option matters here. You do not want to pay for a VM just to run a twenty-minute job. A free tier that can run cron jobs is exactly the right size for this workload.

The output lands in a file. The next morning, you check the log. If the script exited non-zero, the report tells you which prompts drifted and how far.

The decision table: how to compare outputs

Similarity is not always the right tool. Choose based on your output type.

Output type	Comparison method	Threshold	Why
Structured JSON	Parse and compare fields	Exact match per field	Order and whitespace do not matter
Single label	Exact string match	100%	One word, no ambiguity
Free-form text	SequenceMatcher or embedding distance	0.85	Semantics matter more than wording
Code	Compile + run tests	Pass/fail	Behavior is the contract
Mixed	Parse first, then similarity on text fields	Per-field	Each field has its own contract

The threshold is a starting point, not a law. Run the harness for a week, look at the scores, and set thresholds that separate real drift from normal variation.

Limitations

This harness does not catch semantic drift. A model can rephrase a response in a way that passes a similarity check but changes the meaning. If your use case is sensitive to meaning, add an LLM-as-judge step. That costs tokens, which is another reason the free model access matters.

The baseline goes stale. Prompts change. Products change. Review the golden set monthly. Remove dead prompts. Add new ones. A golden set that does not evolve becomes a museum.

The harness tests prompts, not pipelines. It will not catch a broken retriever, a changed embedding model, or a bug in your orchestration code. Those need their own tests.

Who should not use this

Teams with no stable prompts. If your prompts change every week, the baseline churns and the report is noise.

Teams that already have a vendor-provided evaluation suite. Use theirs. Do not build a parallel system.

Teams with zero tolerance for false alarms. A nightly regression report that nobody reads is worse than no report. It trains the team to ignore the signal.

Closing

Model drift is real. It is silent. It breaks production without a commit.

A nightly regression harness turns that silence into a report. Golden prompts, a comparison script, and a cron job. That is the whole system. Free models make the token cost manageable. A free server makes the schedule cost zero.

Start with ten prompts. Run it for a week. Read the first report. Then decide if you need more.

The harness is the early warning system. The model is the weather. You cannot control the weather. You can control whether you are surprised by it.

A Response Cache for Token-Limited LLM APIs (Before You Burn the Free Grant)

Finley Sun — Sat, 22 Aug 2026 14:41:05 +0000

Monday at 9:30 AM, the CI queue runs three deep. Your review bot processes the same pull request for the fourth time. The prompt carries yesterday's context, and the model returns the same verdict. The token counter ticks up anyway.

Free token grants look generous at first glance. Ten million tokens sounds infinite until your team shares one endpoint. The grant shrinks faster than anyone expects, and repetition is the silent killer.

MonkeyCode offers free model access and a free hosted server. The allocation is real, but the spending discipline is on you.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Before you build on a free tier, build a cache. The cheapest LLM call is the one you never make.

The repetition problem

LLM calls are not all unique in practice. CI reviews repeat the same diff many times, and test generators rerun identical scenarios. Support bots answer the same question twice per hour, and each duplicate burns tokens without adding any value.

A cache turns duplicates into free hits instantly. The first call pays the full price, and the next hundred calls pay nothing.

Three layers of caching

Exact matching is the cheapest layer to implement. Hash the normalized prompt and compare hashes, and zero model calls are needed for a match.

Normalized matching handles formatting noise without extra cost. Extra spaces and reordered keys disappear, and same meaning produces the same hash.

Semantic matching catches paraphrases, but it needs an embedding call. It costs tokens, so use it only when the first two layers miss.

"""A layered response cache for OpenAI-compatible chat endpoints."""
import hashlib
import json
import os
import re
import time
from collections import OrderedDict

import requests

BASE_URL = os.environ["OPENAI_BASE_URL"].rstrip("/")
API_KEY = os.environ["OPENAI_API_KEY"]
MODEL = os.environ["MODEL_NAME"]
CACHE_SIZE = 256
TTL_SECONDS = 3600


def normalize_prompt(messages):
    """Normalize whitespace and key order for stable hashing."""
    def clean(text):
        return re.sub(r"\s+", " ", text.strip())
    return json.dumps(
        [{"role": m["role"], "content": clean(m["content"])} for m in messages],
        sort_keys=True,
    )


class ResponseCache:
    def __init__(self, capacity=CACHE_SIZE, ttl=TTL_SECONDS):
        self.capacity = capacity
        self.ttl = ttl
        self._store = OrderedDict()

    def _key(self, messages):
        raw = normalize_prompt(messages)
        return hashlib.sha256(raw.encode()).hexdigest()

    def get(self, messages):
        key = self._key(messages)
        if key not in self._store:
            return None
        entry = self._store[key]
        if time.time() - entry["ts"] > self.ttl:
            del self._store[key]
            return None
        self._store.move_to_end(key)
        return entry["response"]

    def put(self, messages, response):
        key = self._key(messages)
        self._store[key] = {"ts": time.time(), "response": response}
        self._store.move_to_end(key)
        while len(self._store) > self.capacity:
            self._store.popitem(last=False)


cache = ResponseCache()


def chat_completion(messages):
    cached = cache.get(messages)
    if cached:
        return cached
    resp = requests.post(
        f"{BASE_URL}/chat/completions",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"model": MODEL, "messages": messages, "stream": False},
        timeout=60,
    )
    resp.raise_for_status()
    body = resp.json()
    cache.put(messages, body)
    return body

The cache key is a SHA-256 hash of the normalized prompt. Normalization matters more than the hash function itself. Whitespace, key order, and role labels all affect the hash, so the normalizer strips and sorts them away.

Temperature and max_tokens also change responses. Include them in the cache key if you vary them. A cached response for temperature 0.7 is wrong for temperature 0.2.

Measuring hit rate

A cache without a hit-rate metric is a guess. Add a counter subclass to track hits and misses.

class CountingCache(ResponseCache):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.hits = 0
        self.misses = 0

    def get(self, messages):
        result = super().get(messages)
        if result is not None:
            self.hits += 1
        else:
            self.misses += 1
        return result

    @property
    def hit_rate(self):
        total = self.hits + self.misses
        return self.hits / total if total else 0.0

Run it against a real workload and record the hit rate. Then decide if semantic matching is worth the embedding cost.

A useful test workload has three distinct parts. The mix includes exact duplicates, near duplicates, and unique prompts. A realistic mix mirrors how your team actually calls the API.

Workload mix	Expected hit rate	Action
60% duplicates	above 50%	Keep exact cache
30% near-duplicates	10-30%	Add normalization
10% unique	below 10%	Accept the misses

These numbers are starting points, so measure your own traffic.

Concurrency and staleness

The cache is not thread-safe by default. Two threads can miss simultaneously and issue duplicate calls. A double-checked lock prevents that waste.

import threading

lock = threading.Lock()

def chat_completion_threadsafe(messages):
    cached = cache.get(messages)
    if cached:
        return cached
    with lock:
        cached = cache.get(messages)
        if cached:
            return cached
        resp = requests.post(
            f"{BASE_URL}/chat/completions",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={"model": MODEL, "messages": messages, "stream": False},
            timeout=60,
        )
        resp.raise_for_status()
        body = resp.json()
        cache.put(messages, body)
        return body

The lock is held during the check-and-set window. The HTTP request happens inside the lock, which serializes duplicate calls. That is acceptable because the duplicates are the waste you are eliminating.

A cached response is also a snapshot of the past. Models change and contexts drift, so cached answers go stale. Set a TTL that matches your data freshness. One hour works for code review, and one minute works for live data. Anything time-critical should use a zero TTL.

The cache hides quality regressions too. If the model improves, your cache serves the old answer. Flush the cache after every model update.

Who should skip this

Personal projects with low traffic do not need a cache. The complexity is not worth the token savings.

Applications with unique prompts get no benefit at all. If every call is different, the cache is dead weight.

Security-sensitive workloads should never cache model responses. Prompt injection can poison a cache, and a poisoned response spreads fast.

The habit

Build the cache before you optimize anything else. Token grants are finite by design, and repetition is the easiest waste to eliminate.

On a hosted free server, latency is often higher than a paid endpoint. A cache reduces round trips, which makes the tool feel faster without server-side changes.

MonkeyCode's free tier is a reasonable place to practice this discipline. The cache works with any OpenAI-compatible endpoint, and the habit outlasts the grant.

A Disposable Evaluation Loop for Any Model Release Before You Trust It in CI

Finley Sun — Fri, 14 Aug 2026 11:27:49 +0000

Why this is worth reading: release-day model posts usually tell you what changed, not whether the model helps your specific Python tooling. This guide gives you a small, repeatable evaluation loop you can run on a free server, so the next trending model name becomes a candidate instead of an assumption.

You keep seeing the same pattern: a new model is announced, the screenshots look good, and within hours it appears in someone else's CI config. The problem is not the model; it is that the evaluation happened on a benchmark you did not define. You need a fixed task, an isolated runner, and a scoring function that fails a bad switch before it reaches your main branch.

One way to keep that loop cheap is to run it against MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat the free tier as disposable infrastructure, not as an availability or latency guarantee. The runner below works with any OpenAI-compatible endpoint, so the method remains useful if you swap in another provider.

Start with a fixed task, not a leaderboard

A leaderboard is useful for narrowing a field, but it does not tell you how a model handles the Python tracebacks, shell commands, and data-shape bugs you actually see. Before you touch a provider, write down one or two tasks that have caused real failures in your review process. If a candidate cannot beat a simple check, you should not spend time wiring it into CI.

The example tasks here are intentionally small. They test whether the model can explain a type error, produce a safe file-search command, and use a defensive dictionary lookup. Keep the tasks small enough that you can read the output, because the first sign of a bad model is often an answer that looks fluent but misses the required pattern.

Keep the runner disposable

Run the evaluation from a fresh virtual environment and treat the server as ephemeral. Do not install the candidate into your main development machine until it passes the gate. The free server option is useful here because you can point the same script at a separate endpoint and avoid contaminating your local state.

A candidate model name belongs in an environment variable, not in a hard-coded config. You may see a model named Grok 4.6 or DeepSeek-V4-Pro-0813 in a release thread. The name is just a string until the loop returns a clean score for your tasks.

The evaluation loop

Create eval_candidate.py with the following content.

# eval_candidate.py
import json
import os
import time
from dataclasses import dataclass

from openai import OpenAI

@dataclass
class Task:
    name: str
    prompt: str
    required: list[str]
    forbidden: list[str]

TASKS = [
    Task(
        name='traceback',
        prompt='Explain this error in plain language: TypeError: can only concatenate str (not int) to str',
        required=['str', 'int', 'convert'],
        forbidden=['restart'],
    ),
    Task(
        name='file-search',
        prompt='Give a POSIX shell command to list the five largest files under /var/log.',
        required=['du', 'sort', 'head'],
        forbidden=['rm'],
    ),
    Task(
        name='json-field',
        prompt='Write a Python expression to extract the value of the id field from a dict named record, without raising if the key is missing.',
        required=['record.get'],
        forbidden=['record['],
    ),
]

def call_model(client, model, prompt):
    start = time.time()
    response = client.chat.completions.create(
        model=model,
        messages=[{'role': 'user', 'content': prompt}],
        temperature=0,
        max_tokens=256,
        timeout=10,
    )
    elapsed = time.time() - start
    return response.choices[0].message.content or '', elapsed

def score(content, task):
    text = content.lower()
    points = 0
    for needle in task.required:
        if needle.lower() in text:
            points += 1
    for needle in task.forbidden:
        if needle.lower() in text:
            points -= 2
    return points

def main():
    model = os.environ.get('MODEL', '').strip()
    base_url = os.environ.get('EVAL_BASE_URL', '').strip()
    api_key = os.environ.get('EVAL_API_KEY', '').strip()
    if not model or not base_url:
        raise SystemExit('Set MODEL and EVAL_BASE_URL before running.')

    client = OpenAI(base_url=base_url, api_key=api_key)
    total = 0
    passed = 0

    for task in TASKS:
        content, elapsed = call_model(client, model, task.prompt)
        points = score(content, task)
        total += points
        if points >= len(task.required):
            passed += 1
        print(json.dumps({
            'task': task.name,
            'required_found': [r for r in task.required if r.lower() in content.lower()],
            'forbidden_found': [f for f in task.forbidden if f.lower() in content.lower()],
            'points': points,
            'elapsed_seconds': round(elapsed, 2),
        }))

    print(f'Passed {passed}/{len(TASKS)} tasks; total score {total}')

if __name__ == '__main__':
    main()

Then run it with three environment variables.

python -m venv .venv
source .venv/bin/activate
pip install openai
export MODEL='your-candidate-model'
export EVAL_BASE_URL='https://your-free-server.example'
export EVAL_API_KEY='your-key'
python eval_candidate.py

The base URL and key come from the provider's console. For the MonkeyCode free server option, collect the endpoint and key from the same place you would normally get the model access. Keep those values in your shell or a local env file, never in the repository.

Read the result, not the model name

The output is deliberately boring: one JSON line per task and a final pass count. That makes it easy to compare two candidates and easy to notice when a model changes behavior from one run to the next.

Result	Decision
Passed 3/3 and no forbidden matches	Promote to a small shadow run
Passed 2/3	Keep as a manual-review candidate
Any forbidden match or timeout	Reject for this task
Different scores on repeated runs	Reject or pin `temperature=0` and rerun

Run each candidate three times before you trust a pass. A single clean answer may be luck, especially with a small prompt and a short output limit.

Limitations and who should skip this

This is a smoke test, not a benchmark. The pattern checks catch obvious misses and risky commands, but they do not measure semantic quality, cost, truthfulness, or long-context behavior. A model can score 3/3 here and still produce poor code in a real application.

Skip this approach if you need a general model comparison, production latency data, or a safety evaluation. You also need to be comfortable passing an API key to a temporary endpoint and reading raw responses. If your task requires private code or customer data, use a local or approved environment instead.

Start with three tasks that reproduce actual failures you have seen in your own review or bug reports. That keeps the gate useful without turning into another benchmark that you will stop trusting in a week.

A Reusable Smoke-Test Harness for Newly Released Open Models (Before You Bet a Project on One)

Finley Sun — Mon, 10 Aug 2026 10:34:50 +0000

Every few weeks a new open-weights model drops — the recent wave of releases from teams like MiniMax is a good example — and my timeline fills with benchmark screenshots. The problem: those benchmarks rarely answer the question I actually have, which is "does this model survive the boring tasks in my repo?"

So instead of reading another leaderboard, I built a small, reusable smoke-test harness. It runs the same fixed set of prompts against any OpenAI-compatible endpoint, scores the outputs with cheap deterministic checks, and writes a report I can diff across models. When a new model is announced, I can have a first honest signal in under half an hour — often using free tiers and free hosted compute, so the experiment costs nothing but time.

This post is the harness, the reasoning behind it, and its limits.

Why not just trust the published benchmarks?

Published benchmarks measure aggregate capability. My risks are narrower and weirder:

Does the model follow my project's formatting conventions (e.g., type hints, no Any)?
Does it hallucinate imports that don't exist?
Does it stay coherent on the second turn, when I ask it to modify its own output?
Is latency acceptable for interactive use, not just batch jobs?

A 20-prompt harness tuned to my actual work answers these better than any public score. And because it's just HTTP calls, it works against any provider that exposes an OpenAI-compatible API — which is now most of them, including the open-weight releases everyone is currently talking about.

Where to run it for free

Two things usually block this kind of experiment: API cost and a place to run the harness. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode happens to remove both blockers for this workflow — it offers free access to a selection of models and a free server option, so you can point the harness at a model you're curious about and run the whole evaluation on hosted compute without a local GPU or a credit card. That's the entire role it plays here; the harness itself works against any compatible endpoint, and you should treat free availability as something that can change, not a guarantee.

There's also a point worth making about the open source spirit here. The reason a harness like this is even possible is that providers converged on open, documented API shapes, and that teams releasing open weights — MiniMax and others — let anyone inspect, host, and poke at their models instead of hiding them behind a black box. MonkeyCode's approach fits the same pattern: free access lowers the barrier to verifying things yourself rather than trusting marketing. Open ecosystems compound because they let outsiders test claims. This harness is my small contribution to that loop — take it, run it against whatever model just launched, and publish your own results.

The harness

The design constraints:

Fixed prompt set, versioned. Prompts live in a JSON file so results are comparable across models and across time.
Deterministic scoring where possible. Compile checks, regex checks, import existence — things that don't require a judge model.
Raw outputs saved. So a human (me) can re-review borderline cases later.

prompts.json — a minimal starter set:

[
  {
    "id": "py_parse_logs",
    "kind": "python_function",
    "prompt": "Write a Python function `parse_durations(lines: list[str]) -> dict[str, float]` that reads log lines like 'task=backup duration=1.5s' and returns total seconds per task. Include type hints. No third-party imports.",
    "entrypoint": "parse_durations",
    "test": "result = parse_durations(['task=a duration=1.5s', 'task=a duration=0.5s', 'task=b duration=2s']); assert result == {'a': 2.0, 'b': 2.0}, result"
  },
  {
    "id": "py_fix_own_code",
    "kind": "python_followup",
    "prompt": "Now change the function so invalid lines are skipped instead of raising. Return only the updated function.",
    "entrypoint": "parse_durations",
    "test": "result = parse_durations(['garbage line', 'task=a duration=1s']); assert result == {'a': 1.0}, result"
  }
]

smoke.py:

"""Smoke-test an OpenAI-compatible model endpoint against a fixed prompt set.

Usage:
    export SMOKE_BASE_URL="https://your-endpoint/v1"
    export SMOKE_API_KEY="..."
    export SMOKE_MODEL="model-name-as-served"
    python smoke.py prompts.json report_dir/
"""
import json
import re
import sys
import time
import urllib.request
from pathlib import Path


def chat(messages: list[dict], base_url: str, api_key: str, model: str) -> tuple[str, float]:
    body = json.dumps({
        "model": model,
        "messages": messages,
        "temperature": 0.0,
    }).encode()
    req = urllib.request.Request(
        f"{base_url.rstrip('/')}/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {api_key}",
        },
    )
    start = time.monotonic()
    with urllib.request.urlopen(req, timeout=120) as resp:
        payload = json.loads(resp.read())
    elapsed = time.monotonic() - start
    return payload["choices"][0]["message"]["content"], elapsed


def extract_code(text: str) -> str:
    """Pull the first fenced code block, or fall back to the raw text."""
    m = re.search(r"```

(?:python)?\n(.*?)

```", text, re.DOTALL)
    return m.group(1) if m else text


def run_python_check(code: str, entrypoint: str, test: str) -> tuple[bool, str]:
    """Execute generated code plus its assertion in a throwaway namespace.

    NOTE: this executes model output. Run in a disposable environment
    (container, free hosted server, VM) — never on a machine with secrets.
    """
    namespace: dict = {}
    try:
        exec(compile(code, "<generated>", "exec"), namespace)
        assert callable(namespace.get(entrypoint)), f"{entrypoint} not defined"
        exec(compile(test, "<test>", "exec"), namespace)
        return True, ""
    except Exception as exc:  # noqa: BLE001 - we want the failure text in the report
        return False, f"{type(exc).__name__}: {exc}"


def main() -> None:
    import os

    prompts_path, report_dir = Path(sys.argv[1]), Path(sys.argv[2])
    report_dir.mkdir(parents=True, exist_ok=True)
    base_url = os.environ["SMOKE_BASE_URL"]
    api_key = os.environ["SMOKE_API_KEY"]
    model = os.environ["SMOKE_MODEL"]

    prompts = json.loads(prompts_path.read_text())
    results = []
    history: list[dict] = []

    for item in prompts:
        if item["kind"] != "python_followup":
            history = []  # fresh conversation per task chain
        history.append({"role": "user", "content": item["prompt"]})
        try:
            reply, latency = chat(history, base_url, api_key, model)
        except Exception as exc:  # noqa: BLE001
            results.append({"id": item["id"], "passed": False, "error": str(exc)})
            continue
        history.append({"role": "assistant", "content": reply})

        code = extract_code(reply)
        passed, detail = run_python_check(code, item["entrypoint"], item["test"])
        (report_dir / f"{item['id']}.py").write_text(code)
        results.append({
            "id": item["id"],
            "passed": passed,
            "latency_s": round(latency, 2),
            "detail": detail,
        })

    passed_count = sum(r["passed"] for r in results)
    report = {
        "model": model,
        "passed": f"{passed_count}/{len(results)}",
        "results": results,
    }
    (report_dir / "report.json").write_text(json.dumps(report, indent=2))
    print(json.dumps(report, indent=2))


if __name__ == "__main__":
    main()

The whole thing is stdlib-only on purpose: no dependencies means it runs anywhere, including a freshly provisioned free server, with zero setup beyond three environment variables.

How I read the results

The pass count is the least interesting number. What I actually look at:

Failure mode. A syntax error in extracted code suggests formatting problems; a wrong assertion result suggests reasoning problems; a missing entrypoint suggests instruction-following problems. Different failures disqualify a model for different jobs.
Follow-up behavior. The py_fix_own_code prompt is where many models quietly degrade — they rewrite the whole thing and break the original behavior, or ignore "return only the updated function."
Latency distribution. One slow response is noise; consistently slow responses rule out interactive use.
The saved .py files. I skim every one. Deterministic checks can't catch code that passes the test but is ugly, fragile, or full of phantom imports elsewhere.

Because each run writes into its own report_dir/, comparing two models — or the same model two months apart — is just diff -ru.

Limitations, and who shouldn't do this

Twenty prompts is not an evaluation. It's a smoke test. It can disqualify a model for your workflow; it cannot rank models in general. If you need rigorous comparison, look at proper eval frameworks and far larger task sets.
Executing generated code is dangerous. The harness runs whatever the model emits. Only run it in an environment with no secrets, no network access to internal systems, and no persistent disk you care about. A disposable free server instance is actually a reasonable fit here for exactly this reason.
temperature=0.0 is not fully deterministic on most serving stacks. Rerun before drawing strong conclusions from a one-run difference.
Free tiers change. Don't build CI that hard-depends on any provider's free model access or free server remaining available; treat them as experiment infrastructure, not production infrastructure. Check the provider's current terms before relying on either.
If your tasks need judgment-based scoring (design reviews, prose quality), deterministic checks won't get you far — you'd need a judge model or human review, which is a bigger project.

Try it on the next launch

The next time an open model drops and the hot takes start flying, clone this harness, write five prompts from your actual current project, and run it before forming an opinion. Half the value is the harness; the other half is the habit of verifying claims yourself — which is, at its core, what the open source ecosystem is supposed to make easy. If you want a zero-cost sandbox for that first run, MonkeyCode's free model access and free server option are one way to get both the endpoint and the disposable environment in the same place — but the harness doesn't care where you point it.

A Local AI Review Pass for Python Diffs Before You Open the PR

Finley Sun — Mon, 10 Aug 2026 08:08:48 +0000

My last post here was about a pre-push gate that fails loud instead of often. Since then I've been running an experiment on top of it: adding an AI review pass to the same gate, so a second set of (artificial) eyes looks at my diff before any human does.

The constraint I set for myself: it had to cost nothing to run, it had to work on plain git diff output, and it had to be a gate — meaning it produces a decision, not a wall of commentary I ignore.

Here's the workflow and the script. Everything below runs locally against a diff; the only network call is the model inference.

The problem with AI review as usually practiced

Most AI-assisted review I've seen fails in one of two ways:

It's a chat, not a gate. You paste code into a chat window, get prose back, and nothing forces a decision. The signal-to-noise ratio decays fast.
It reviews files, not diffs. Reviewing a whole file with a model means it flags pre-existing issues you didn't touch, and you learn to ignore it.

The fix for both: review only the diff, require structured output, and make the script exit non-zero above a severity threshold. Same philosophy as a linter with --max-warnings.

Why free models are good enough for this

A diff-review gate is a high-volume, low-stakes-per-call workload. You run it on every push, most pushes are fine, and a false positive costs you thirty seconds of reading. That's exactly the profile where a free model tier makes sense: you'd never justify a paid API call for "check whether my 40-line diff does something dumb," but you'll happily run it if it costs nothing.

I'm using MonkeyCode for this, which offers free model access and a free server option, so the whole gate runs without a billing account or a GPU on my machine. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script below is deliberately written against a generic OpenAI-compatible chat endpoint, though — if you have another provider with an OpenAI-style API, change the base URL and it works the same.

One thing I want to be honest about up front: I haven't benchmarked the free tier's latency or rate limits under heavy use, and I don't know what its long-term availability terms are. For a pre-push hook that makes one call per push, none of that has mattered to me so far. If you're gating CI for a team of fifty, you'd want to verify those limits first.

The gate script

Save as ai_review_gate.py, run it from your pre-push hook or manually with python ai_review_gate.py origin/main.

#!/usr/bin/env python3
"""AI review gate for Python diffs.

Reviews only the current diff (not whole files), requires structured
JSON findings, and exits 1 if any finding meets the severity threshold.
"""
import json
import os
import subprocess
import sys

import urllib.request

BASE_URL = os.environ.get("AI_REVIEW_BASE_URL", "https://YOUR-ENDPOINT/v1")
API_KEY = os.environ.get("AI_REVIEW_API_KEY", "")
MODEL = os.environ.get("AI_REVIEW_MODEL", "default")  # whatever free model your endpoint exposes
BLOCK_AT = {"high", "critical"}  # severities that fail the gate
MAX_DIFF_CHARS = 24_000

PROMPT = """You are reviewing a git diff of a Python project.
Report ONLY issues introduced or worsened by this diff. Do not comment
on pre-existing code the diff merely touches.

For each issue, return a JSON array element:
  {"file": str, "line": int|null, "severity": "low"|"medium"|"high"|"critical",
   "kind": "bug"|"security"|"correctness"|"regression-risk",
   "summary": str (one sentence), "suggestion": str (one sentence)}

Return [] if the diff looks fine. Return ONLY the JSON array, no prose.

DIFF:
"""

def get_diff(base: str) -> str:
    diff = subprocess.run(
        ["git", "diff", f"{base}...HEAD", "--", "*.py"],
        capture_output=True, text=True, check=True,
    ).stdout
    if len(diff) > MAX_DIFF_CHARS:
        # Truncate rather than skip silently — a truncated review is
        # still a review; a silent pass is not.
        diff = diff[:MAX_DIFF_CHARS] + "\n# [diff truncated for review]\n"
    return diff

def call_model(diff: str) -> list[dict]:
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": PROMPT + diff}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        f"{BASE_URL}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {API_KEY}"},
    )
    with urllib.request.urlopen(req, timeout=120) as resp:
        payload = json.load(resp)
    text = payload["choices"][0]["message"]["content"].strip()
    # Models sometimes wrap JSON in fences despite instructions.
    if text.startswith("```

"):
        text = text.split("\n", 1)[1].rsplit("

```", 1)[0]
    try:
        findings = json.loads(text)
    except json.JSONDecodeError:
        # Unparseable output = inconclusive, not a pass.
        print("AI review returned unparseable output; treating as inconclusive.")
        sys.exit(2)
    if not isinstance(findings, list):
        sys.exit(2)
    return findings

def main() -> None:
    base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
    diff = get_diff(base)
    if not diff.strip():
        sys.exit(0)  # nothing to review
    findings = call_model(diff)
    blocking = [f for f in findings if f.get("severity") in BLOCK_AT]
    for f in findings:
        loc = f"{f.get('file', '?')}:{f.get('line') or '?'}"
        print(f"[{f.get('severity', '?'):>8}] {loc} ({f.get('kind', '?')})")
        print(f"           {f.get('summary', '')}")
        if f.get("suggestion"):
            print(f"           -> {f['suggestion']}")
    if blocking:
        print(f"\nGate FAILED: {len(blocking)} high/critical finding(s).")
        sys.exit(1)
    print(f"\nGate passed ({len(findings)} non-blocking finding(s)).")

if __name__ == "__main__":
    main()

Three design decisions worth calling out, because they're the difference between a gate and a toy:

1. Unparseable output exits 2, not 0. If the model returns prose instead of JSON, the gate is inconclusive, and my hook treats exit 2 as "warn me but let me decide." Exiting 0 there would train me to trust a gate that sometimes isn't running at all.

2. Truncation is loud. Diffs over ~24k characters get truncated with a visible marker, because silent truncation means the model reviewed a different diff than the one I'm pushing. For genuinely huge diffs, the honest answer is to split the push — which the gate nagging you about is arguably a feature.

3. Severity threshold, not zero-findings. Low and medium findings print but don't block. If every style nit failed the gate, I'd disable it within a week. The gate exists to catch the "wait, that's a mutable default argument" class of mistake.

Wiring it into pre-push

# .git/hooks/pre-push (or via pre-commit framework)
#!/bin/sh
python ai_review_gate.py origin/main
code=$?
if [ $code -eq 1 ]; then
  echo "AI review gate failed. Fix findings or push with --no-verify (and own it)."
  exit 1
fi
# exit 2 (inconclusive) warns but allows the push
exit 0

Note the --no-verify escape hatch stays available. A gate with no override is a gate people uninstall; the point is to make skipping it a visible, deliberate act, not to make it impossible.

What I feed it and what I don't

Scenario	Verdict
Solo side projects, every push	Runs unconditionally; cost is zero, false positives are cheap
Work repo with human review anyway	Still useful — catches things before a colleague has to
Diff touching auth/crypto/payments paths	Gate runs, but I treat a "pass" as meaningless; human review required regardless
Diff > ~500 lines changed	Split the change; both the model and your reviewer will miss things
Generated code (migrations, lockfiles, protobuf)	Excluded via the `*.py` pathspec and a per-repo ignore list

Limitations, honestly

It has no project context. The model sees a diff, not your architecture. It will flag things that are fine given your conventions, and miss things that are wrong given your invariants. It catches local mistakes, not design errors.
Severity calibration drifts between models. I tuned BLOCK_AT against the free model I was using; if you switch models, re-check what it calls "high" for a week before trusting the threshold.
Free tiers change. Free access and free servers are availability claims as of when I set this up, not a contract. The script points at an OpenAI-compatible endpoint precisely so that if the free option disappears or its limits stop fitting my usage, I swap AI_REVIEW_BASE_URL and keep the gate.
Non-determinism is real even at temperature 0 across provider-side changes. The gate is a heuristic net, not a proof.

Who should skip this

If your team already has fast, thorough human review with short queues, this adds little — the human pass subsumes it. If your diffs are routinely enormous, fix that first; no review method, human or machine, survives a 2,000-line diff. And if you're in a regulated environment where code can't leave your perimeter, a hosted free endpoint is the wrong tool entirely — run a local model or don't do this.

Where this leaves the gate

My pre-push stack is now: lint and type checks (fast, deterministic), then this AI pass (slower, probabilistic), then a human for anything that matters. The AI layer catches a different class of mistake than the linters — "this logic inverts the flag" rather than "this line is too long" — and at zero marginal cost per push, the trade is easy.

If you try it, the thing I'd most like to hear is where the severity threshold lands for you. My guess is "high" is right for solo work and wrong for teams, but I only have my own repos to judge from. If you want to run the same setup, MonkeyCode's free tier is what I pointed the script at; the README for whatever endpoint you choose will have the base URL and model identifier to drop into the env vars.