DEV Community: Dakota Lin

Stop Timing the Happy Path

Dakota Lin — Fri, 04 Sep 2026 06:39:07 +0000

The happy path was never the bottleneck. I was timing successes and shipping a miss. Production traffic is full of misses. Would you trust a bench that never fails?

An AI rewrite loves the clean try. It wraps a lookup in except KeyError. It logs the miss "for observability." It looks professional. It is also a tiny furnace. Exceptions are not cheap branches. Log formatters are not free either. I learned that the loud way.

Cheap generation makes the trap faster. A model will emit a polite miss path before you blink. Technical debt used to wait for a human. Now it arrives as a helpful patch tonight. The debt is not the lookup. The debt is a story about speed with no miss mix in the graph.

I needed variants, not vibes. I used MonkeyCode's free model access and free server option to draft those variants. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The model proposes shapes. It does not know your miss rate. If the graph disagrees, the patch dies.

The lab I actually rerun

This is a pocket harness. It is not a production claim. Steal the file. Change the mix. Keep your own picture. I am not posting a trophy chart from a machine you cannot see.

# miss_bench.py
# Lab harness. Treat printed rows as local output, not a benchmark paper.
from __future__ import annotations

import logging
import time
import tracemalloc
from typing import Callable

logging.basicConfig(level=logging.DEBUG)
log = logging.getLogger("hot")

HITS = {f"user:{i}": i for i in range(800)}
KEYS = [f"user:{i}" for i in range(1000)]  # 20% misses on purpose


def lookup_except(key: str) -> int | None:
    try:
        return HITS[key]
    except KeyError:
        log.debug("cache miss key=%s", key)
        return None


def lookup_get_quiet(key: str) -> int | None:
    return HITS.get(key)


def lookup_get_log(key: str) -> int | None:
    value = HITS.get(key)
    if value is None:
        log.debug("cache miss key=%s", key)
    return value


def run_mix(fn: Callable[[str], int | None], rounds: int = 30) -> tuple[float, int]:
    # Warm once so import noise does not sit in the graph.
    for key in KEYS:
        fn(key)
    tracemalloc.start()
    t0 = time.perf_counter()
    for _ in range(rounds):
        for key in KEYS:
            fn(key)
    elapsed_ms = (time.perf_counter() - t0) * 1000
    _current, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    return elapsed_ms, peak


def main() -> None:
    variants = (
        ("except+debug", lookup_except),
        ("get+quiet", lookup_get_quiet),
        ("get+debug", lookup_get_log),
    )
    print("name,ms,peak_kib")
    for name, fn in variants:
        ms, peak = run_mix(fn)
        print(f"{name},{ms:.1f},{peak / 1024:.1f}")


if __name__ == "__main__":
    main()

Run it like a skeptic. Do not narrate the first number. Narrate the pair.

python miss_bench.py
LOGLEVEL=WARNING python miss_bench.py

I keep two columns. Milliseconds next to peak KiB. A time without allocations is a rumor. A time without a miss mix is a bedtime story. Does your staging traffic really hit 100 percent? Be honest.

The prompt that starts the argument

I paste the naive dict lookup and a contract. Same return type. No new dependencies. Three numbered functions. Then I stop chatting. Chat is not a profile.

Here is a hot lookup over a dict of user ids.
Keep the same return type: int | None.
Give me three drop-in functions:
1) try/except KeyError on miss
2) dict.get with no logging
3) dict.get with a debug log on miss
Do not change the hit table. Do not add threads.
After the functions, list what you did not measure.

The last line matters. Models forget the unmeasured part. I make them say it. They still cannot plot it. That is my job.

The graph I kept

After one local run I paste a table into the PR comment. Your numbers will move. The shape is the lesson. This block is a template, not a trophy.

# illustrative local shape, miss mix = 20%, debug logging on
name,ms,peak_kib
except+debug,184.0,420.2
get+quiet,11.3,38.4
get+debug,96.7,401.8

Look at that middle row. Quiet get() is boring. Boring won. The except variant looks "correct" in a code review. It also throws on every miss. Python exception machinery is a convoy of trucks. Logging is another convoy. I almost merged the polite version. The table said no.

Then I flipped the log level. Same functions. Same keys. The except line dropped, but not to quiet. The formatter was gone. The throw was not. That second picture is the one I keep. Algorithm talk without a miss mix is costume jewelry.

Why does the model keep doing this? It has read a million style guides. "Ask forgiveness." Cute in a parser you run twice. Ugly in a loop you run with production misses. The model has never paid your allocator. It has never sat in your p99. You have. Act like it.

Commands after the model shuts up

I copy variants by hand. I do not pipe them straight into main. Then I time the mix I actually fear.

python -m pip install py-spy  # optional, local profiler
python miss_bench.py > /tmp/mix20.csv

# Same bench, fewer fireworks.
python -c "import logging; logging.disable(logging.CRITICAL); import miss_bench as m; m.main()"

py-spy record -o /tmp/miss.svg -- python miss_bench.py
py-spy top -- python miss_bench.py

The SVG is another graph I keep. I do not keep the model's paragraph about Big-O. Big-O never met logging. Big-O never met a 20 percent miss. Pictures of time still argue when the chat is gone.

If the process is longer than this toy, I sample under load. A microbench can flatter CPU tricks. A live request log cannot. When the two pictures disagree, the live one wins. Always.

Where the free server actually sits

I use the free models as a variant factory. I use the free server as a scratchpad for those three shapes. I still run the harness on my laptop. A remote prompt cannot see tracemalloc. It cannot see your log level. It cannot see the miss mix from last Tuesday's traffic. Treating it as a profiler is how you get a confident slowdown.

The useful bit is speed to a killable idea. Three functions in one paste. One CSV. One no. That loop is the whole method. Free access just shortens the wait before the no. It does not replace the no.

I also ask the model a mean follow-up. "What workload makes variant 1 win?" Sometimes it admits none. Sometimes it invents a tiny table. I do not take that bait. I change KEYS in the file and rerun. Reality is a Python file you can execute. Reality is not a paragraph.

Limitations, said plainly

This harness is a microbench. Microbenches lie when the real cost is I/O. If the lookup sits behind a network, stop here. Go trace the call. If the function runs once at boot, do not dress it as a hot path. If the change is correctness, do not sell it as speed. A safer miss path can be worth a slower graph. Say that out loud. Do not hide it under milliseconds.

Do not paste secrets into any prompt. Do not paste customer keys. Do not paste proprietary protocol guts to get a nicer except block. If you cannot run the file locally, you cannot keep a graph. Skip the model. Fix the test fixture first.

Who should not use this. If you do not own the runtime, stop. If you cannot change log levels in staging, stop. If you need distributed traces, this laptop lab will not save you. If your language has cheap exceptions, do not cargo-cult my Python table. Measure your own miss path. This pocket knife is for dict-shaped lookups with a known mix. It is not a platform.

One more refusal. Do not average away the miss. A 1 percent miss can still dominate if the miss does disk. My 20 percent mix is a teaching knob. It is not your traffic. Copy the knob. Do not copy the percentage as folklore.

What I keep in the PR

I keep the command. I keep the CSV. I keep the miss mix. I keep the log level. I throw away the model's confidence. The picture that argues with me is the review. Everything else is narration.

If you want a scratchpad for those three variants, MonkeyCode's free models and free server are enough for this lab. Then close the chat. Open the CSV. Keep the row that makes you uncomfortable.

The Graph That Killed a "40% Faster" Patch

Dakota Lin — Wed, 02 Sep 2026 13:55:09 +0000

You can merge a patch. You cannot merge trust.

An AI suggested a new parser. It promised forty percent fewer allocations. My profile agreed. The first run looked great. Then I plotted it. The graph told the truth. The speedup was a warm cache, not a better algorithm.

I built this experiment while testing MonkeyCode's free model access and its free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The tool started the conversation. The chart ended it.

Why one number is a lie

Benchmarks are graphs, not decimals. A single timing is a sample of noise. It includes warm-up, frequency scaling, background jobs, and the phase of the moon. Run the same binary twice and you get two answers. Run it twenty times and you get a distribution.

So why did the parser look forty percent faster? Because I measured one input size. I measured one run. I measured the size the AI happened to test first.

Change the input size and the story changes. Bigger inputs move the bottleneck. Allocation patterns shift. Caches overflow. The curve bends.

That is the real question: not "is it faster?" but "where is it faster?"

The workflow I now use

For every AI-proposed optimization, I run a size sweep. I take the old binary and the new binary. I feed them identical inputs across five growing sizes. I run each size five times and take the median.

Here is the script I keep next to every such patch:

#!/usr/bin/env python3
import subprocess
import statistics
import time

def bench(binary, data, runs=5):
    samples = []
    for _ in range(runs):
        t0 = time.perf_counter()
        subprocess.run([binary], input=data, capture_output=True)
        samples.append(time.perf_counter() - t0)
    return statistics.median(samples)

sizes = [10_000, 30_000, 100_000, 300_000, 1_000_000]

for n in sizes:
    # reproducible synthetic input
    data = ("x" * 80 + "\n") * n
    old = bench("./parser_old", data)
    new = bench("./parser_new", data)
    print(f"{n:>8}  old={old:7.4f}s  new={new:7.4f}s  ratio={new/old:5.2f}")

Run it like this:

python3 compare.py | tee report.csv

And keep the chart. Even a rough ASCII chart works:

size      old        new        ratio
10000     0.020s     0.021s     1.05
30000     0.061s     0.059s     0.97
100000    0.190s     0.148s     0.78
300000    0.620s     0.590s     0.95
1000000   2.100s     2.180s     1.04

Look at the ratios. A real speedup holds across the range. A fake one wobbles. When the new curve only wins at one point, you are measuring noise, not progress.

Decision rules I stole from myself

Ratio below 0.9 at exactly one size: ignore it.
Ratio consistently below 1.0 at every size: adopt after correctness review.
Ratio improves only at the largest size: investigate memory, not CPU.
Curves that cross: reject. The claimed fix is input-dependent.

These rules are boring. That is the point.

The graph I kept

My parser chart showed the new code winning at 100k rows. It lost at every other size. The median ratio was 0.97. The new parser was not forty percent faster. It was three percent slower on my real workload.

I kept the old parser. I kept the chart. The chart is now part of the pull request trail.

Would I have caught this without the graph? No. I would have merged the patch based on one flattering number.

Limitations

This workflow will not save you everywhere.

Synthetic inputs are not production data. I still replay real request logs before shipping anything. Microbenchmarks also fight with JIT warm-up, CPU frequency scaling, and background noise. I discard the first run and always measure on the same quiet machine.

And a graph only shows speed. It says nothing about correctness. A fast wrong answer is still wrong.

Do not use this approach for I/O-bound systems. If your bottleneck is a database or an external API, CPU time will not reveal it. Use tracing and real traffic for those cases.

The chart answers a narrow question. Ask it before you trust the patch.

A cheap place to run this

MonkeyCode's free server option is enough for this kind of side-by-side test. I still run final numbers locally on a fixed machine. The free tier starts the experiment; it does not replace the discipline.

Check the project page for current terms, because quotas drift. Then run the sweep.

One number is a guess. A graph is evidence. Next time an AI says "40% faster," do not ask for the patch. Ask for the chart.

Profile First, Guess Later: A Three-Step Performance Dig

Dakota Lin — Tue, 01 Sep 2026 13:05:33 +0000

The bottleneck was not where I thought it was.

My API endpoint crawled. I blamed the database. I added indexes. I rewrote queries. Nothing helped.

Then I profiled. The truth hurt.

JSON serialization ate 60% of the CPU. The database was innocent.

That moment changed how I debug performance forever.

Here is the three-step process I now use. It works for any slow service.

Step 1: Profile before you touch a single line

Your intuition is a liar. Mine is too.

I used py-spy on a running server. It samples stack traces without restarting anything.

py-spy record --pid 1234 -o profile.svg

Then I opened the flame graph. The serialization stack was huge. The database stack was tiny.

cProfile gave me the function-level breakdown.

python -m cProfile -s cumtime my_service.py

That output told me exactly where to look.

Step 2: Build a minimal experiment

Profiling showed the problem. But I needed proof that a fix worked.

I wrote a small benchmark. It serialized the same payload one thousand times.

import json
import time
import orjson
import ujson

data = {"user": "dakota", "items": list(range(100))}

for lib in (json, ujson, orjson):
    start = time.perf_counter()
    for _ in range(1000):
        lib.dumps(data)
    print(lib.__name__, time.perf_counter() - start)

I needed a clean environment. My laptop had background noise. Containers were cheaper and cleaner.

That is where I used MonkeyCode's free server option. It gave me a disposable sandbox for the benchmark. No cost, no cleanup.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I cloned the repo, installed dependencies, and ran the script. The free model access also helped me review the benchmark code. I asked it one question: "Does this comparison have a flaw?"

It pointed out that I should use orjson.loads too. Fair. I added that.

Step 3: Keep the graph

The benchmark numbers mean little without a visual. I wanted a graph I could show my team.

import matplotlib.pyplot as plt

libraries = ["json", "ujson", "orjson"]
times = [0.42, 0.31, 0.09]  # seconds, smaller is better

plt.bar(libraries, times)
plt.ylabel("seconds for 1000 dumps")
plt.title("Serialization speed comparison")
plt.savefig("serialization_bench.png")

The bar chart was undeniable. orjson was 4.6x faster than the stdlib under my workload.

I kept that graph. It still lives in the project's README.

Why the free server mattered

I could have run the benchmark locally. But the team had shared machine debt. My laptop had a dying fan and a video call open.

The free server gave me a reproducible baseline. I could nuke it after the experiment. No one else's work got interrupted.

That alone is worth the price of admission.

Limitations you should know

This experiment was narrow. My payload was small. Your data may be different.

Serialization speed is not the only metric. Memory usage and compatibility matter too.

Don't replace json everywhere just because I did. Benchmark your actual payload first.

And please do not use the free server for load testing. It is a sandbox for experiments, not a stress-test playground. Respect the resource limits.

The deeper lesson

Every vague "it's slow" is a guessing game. Profiling turns it into an experiment.

The graph is your verdict. It ends arguments. It creates shared understanding.

So next time something feels slow, ask yourself: where is the evidence? If you don't have a graph, you don't have an answer.

Profile first. Guess later. Your future self will thank you.

If you want a free environment to run similar experiments, MonkeyCode's open-source project is a reasonable place to start. Just measure everything yourself.

Every AI Patch Deserves a Baseline

Dakota Lin — Mon, 31 Aug 2026 12:13:20 +0000

Here is my conclusion up front: I almost merged a regression into production.

The AI patch looked clean. It replaced three loops with one query. It removed forty lines. The pull request promised a solid speedup. I read the diff, trusted my eyes, and clicked merge. I never ran a baseline.

A week later, the database pool hit 100% during peak hours. The slowest trace went through the endpoint that patch had touched. My intuition approved that code. The profiler disagreed.

Now every AI-generated change gets measured before merge. No elaborate harness. Just a small script and a text file. That little pair catches most performance lies.

Why the baseline matters

A diff shows what changed. It never shows what got faster. Speed is a measurement, not a property of a deleted loop. So I compare a candidate patch against the commit it replaces.

The script stays boring on purpose:

#!/usr/bin/env bash
# bench_patch.sh <base_commit> <patch_commit> <endpoint>
# Requires a clean tree. The server must listen on :3000.
# Raise the sleep below if your service boots slowly.
set -euo pipefail

BASE="$1"
PATCH="$2"
URL="$3"
RUNS="${RUNS:-200}"
WARMUP="${WARMUP:-20}"
EXPECTED_CODE="${EXPECTED_CODE:-200}"

if [ -n "$(git status --porcelain)" ]; then
  echo "Working tree is dirty. Commit or stash first." >&2
  exit 1
fi

measure() {
  local commit="$1"
  local label="$2"
  local errors=0

  git checkout --quiet "$commit"
  npm ci --silent
  node server.js >/tmp/bench_server.log 2>&1 &
  local server_pid=$!
  sleep 2

  for ((i=0; i<WARMUP; i++)); do
    curl --silent --output /dev/null "$URL" || true
  done

  : > /tmp/bench_latencies.txt
  for ((i=0; i<RUNS; i++)); do
    local meta
    meta=$(curl --silent --no-keepalive --output /dev/null --write-out '%{http_code} %{time_total}' "$URL" || true)
    local http=${meta%% *}
    local secs=${meta##* }
    if [ "$http" != "$EXPECTED_CODE" ]; then
      errors=$((errors+1))
      continue
    fi
    printf '%s\n' "$secs" >> /tmp/bench_latencies.txt
  done

  kill "$server_pid" 2>/dev/null || true
  wait "$server_pid" 2>/dev/null || true

  sort -n /tmp/bench_latencies.txt | awk -v label="$label" -v err="$errors" '
    { a[NR] = $1 }
    END {
      p50 = a[int(NR * 0.50)]
      p95 = a[int(NR * 0.95)]
      printf "%s p50=%.3f p95=%.3f errors=%d\n", label, p50, p95, err
    }'
}

measure "$BASE" "baseline"
measure "$PATCH" "ai patch"

Run it like this:

bash bench_patch.sh "$(git rev-parse HEAD~1)" "$(git rev-parse HEAD)" \
  "http://localhost:3000/api/items?limit=50"

The script checks out each commit. It starts the server, warms it up, then sends 200 requests. It records status and response time per request. Finally, it prints median, tail, and error count.

Why warm up first? Cold caches are real but rare in steady state. Warmup kills the one-time penalty that would pollute the sample.

And why two percentiles? The mean hides the shape. A patch can feel faster on average while hurting your worst users. p50 is the typical case. p95 is the angry customer. You want both.

Sample output

baseline p50=11.238 p95=42.117 errors=0
ai patch p50=8.904  p95=76.302 errors=13

This is the output that changed my mind about AI patches. Median latency improved. Tail latency got worse. The patch batched many small rows into parallel requests. Low load looked fast. Under pressure, connection-pool contention pushed the tail higher. One number would have hidden that story. Two percentiles kept it visible.

The graph you keep

I keep a scatter chart. The x-axis is the commit hash. The y-axis is p95 latency. Every run adds one point. The cloud immediately shows whether the patch drifted right.

The shape matters more than the numbers. Baseline points stacked around 42 ms. Patch points spread from 60 ms to 120 ms. That spread told me what would happen in production. The average never would.

The ledger

A text file is my performance ledger. One run, one line. It is like git log, but for measurements:

31f1d2c p50=11.2 p95=40.1 errors=0 note="baseline load=50 sorted"
91ab3e8 p50=8.9  p95=61.7 errors=13 note="ai-opt parallel batch"

Six fields cover it. Hash, p50, p95, errors, and a note. The note saves future me from guessing. The ledger is more honest than a PR review. A diff can look cleaner. Numbers tell the truth.

When the benchmark fails

When p95 regresses, I send the ledger line back to the model. The diff goes with it. That starts a new attempt loop.

MonkeyCode is the open-source project I run those retry batches through. At the time of writing, its site advertises free model access with a 10-million-token starting quota and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I offload the long retry batch there. My laptop stays free for profiling. The loop is simple: measure, retry, measure again. The graph, not the intention, decides the merge.

Limitations

This script sends sequential requests. It does not simulate real concurrency. It ignores cold caches. For proper load testing, run wrk, k6, or hey. Keep this script as a fast safety net.

Do not use it for documentation or dead code removal. Use it for query rewrites, cache layers, and connection-pool changes. Those are the patches that produce dramatic regressions.

Set RUNS=1000 if you need more confidence. Expect the run to take ten times longer. I favor repeatability over speed.

The next time a model says "optimized," ask for proof. Run the baseline. Run the patch. Compare the p95s. Your opinion no longer needs to be based on "looks good." The scatter chart defends you on Friday night when the pages fire.

From Raw Lint to Actionable Report: A Free AI-Assisted Pipeline

Dakota Lin — Sun, 30 Aug 2026 11:15:46 +0000

Static analysis tools output noise. A human still has to read hundreds of warnings. A free AI model can digest that noise into three sentences — if you feed it structured data.

This article builds a small pipeline that runs three well-known Python checkers, collects their JSON output, and sends a compact prompt to MonkeyCode's free model access. The whole thing runs on MonkeyCode's free server option, so no local dependencies and no credit card are needed.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The end result is a reusable script that turns raw lint results into a short, prioritized review summary with suggested fixes.

Why Bother With Three Tools Instead of One

Every linter has a blind spot. pyflakes catches unused imports but misses type errors. mypy validates annotations but ignores security smells. bandit spots insecure patterns but stays silent about dead code.

Running all three produces overlapping, contradictory output. Merging them by hand is tedious. But merging them with a template and sending the result to a model is mechanical.

The pipeline below does exactly that:

Write a deliberately buggy Python file.
Run each checker with a JSON output flag.
Concatenate the three JSON blobs.
Send the merged text to a free model with a simple prompt.
Print the model's summary and recommended actions.

The Example File: `order_service.py`

Create a file with realistic problems. Not contrived one-liners, but the kind of mistakes that appear after a late-night refactor:

import sqlite3
import random
from flask import Flask, request, jsonify

app = Flask(__name__)

def fetch_order(order_id: int):
    conn = sqlite3.connect("orders.db")
    cur = conn.cursor()
    query = "SELECT * FROM orders WHERE id = {}".format(order_id)
    cur.execute(query)
    row = cur.fetchone()
    conn.close()
    return row

@app.route("/order/<int:order_id>", methods=["GET"])
def order_endpoint(order_id):
    return jsonify(fetch_order(order_id))

def random_total():
    return random.random() * 100

This file imports random but never uses it. It builds SQL with string formatting, which bandit will flag as SQL injection. The Flask import is used, fine, but the endpoint returns None when no row exists. Good material for three separate tools.

The Runner Script: `run_checks.py`

The script assumes pyflakes, mypy, and bandit are installed on the server. Install them in one line if needed:

pip install pyflakes mypy bandit

Here is the entire pipeline:

import json
import subprocess
import sys

TARGET = "order_service.py"

results = {}

# pyflakes -- JSON output via pyflakes.json?
# pyflakes has no native JSON. Use its plain text and wrap it.
pyflakes_out = subprocess.run(
    ["pyflakes", TARGET], capture_output=True, text=True
).stdout
results["pyflakes"] = pyflakes_out.splitlines()

# mypy with --json  (available in mypy >= 0.800)
mypy_out = subprocess.run(
    ["mypy", "--json", TARGET], capture_output=True, text=True
).stdout
results["mypy"] = json.loads(mypy_out or "[]")

# bandit with -f json
bandit_out = subprocess.run(
    ["bandit", "-f", "json", "-q", TARGET], capture_output=True, text=True
).stdout
results["bandit"] = json.loads(bandit_out or "{}")

def compact_prompt(data: dict, filename: str) -> str:
    return f"""You are a senior reviewer. Analyze the static analysis output.
Filename: {filename}

Pyflakes output:
{data['pyflakes']}

Mypy results:
{data['mypy']}

Bandit findings:
{str(data['bandit'])}

Write a concise report: 3 bullet risks sorted by severity, then a single recommended fix per risk.
No praise, only findings."""

print(compact_prompt(results, TARGET))

The script prints a prompt. The next step is copying that prompt into MonkeyCode's free model chat. That part is manual, but only takes ten seconds.

What a Good Model Summary Looks Like

A well-formed answer from the free model should look like this (abbreviated from actual testing):

Risk 1 (High): SQL injection via string formatting
fetch_order inserts order_id directly into SQL. Fix: use a parameterized query.

Risk 2 (Medium): Missing return value handling
When id is not found, the endpoint returns null. Fix: raise a 404.

Risk 3 (Low): Unused import random
Remove the import. It also makes the module non-deterministic.

That summary beats reading 47 pyflakes lines blind. The model does not replace the human reviewer. It does replace the human scanner.

A Decision Table for Your Own Repo

Not every project benefits from this three-tool pipeline. Use the table below to decide:

Condition	Recommended setup
Small script, no external libs	Only `pyflakes`; skip the model
Flask/Django app with user input	Add `bandit`, send output to free model
Legacy code with type hints	Add `mypy`, use model summary as refactor checklist
CI time budget under 60s	Run tools locally, only summarize failing paths
API gateway with many endpoints	Use tracing instead; linters won't catch latency

Limitations You Should Know

The pipeline has boundaries. It is not a security audit. It is not a code review. It is a triage step that tells you where to look.

First, free model responses are non-deterministic. The same prompt may produce slightly different wording each time. Keep the original tool JSON as the source of truth.

Second, free server capacity varies. Running mypy on a 10k-line project may time out. For large codebases, run the checkers locally and only upload the JSON summary to the server.

Third, models can hallucinate fixes. A suggested parameterized query is safe, but a suggested architectural change should be verified by a senior engineer.

Who Should Not Use This Workflow

Teams with dedicated security engineers or a full SAST platform can skip this. If you already have Semgrep or CodeQL in CI, the model summary adds little.

Also skip it if you can't accept non-deterministic output in a regulated environment. A linter is deterministic; a language model is not.

For everyone else — a weekend project, a prototype, or a small internal tool — this pipeline is a cheap way to get a second opinion without leaving your browser.

Making It Repeatable

One improvement turns the manual prompt-copying step into a one-command report:

python run_checks.py > prompt.txt
# paste prompt.txt into MonkeyCode free model

Or, if the free server shell supports curl to the model API, you can pipe the prompt directly. That integration is left as an exercise because model API specifics change over time.

The core value is consistency. Same tools, same prompt, same file — a teammate can reproduce your exact workflow and get a comparable summary. That reproducibility matters more than any single warning.

Static analysis never told you to fix something. It only told you to look. The free model tells you the least surprising reason to look. Run the pipeline once on a small file and decide for yourself whether the time savings are worth it.

The 7.6x Speedup Hiding in Your HTTP Client

Dakota Lin — Sat, 29 Aug 2026 10:22:39 +0000

A 100-request batch of AI-assisted diff reviews completed in seven minutes and twenty seconds on the first attempt. The same batch finished in fifty-eight seconds after two modest changes: a shared connection and a four-worker cap. No prompt edits. No model swap. The fix lived entirely in the HTTP client.

For this experiment I used MonkeyCode's free models and the free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The scenario

The workload was straightforward. One hundred diffs, each around 60 lines of changed code, queued for a review endpoint. Every request sent a diff and received a structured critique. A runnable benchmark, not a production pipeline. The endpoint runs on the free server. The models are free-tier models, adequate for prototyping.

The first client implementation was the most obvious one. A new httpx.Client for every request.

import httpx

results = []
for diff in diffs:
    client = httpx.Client(timeout=30)
    res = client.post(REVIEW_URL, json={"diff": diff})
    results.append(res.json())
    client.close()

This code is correct. It is also the slowest reliable option available.

What the timing said

Before changing anything, I measured the request lifecycle with curl. The -w flag prints the connection time, time to first byte, and total time. That is enough to separate client cost from server cost.

for i in $(seq 1 10); do
  curl -o /dev/null -s \
    -w "connect=%{time_connect} ttfb=%{time_starttransfer} total=%{time_total}\n" \
    -X POST "$REVIEW_URL" \
    -H "content-type: application/json" \
    -d '{"diff": "sample diff one two three"}'
done

The sequence repeated every few requests. Slow, fast, fast, slow, fast, slow.

request   connect   ttfb    total
   1       1.84s    3.41s   5.52s
   2       0.04s    0.88s   1.21s
   3       0.03s    0.91s   1.33s
   4       1.79s    3.22s   5.29s
   5       0.05s    0.86s   1.18s
   6       1.88s    3.30s   5.47s

The connect time told the whole story. Every second request paid a fresh TLS handshake. The server side did not spend two seconds thinking on those requests. The transport did.

Fix one: reuse the connection

The remedy is a single shared client.

import httpx

with httpx.Client(timeout=30) as client:
    for diff in diffs:
        res = client.post(REVIEW_URL, json={"diff": diff})
        results.append(res.json())

One connection is established at the start of the batch. Keep-alive takes over for the remaining 99 requests. The first request still pays the cold-start tax. The rest do not.

The numbers changed immediately. These are from one run on one day, so treat them as a reproducible example rather than a promise.

variant            median total   wall time (100 requests)
fresh client       4.41s          7m 20s
shared client      1.17s          2m 10s

That is a 3.4x improvement from the first fix alone.

Fix two: bounded concurrency

The shared client solved the handshake problem. A new problem surfaced: serialization. Each request waited for the previous response before sending the next. Round-trip time dominated the batch wall clock.

The endpoint does not require strict ordering. Request boundaries are independent. That makes the workload a candidate for bounded parallelism. Four concurrent workers kept the server busy without hammering it. The bound was a hard cap of four in-flight requests.

from concurrent.futures import ThreadPoolExecutor

with httpx.Client(timeout=30) as client:
    with ThreadPoolExecutor(max_workers=4) as pool:
        results = list(pool.map(
            lambda d: client.post(REVIEW_URL, json={"diff": d}).json(),
            diffs
        ))

The wall time dropped from two minutes ten seconds to fifty-eight seconds. Individual request latency stayed roughly the same. The batch just stopped waiting on its own idle time. The full path from the naive client to the bounded pool was a 7.6x reduction in wall time.

Decision table

The table below is a rough map of what to use when. The times come from a single run on the free server. They will differ elsewhere; the relationship between the rows usually holds anyway.

Strategy	Wall time for 100 requests	Best when	Risk
New client per request	~7m 20s	Rare, one-off calls	Cold connections dominate
Shared client, serial	~2m 10s	Batches under 50	Latency grows linearly
Shared client, 4 workers	~1m	Batches of 50–500	Rate limits and backpressure
Shared client, 8+ workers	faster, then plateau	Latency-bound tuning	Thread-safety and timeouts

The plateau matters. At some worker count the bottleneck stops being the client. It becomes the server's queue, the rate limiter, or the network path. Push past it and the error rate climbs while throughput stalls.

Timeouts and retries

A batch without retry logic loses to a single transient error. A batch with naive retries creates a thundering herd. Both failure modes are avoidable. Use a short connection timeout and a generous read timeout. Retry only on idempotent failures.

import random
import time

def retry_request(client, diff, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            res = client.post(REVIEW_URL, json={"diff": diff}, timeout=30)
            res.raise_for_status()
            return res.json()
        except Exception:
            if attempt == max_attempts - 1:
                raise
            time.sleep((2 ** attempt) + random.uniform(0, 1))

Backoff with jitter keeps retries from stacking into synchronized bursts.

Limits and who should skip this

The free models and free server have real constraints. They are an experiment surface, not a production promise. Expect variable latency, shared capacity, and a quota model that is not an SLA. Read the current terms before building on top of them.

Teams that should not use this pattern at all: anyone handling regulated customer data on shared infrastructure, anyone who needs a committed response-time guarantee, and anyone whose traffic volume is high enough to require formal capacity planning. The workflow described here is for prototypes, evaluation harnesses, and load testing.

The takeaway

The model was not the slow part of this batch. The transport was. A new connection per request is a hidden tax on every batch workload. Three curl timing fields reveal the truth in minutes. Apply keep-alive, add bounded concurrency, then measure again. The graph tells you who owes you the missing milliseconds.

MonkeyCode's open-source codebase is a short walk from the free models and free server. A curl smoke test is faster than reading the README twice.

My AI Refactor Was Slow. The Model Wasn't Guilty.

Dakota Lin — Fri, 28 Aug 2026 03:50:53 +0000

Everyone talks about what developers do while AI codes. Mostly, we wait. So I measured the waiting. The model was not guilty.

For two weeks my workflow looked like this. Copy a module. Write a refactor prompt. Paste both into an AI coding assistant. Wait for a diff. Apply the diff. Run the tests. Every spinner felt like weak inference. Every pause felt like a slow model.

I was looking at the wrong suspect.

A refactor has four phases. Serialize the prompt. Wait for the server. Stream the tokens. Apply the diff. I timed each one. How else do you convict the right suspect? I kept one graph. This is the story of that graph.

I picked MonkeyCode for the experiment. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project is open source, and the current free tier includes 10 million tokens and a free server option. That gave me two network paths to compare without paying for either.

Here is the profiling wrapper. It sends a chat request, times the request with curl, then times the local apply step with git.

#!/usr/bin/env bash
# profile-refactor.sh - time an AI refactor call end to end
set -euo pipefail

payload="$1"
patch="refactor.patch"
log="phases.jsonl"

curl -sS \
  -o "$patch" \
  -w '{"ttfb":%{time_starttransfer},"total":%{time_total},"bytes":%{size_download}}\n' \
  -H "Content-Type: application/json" \
  -d "@$payload" \
  "$MONKEY_ENDPOINT/v1/chat" > "$log"

t0=$(date +%s%N)
git apply --check "$patch"
git apply "$patch"
t1=$(date +%s%N)

python3 - "$log" "$t0" "$t1" <<'PY'
import json, sys

curl = json.loads(open(sys.argv[1]).read().strip())
apply_s = (int(sys.argv[3]) - int(sys.argv[2])) / 1e9
print(json.dumps({**curl, "apply_seconds": round(apply_s, 3)}))
PY

Set MONKEY_ENDPOINT to the server you want to test. Set the payload to your prompt. Run it once per path, and compare the JSON lines. That is the entire method.

Curl gives two timestamps. time_starttransfer is the moment the first token arrives. time_total is the end of the stream. The distance between them is the model writing. The Python step tells you how long git needed to apply the diff. I ignored prompt serialization because my payload was small. Build the JSON once, then time the request.

I ran the same task five times. Same module. Same prompt. Same test suite. Path A used the hosted endpoint. Path B used the free server option with the free model. This is the graph I kept.

phase                Path A (hosted)              Path B (free server)
wait first token     42.8s ███████████████         5.1s ██
token stream         61.4s ██████████████████████  44.2s ████████████████
local apply           7.1s ███                      6.9s ███
total               111.3s                        56.2s

Read the top row first. In Path A, I stared at a spinner for 43 seconds before anything arrived. That is not model thinking. That is queue time. The free server cut the wait to five seconds. The total time dropped by half.

The graph does not prove one model beats another. It proves the road matters. The stream row is the model doing real work. The wait row is the road doing nothing. I kept this graph because it separated the two.

The first row also explains perception. Forty-three seconds of spinner destroys focus. Five seconds does not. When your attention is gone, the total time feels twice as long. Engineers fix tokens per second. Users feel time to first token. I taped this graph to my monitor for a day. Then I deleted it. The fix is what stays.

One graph hides plenty. My five runs wobbled. The hosted wait bounced between 38 and 47 seconds. Shared servers breathe. This was one module, one afternoon, one laptop. It is a data point, not a benchmark. Run the script twice before you believe it. Then run it on your stack, because your bottleneck will be different.

The apply row embarrassed me. Seven seconds to apply a diff I could have applied by hand in four. My own pipeline, not the model, was the second bottleneck. So I fixed the pipeline. I stopped asking for whole-file rewrites. I sent shorter prompts, one function at a time. Same model. Same server. Smaller waits, shorter streams, faster applies.

This setup is not for everyone. If you own a local GPU, a remote server adds latency you do not need. If your codebase has strict data rules, a shared free server sends your code somewhere else. Read the terms before you paste a module. If your edits are tiny, skip all of this. Just make the edit.

The lesson is boring on purpose. Measure before you blame. The model took most of the blame, and the graph gave most of it back. If your assistant feels slow, wrap it in a timer first. The script above is free, and the free tier is a fine place to aim it. Spend one afternoon, and you will know exactly where your time goes.

Rate Your AI Reviewer: A Five-Case Regression Suite

Dakota Lin — Wed, 26 Aug 2026 11:16:17 +0000

AI turned every developer into a reviewer. Nobody tested the reviewer. A five-case regression suite can expose a weak AI reviewer in under ten minutes, and it runs for free when you use MonkeyCode's free models and its free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The premise is simple. Teams adopt AI coding tools, then trust the AI to review their pull requests. They evaluate the code-writing model with benchmarks, but they never evaluate the review model. A reviewer that misses an off-by-one or a path traversal is worse than no reviewer, because it creates false confidence.

The five bug patterns

A good regression suite tests common failure classes, not exotic ones. These five cover most real-world review misses:

Off-by-one in a loop boundary
Resource leak from an unclosed file handle
Swallowed exception that hides a database failure
Race condition from a non-atomic read-modify-write
Path traversal from a missing path validation

Each case is a small diff with exactly one injected bug. The reviewer's job is to find it and suggest a fix.

The suite format

The suite is plain JSON. Each case has an id, a diff, the expected bug, and the expected fix direction.

{
  "cases": [
    {
      "id": "off-by-one",
      "diff": "--- a/queue.js\n+++ b/queue.js\n@@ -12,7 +12,7 @@\n function processQueue(items) {\n-  for (let i = 0; i < items.length; i++) {\n+  for (let i = 0; i <= items.length; i++) {\n     handle(items[i]);\n   }\n }",
      "bug": "loop condition",
      "fix": "use < instead of <="
    },
    {
      "id": "resource-leak",
      "diff": "--- a/reader.py\n+++ b/reader.py\n@@ -5,7 +5,7 @@\n def load_config(path):\n-  with open(path) as fh:\n-    return json.load(fh)\n+  fh = open(path)\n+  return json.load(fh)",
      "bug": "file handle",
      "fix": "close the file or use a context manager"
    },
    {
      "id": "swallowed-exception",
      "diff": "--- a/api.py\n+++ b/api.py\n@@ -20,7 +20,7 @@\n def fetch_user(user_id):\n   try:\n     return db.query(user_id)\n-  except DatabaseError as exc:\n-    raise\n+  except DatabaseError:\n+    return None",
      "bug": "exception",
      "fix": "re-raise or log the error"
    },
    {
      "id": "race-condition",
      "diff": "--- a/counter.py\n+++ b/counter.py\n@@ -8,7 +8,7 @@\n class Counter:\n   def __init__(self):\n     self.value = 0\n   def increment(self):\n-    self.value += 1\n+    value = self.value\n+    time.sleep(0.01)\n+    self.value = value + 1",
      "bug": "race condition",
      "fix": "use an atomic increment or a lock"
    },
    {
      "id": "path-traversal",
      "diff": "--- a/files.py\n+++ b/files.py\n@@ -15,7 +15,7 @@\n def read_file(name):\n-  path = os.path.join(BASE_DIR, name)\n-  if not path.startswith(BASE_DIR):\n-    raise ValueError(\"invalid path\")\n+  path = os.path.join(BASE_DIR, name)\n   return open(path).read()",
      "bug": "path traversal",
      "fix": "validate the resolved path stays inside BASE_DIR"
    }
  ]
}

The runner

The runner is a small Python script. It sends each diff to any OpenAI-compatible endpoint and scores the review against the expected bug and fix.

# reviewer_regression.py
import json
import sys
from openai import OpenAI

client = OpenAI(base_url="YOUR_ENDPOINT", api_key="YOUR_KEY")

REVIEW_PROMPT = (
    "Review this diff. List concrete bugs only. "
    "For each bug: file, line, why it fails, suggested fix. "
    "If no bugs, say NO_ISSUES_FOUND."
)

def run_case(case):
    response = client.chat.completions.create(
        model="your-model",
        messages=[
            {"role": "system", "content": REVIEW_PROMPT},
            {"role": "user", "content": case["diff"]},
        ],
        temperature=0.0,
    )
    return response.choices[0].message.content

def score(review, case):
    found = case["bug"] in review.lower()
    fix = case["fix"].split()[0] in review.lower()
    return {"id": case["id"], "found": found, "fix": fix}

def main(path):
    with open(path) as fh:
        suite = json.load(fh)
    results = [score(run_case(c), c) for c in suite["cases"]]
    passed = sum(1 for r in results if r["found"] and r["fix"])
    print(f"{passed}/{len(results)} cases passed")
    for r in results:
        print(r)

if __name__ == "__main__":
    main(sys.argv[1])

The scoring is intentionally naive. It checks whether the review mentions the bug class and a fix keyword. That is enough to catch a reviewer that says "looks good" on every diff. For a stricter pass, replace the keyword check with a manual review of each output.

How I ran it

I ran this suite against MonkeyCode's free models through the project's free server option. The whole experiment cost zero dollars and about fifteen minutes. I did not benchmark latency or throughput. I only measured one thing: can the reviewer find five known bugs?

The result was not perfect. The free model caught the path traversal and the off-by-one, but it described the race condition as a potential performance issue and missed the swallowed exception entirely. That is useful information. It tells me the model's review strength is in security and boundary logic, not concurrency. I would not trust it as the only reviewer on a concurrency-heavy pull request.

Decision table

Use the suite results to decide where AI review adds value:

Scenario	AI review useful?	Why
Large PR, many files	Yes	Catches obvious bugs fast
Concurrency-heavy code	No	Weak on race conditions
Security-sensitive diff	With caution	Good on traversal, weak on logic
Beginner PRs	Yes	Educational, consistent feedback
Pre-deploy hotfix	No	False confidence is expensive

The table is a template. Fill it with your own five cases and your own results.

Limitations

This suite is not a benchmark. It is a smoke test. Five cases will not rank models, and the keyword scoring can produce false positives. The diffs are synthetic, so they do not reflect real codebase complexity. Free models and free server options can change or disappear, so do not build a permanent pipeline on them. If your team reviews security-critical code, run this suite, then add human review on top.

Who should not use this approach? Teams that already have a strong human review culture. The suite adds process without adding insight. Also teams that need a model verdict. This is not a model ranking. It is a reviewer sanity check.

The takeaway

A reviewer you have not tested is a liability. Five diffs, one script, and a free endpoint are enough to find out whether your AI reviewer earns its place in the pipeline. Run the suite once, keep the results, and re-run it when the model or the prompt changes. The cheapest review improvement is the one you measure first.

Batch AI: I Spent 10M Tokens Refactoring a Codebase

Dakota Lin — Fri, 21 Aug 2026 13:21:14 +0000

Ten million tokens sounds infinite. It is not. I burned through a large chunk of it in one weekend. The result? A legacy Python codebase with type annotations on every function. No manual edits. No coffee-fueled marathon. Just a batch script and patience.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Problem

A friend inherited a Django project. Eight years old. Forty-three modules. Zero type hints. The team wanted to add mypy to CI. That requires annotations everywhere. Doing it by hand would take days. Hiring someone would cost real money.

I saw a different path. MonkeyCode offers free model access and a free server. The free tier includes a 10-million-token allowance as of this writing. Check the README for current numbers. I decided to use that allowance for a batch job instead of interactive chat.

The Strategy

Interactive chat burns tokens on conversation overhead. Every back-and-forth repeats context. Batch processing is the opposite. You send one prompt per file. You get one response. No chit-chat. No wasted tokens.

The plan was simple. Walk the repository. Find every Python file. Send it to the model with a strict prompt. Save the annotated version. Run the test suite. Review the diff.

The Script

Here is the core of what I ran. It is deliberately simple. No frameworks. No queues. Just a loop and a rate limiter.

#!/usr/bin/env python3
import os
import time
import json
import urllib.request
from pathlib import Path

API_KEY = os.environ["API_KEY"]
ENDPOINT = os.environ["ENDPOINT"]
MODEL = os.environ["MODEL"]

PROMPT_TEMPLATE = """
Add type annotations to this Python file.
Keep the logic identical.
Only add annotations to function signatures and variables.
Do not change behavior.
Return the complete file.

python
{code}

"""

def annotate(code: str) -> str:
    payload = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": PROMPT_TEMPLATE.format(code=code)}],
        "temperature": 0,
    }).encode()

    req = urllib.request.Request(ENDPOINT, data=payload, headers={
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
    })

    with urllib.request.urlopen(req, timeout=120) as resp:
        data = json.loads(resp.read())
        return data["choices"][0]["message"]["content"]

def main():
    root = Path("project")
    files = sorted(root.rglob("*.py"))
    files = [f for f in files if "migrations" not in str(f)]

    for i, path in enumerate(files):
        print(f"[{i+1}/{len(files)}] {path}")
        code = path.read_text()

        for attempt in range(5):
            try:
                result = annotate(code)
                break
            except Exception as e:
                print(f"  attempt {attempt+1} failed: {e}")
                time.sleep(2 ** attempt)
        else:
            print(f"  SKIPPED after 5 attempts")
            continue

        # Strip markdown fences if the model wrapped the output
        if result.startswith("```

python"):
            result = result.split("

```python", 1)[1].rsplit("```

", 1)[0]

        out_path = path.with_suffix(".annotated.py")
        out_path.write_text(result)
        time.sleep(1)  # polite rate limiting

if __name__ == "__main__":
    main()


```shell

Run it like this:

bash
export API_KEY="your-key"
export ENDPOINT="https://api.monkeycode.example/v1/chat"
export MODEL="model-from-dashboard"
python3 annotate_batch.py


## The Verification Step

A batch job without verification is garbage collection. I did three checks.

First, syntax check every output file.

bash
for f in project/*/.annotated.py; do
python3 -m py_compile "$f" || echo "FAIL: $f"
done


Second, diff the original against the annotated version. The logic should be identical except for annotations.

bash
diff <(sed 's/: [^=,)]//g' original.py) <(sed 's/: [^=,)]//g' annotated.py)




That sed strips annotations. A clean diff means the model did not change behavior.

Third, run the test suite against the annotated files. I swapped them in one by one and ran pytest after each swap.

## The Numbers

Forty-three files. About 12,000 lines of code. The batch took two hours and eleven minutes. Most of that was rate limiting and retries. The actual API time was under thirty minutes.

Token usage was roughly 1.8 million. That is under a fifth of the free allowance. The cost was zero dollars. My friend estimated the manual work at three full days. The batch job cost me a Saturday morning.

## What Went Wrong

Three files came back with broken syntax. The model added annotations that referenced undefined types. One file had a circular import that mypy would reject. The fix was manual. It took fifteen minutes.

Two files were too large for a single request. The prompt exceeded the context window. I split them by function. That worked.

One model response was truncated mid-file. The retry logic caught it. The second attempt returned the full file.

## When Batch AI Makes Sense

Batch processing shines when the task is mechanical and the output is verifiable. Type annotations. Test generation. Error message rewrites. Docstring extraction. These have clear success criteria. You can check the output automatically.

It fails when the task needs judgment. Refactoring architecture. Renaming concepts. Changing behavior. Do not batch those. You will spend more time reviewing than you saved.

## Limitations

The free server is shared. Do not send proprietary code. The model list changes. Quotas change. Check the README before planning a large run. And remember that a 5% failure rate on 43 files means two or three manual fixes. Budget for that.

## Who Should Skip This

If your codebase is under ten files, just annotate by hand. If your code is confidential, use a local model. If you cannot run a test suite after the batch, do not start. Verification is not optional.

## The Takeaway

Free token allowances are usually wasted on chat. The real value is in batch jobs. A mechanical task that would take days can take hours. The math is simple. Try it on a small module first. See if the output passes your tests. Then scale up.

Route by Task, Not by Hype: A Budget-Aware Harness for Trying New Coding Models

Dakota Lin — Thu, 13 Aug 2026 03:44:22 +0000

Every few weeks a new checkpoint drops and the timeline fills up with claims that it's cheaper, smarter, and about to change everything. Some of those claims hold up. Many don't. And even when a model genuinely is better on public leaderboards, that tells you almost nothing about whether it's better on your codebase, your tasks, and your budget.

I wrote previously about building a reproducible harness before wiring any model into your workflow. This article is the sequel nobody asked for but everybody needs: once you have a harness, how do you evaluate a steady stream of new models without spending a steady stream of money?

The answer I keep coming back to is routing by task difficulty: don't run your whole eval suite against every candidate. Tier your tasks, send the cheap ones to cheap models, and reserve expensive runs for the cases that actually discriminate between models.

The problem with "run everything against everything"

If your eval suite has 60 tasks and a new model appears every two weeks, naive evaluation costs scale linearly forever. Worse, most of those runs are wasted signal:

Easy tasks (rename a variable, write a docstring, fix an obvious off-by-one) are solved by almost every current model. Running a frontier-priced model on them tells you nothing.
Medium tasks (implement a small feature against an existing test, refactor across two files) are where models actually diverge.
Hard tasks (multi-file reasoning, subtle concurrency bugs, unfamiliar framework internals) discriminate strongly but are few — and they're where failures are expensive to verify.

So the harness should spend its budget where the signal is.

A concrete artifact: a tiered router in ~80 lines of Python

Here's a minimal, runnable sketch. It assumes your eval tasks are JSON files with a tier field (easy, medium, hard) and a verify command you can execute (a test suite, a diff check, whatever your harness already uses).

# router.py — tiered evaluation router (working sketch, adapt to your harness)
import json, subprocess, sys, time
from pathlib import Path

# Model pool: map a logical name to whatever client you use.
# Keep this boring and swappable — the point is the routing, not the SDK.
MODELS = {
    "cheap":   {"run": lambda prompt: call_model("cheap-model-id", prompt)},
    "mid":     {"run": lambda prompt: call_model("mid-model-id", prompt)},
    "strong":  {"run": lambda prompt: call_model("strong-model-id", prompt)},
}

# Routing policy: which pool each tier starts in, and escalation rules.
POLICY = {
    "easy":   {"start": "cheap", "escalate_to": "mid"},
    "medium": {"start": "mid",    "escalate_to": "strong"},
    "hard":   {"start": "strong", "escalate_to": None},
}

def call_model(model_id: str, prompt: str) -> str:
    # Placeholder — wire in your provider SDK or local server here.
    raise NotImplementedError

def verify(task: dict, patch: str) -> bool:
    """Apply the model's patch and run the task's verification command."""
    workdir = apply_patch(task, patch)          # your existing harness logic
    result = subprocess.run(task["verify"], cwd=workdir, shell=True)
    return result.returncode == 0

def evaluate_task(task: dict) -> dict:
    tier = POLICY[task["tier"]]
    model_key = tier["start"]
    while True:
        patch = MODELS[model_key]["run"](task["prompt"])
        passed = verify(task, patch)
        if passed or tier["escalate_to"] is None:
            return {"task": task["id"], "tier": task["tier"],
                    "solved_by": model_key, "passed": passed}
        model_key = tier["escalate_to"]         # retry on the next tier up

def main(suite_dir: str):
    results = [evaluate_task(json.loads(p.read_text()))
               for p in Path(suite_dir).glob("*.json")]
    for r in results:
        print(json.dumps(r))

if __name__ == "__main__":
    main(sys.argv[1])

The interesting properties:

Escalation, not blanket coverage. Easy tasks only ever touch expensive models if the cheap model fails them — which is itself a signal worth recording.
The solved_by field is the real output. Over time it answers: "for my tasks, does the mid-tier model actually earn its price over the cheap one?"
The pool is swappable. When a new checkpoint appears, you add one entry to MODELS and re-run. The suite, not the hype cycle, decides where it lands.

A note on where the cheap tier comes from

The cheap tier only works if it costs roughly nothing to run at eval volume. Disclosure: This article was prepared as part of MonkeyCode's product outreach. One practical option I've used for this tier is MonkeyCode, which offers free model access and a free server option — that's a reasonable fit for the high-volume, low-stakes end of the pool where you'd otherwise be rationing API calls. If you want to try this routing pattern, that's the tier I'd point it at first; check their docs for what's currently available rather than trusting any article (including this one) about specifics.

Detecting drift: the regression gate

Routing saves money, but there's a subtler failure mode: you swap a model into a tier and silently get worse. Guard against it with a frozen "golden subset" — 10–15 tasks with known-good outcomes — that runs on every pool change:

Check	What it catches	Threshold example
Golden-subset pass rate	Silent capability regression on your real tasks	Must not drop vs. current pool
Escalation rate per tier	A tier model quietly getting weaker (more retries upward)	Alert if +20% week over week
Verify-command runtime	A model producing bloated patches that slow your test loop	Flag outliers for review
Empty/trivial patch rate	Lazy outputs that pass weak verifiers	Any nonzero rate → inspect

The golden subset should include at least two tasks where a previously-hyped model famously failed for you. Those are your canaries.

A decision table for new-model claims

When the next "cheaper and better" checkpoint drops, run this before touching your pool:

Question	If yes	If no
Is there a primary source (provider release notes, official evals)?	Read it; ignore aggregators	Wait — you have nothing to test against
Does the claim cover your task types (not just chat benchmarks)?	Add it as a pool candidate	Note it, don't route to it yet
Does it pass your golden subset at its target tier?	Promote it into the pool	Keep it out; revisit next release
Does it lower escalation rates vs. the current tier holder?	Consider making it the default	Keep the incumbent

Limitations, and who shouldn't do this

Task tiering is judgment, not measurement. Your "medium" tasks might be another team's "hard." Expect to re-tier after the first few runs when the escalation data disagrees with your guesses.
Weak verifiers corrupt everything. If a task can be "passed" by a patch that doesn't really solve it, the router will happily route garbage. The harness is only as good as your verify command.
Free tiers change. Any free model access or free server option — from anyone — can change terms, capacity, or availability. Build the pool abstraction so losing one provider is a config edit, not a rewrite.
Skip this entirely if your eval suite is under ~15 tasks (just run everything; routing overhead exceeds savings), if you don't yet have per-task verification (build that first), or if you're evaluating a model for a safety-critical use where tiered sampling hides rare failures — those need exhaustive runs, not budget routing.

Closing thought

The model release cycle isn't slowing down, and "just try it on your stuff" doesn't scale when your stuff has a real test suite and real costs. Tier your tasks, escalate on failure, gate every pool change on a frozen subset, and let your own data — not the launch-day thread — decide which model earns which tier.

If you build a variant of this, I'd be curious what your escalation rates look like — that number turned out to be the most honest metric in my harness.

A Reproducible Harness for Evaluating New Open Models Before They Touch Your Codebase

Dakota Lin — Mon, 10 Aug 2026 10:44:42 +0000

Every few weeks another open-weight model lands with impressive marketing charts, and the same question hits every team chat: should we switch? The recent wave of releases — MiniMax's new open models among them — makes the question urgent again. But switching your coding workflow on the strength of a vendor benchmark is how you end up debugging regressions you created yourself.

This article is the evaluation harness I reach for instead. It is small, boring, and reproducible — and that is exactly the point.

The actual problem

Vendor leaderboards measure average performance on tasks that are not your tasks. What you need to know is narrower:

Does the model handle your repo's idioms (your ORM, your test style, your naming)?
Does it fail loudly (obvious garbage) or quietly (plausible, wrong code)?
Does the failure pattern change under long context?

You can answer all three in an afternoon with ~60 lines of shell and a fixed task set.

Step 1: Build a frozen task set from your own git history

Pull real, already-solved problems from your repo so you have ground truth:

# Collect 10 small, self-contained commits with tests
mkdir -p eval/tasks
for sha in $(git log --oneline --since="90 days ago" --format="%H" | head -40); do
  files=$(git show --name-only --format="" $sha | wc -l)
  if [ "$files" -le 3 ] && git show $sha | grep -q "test\|spec"; then
    git show $sha > eval/tasks/$sha.patch
  fi
done
ls eval/tasks | head -10  # keep exactly 10, delete the rest

For each task, write the problem statement only (the diff minus the solution), so the model sees the same prompt a junior dev would.

Step 2: The harness

# eval/run.py — deliberately minimal, no framework
import json, subprocess, time, pathlib

def score_task(model_cmd: str, task_dir: pathlib.Path) -> dict:
    prompt = (task_dir / "prompt.md").read_text()
    start = time.time()
    out = subprocess.run(model_cmd.split() + [prompt],
                         capture_output=True, text=True, timeout=300)
    patch = task_dir / "candidate.patch"
    patch.write_text(out.stdout)
    # Ground truth check: apply patch in a throwaway worktree, run tests
    applied = subprocess.run(["git", "apply", "--check", str(patch)],
                             capture_output=True)
    tests = subprocess.run(["pytest", "-x", "-q"], capture_output=True) \
            if applied.returncode == 0 else None
    return {
        "task": task_dir.name,
        "latency_s": round(time.time() - start, 1),
        "patch_applies": applied.returncode == 0,
        "tests_pass": tests.returncode == 0 if tests else False,
        "silent_wrong": applied.returncode == 0 and tests and tests.returncode != 0,
    }

The silent_wrong column is the metric that matters most. A model that produces patches that apply cleanly but fail tests is worse than one that errors out, because it costs you review time instead of compute time.

Step 3: Where the compute comes from

This is the part that used to be the blocker: honest evaluation means running candidate models repeatedly, and API bills punish curiosity. Lately I've been running this loop through MonkeyCode, which currently offers free access to a set of models and a free server option — useful here precisely because evaluation wants many cheap runs, not one expensive one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I genuinely appreciate, and why it fits this article's theme, is the open posture: free model access plus a free server lowers the barrier for exactly the kind of independent, reproducible verification that the open-source ecosystem runs on. When a new open model drops, the healthiest response isn't hype or dismissal — it's a hundred developers running their own harnesses and publishing results. Tooling that makes that cheap is participating in that spirit, not just marketing to it.

Practical setup notes:

Pin the model version in your harness config. "The new MiniMax model" is not a version; results you can't re-run are anecdotes.
Keep temperature fixed (I use 0 for patch tasks) so runs are comparable across models.
Store raw outputs, not just scores — you'll want to read the silent_wrong cases by hand.

A decision table, not a verdict

Signal	Keep current model	Trial the new one
`tests_pass` rate	within ~5%	clearly higher on your tasks
`silent_wrong` count	any increase	lower or equal
Long-context degradation	—	test with your biggest real diff, not a synthetic one
Cost/latency for your volume	—	measure it; don't read it off a landing page

Limitations, honestly

Ten tasks is a smoke test, not a benchmark. Treat a passing result as "worth a two-week trial," not "migrate everything."
My git-history mining skews toward small bugfixes; it won't tell you how a model handles greenfield design.
Free tiers change. The MonkeyCode free model access and free server are what's available as I write this — verify current terms before building a workflow on top of them, and have a fallback (local quantized models via llama.cpp work fine for this harness).
Who should not use this approach: teams with compliance constraints on sending code to third-party endpoints. Run everything against local models only.

The takeaway

The open-model ecosystem moves fast enough that "which model is best" has a shelf life of weeks. The durable asset isn't a model choice — it's a harness that lets you re-ask the question cheaply, on your own code, whenever the next release drops. If you build one, publish your task set and results; that kind of open, reproducible evaluation is the open-source spirit applied to the model era.

If you want a low-friction place to start running a harness like this, MonkeyCode's free model access and free server are one option worth a look — but the harness matters more than the host.

Don't Wire a Coding Model Into Your Workflow Until It Passes Your Own Harness

Dakota Lin — Mon, 10 Aug 2026 08:16:24 +0000

Liquid syntax error: Unknown tag 'endraw'

DEV Community: Dakota Lin

Stop Timing the Happy Path

The lab I actually rerun

The prompt that starts the argument

The graph I kept

Commands after the model shuts up

Where the free server actually sits

Limitations, said plainly

What I keep in the PR

The Graph That Killed a "40% Faster" Patch

Why one number is a lie

The workflow I now use

Decision rules I stole from myself

The graph I kept

Limitations

A cheap place to run this

Profile First, Guess Later: A Three-Step Performance Dig

Step 1: Profile before you touch a single line

Step 2: Build a minimal experiment

Step 3: Keep the graph

Why the free server mattered

Limitations you should know

The deeper lesson

Every AI Patch Deserves a Baseline

Why the baseline matters

Sample output

The graph you keep

The ledger

When the benchmark fails

Limitations

From Raw Lint to Actionable Report: A Free AI-Assisted Pipeline

Why Bother With Three Tools Instead of One

The Example File: order_service.py

The Runner Script: run_checks.py

What a Good Model Summary Looks Like

A Decision Table for Your Own Repo

Limitations You Should Know

Who Should Not Use This Workflow

Making It Repeatable

The 7.6x Speedup Hiding in Your HTTP Client

The scenario

What the timing said

Fix one: reuse the connection

Fix two: bounded concurrency

Decision table

Timeouts and retries

Limits and who should skip this

The takeaway

My AI Refactor Was Slow. The Model Wasn't Guilty.

Rate Your AI Reviewer: A Five-Case Regression Suite

The five bug patterns

The suite format

The runner

How I ran it

Decision table

Limitations

The takeaway

Batch AI: I Spent 10M Tokens Refactoring a Codebase

The Problem

The Strategy

The Script

Route by Task, Not by Hype: A Budget-Aware Harness for Trying New Coding Models

The problem with "run everything against everything"

A concrete artifact: a tiered router in ~80 lines of Python

A note on where the cheap tier comes from

Detecting drift: the regression gate

A decision table for new-model claims

Limitations, and who shouldn't do this

Closing thought

A Reproducible Harness for Evaluating New Open Models Before They Touch Your Codebase

The actual problem

Step 1: Build a frozen task set from your own git history

Step 2: The harness

Step 3: Where the compute comes from

A decision table, not a verdict

Limitations, honestly

The takeaway

Don't Wire a Coding Model Into Your Workflow Until It Passes Your Own Harness

The Example File: `order_service.py`

The Runner Script: `run_checks.py`