DEV Community: Sam Li

Your Free AI Token Allowance Is a Budget, Not a Gift

Sam Li — Sat, 29 Aug 2026 10:11:11 +0000

A token allowance is a budget with real economics, and teams that treat it that way finish whole sprints on the free tier while others run dry mid-week. This article covers a two-week experiment in which I metered every request that went through the free models and the free server of an open-source coding assistant called MonkeyCode, because the workflow is the point, not the hype. The outcome was a small estimation script, two workflow rules, and a burn rate per task that predicted the week's exhaustion before the week ended.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

At the time of writing, MonkeyCode offers free model access, a free hosted server option, and a token allowance that the project README lists at ten million tokens. Those terms have changed before and will change again, so check the repository before you build any plan on the exact figure. What mattered for this experiment was not the precise number but the habit of recording a consumption estimate for every task, the way you log cloud spend before the bill arrives.

Context resends are the silent tax

The leak that most token budgets ignore is silent context resending. You ask a coding agent to review one changed file, and the runtime quietly re-reads the surrounding modules to rebuild state it already has in memory. Over a week of small reviews, that invisible resending can consume more tokens than the actual changes you asked for.

The remedy is not a bigger allowance; it is a short state file that lets the next task start from a summary instead of a full re-read. After every task, write at most five lines to STATE.md covering what changed, what is still failing, and what comes next, and the agent should read that file before touching the tree again. The discipline feels over-engineered on day one, and it pays for itself by day three.

The estimator that made the budget visible

The estimator below is deliberately crude, because precision is not the goal; the goal is to make every task produce a number that gets logged somewhere. It assumes roughly four characters per token, which is a common rule of thumb, and it records the exit code, the elapsed time, and separate prompt and completion estimates into one JSONL file. That file becomes the audit trail for every decision you make about the agent's budget.

#!/usr/bin/env python3
# meter.py - rough per-task token accounting for coding-agent batches.
# Estimate: ~4 characters per token. Real API usage will vary.
import json
import subprocess
import sys
import time

CHARS_PER_TOKEN = 4.0
LOG_PATH = 'token-budget.ndjson'


def estimate_tokens(text: str) -> int:
    return max(1, int(len(text) / CHARS_PER_TOKEN))


def run_task(label: str, command: str) -> dict:
    started_at = time.strftime('%Y-%m-%dT%H:%M:%S')
    start = time.monotonic()
    result = subprocess.run(command, shell=True, capture_output=True, text=True)
    elapsed = time.monotonic() - start
    entry = {
        'task': label,
        'started_at': started_at,
        'elapsed_seconds': round(elapsed, 2),
        'exit_code': result.returncode,
        'estimated_prompt_tokens': estimate_tokens(len(result.args)),
        'estimated_completion_tokens': estimate_tokens(
            len(result.stdout) + len(result.stderr)
        ),
    }
    entry['estimated_total_tokens'] = (
        entry['estimated_prompt_tokens'] + entry['estimated_completion_tokens']
    )
    with open(LOG_PATH, 'a') as fh:
        fh.write(json.dumps(entry) + chr(10))
    return entry


if __name__ == '__main__':
    label = sys.argv[1]
    command = sys.argv[2]
    print(json.dumps(run_task(label, command), indent=2))

python3 meter.py review-auth 'your-agent-cli run --task review-auth'
python3 meter.py diff-types  'your-agent-cli run --task diff-types'
python3 meter.py add-logger 'your-agent-cli run --task add-logger'

I ran that pattern twice a day for two weeks, and the first log already showed something predictable and still annoying: the assistant kept restating context it had already been given. The state file from the previous section fixed that leak within a couple of days, and the daily totals dropped accordingly. The second discovery was retries, the quietest kind of waste because retries look like progress.

Two failures in that period, an unstubbed network call and a missing fixture, each produced several repeated attempts, and every attempt re-sent the same large context with a slightly different prompt. A one-line rule stopped that category: never repeat a command that produced the same failure twice, because the next move must be a change in approach, not a louder echo. That rule alone cut the most expensive block of the whole experiment.

A handoff guide for free models

The decision table below is the handoff guide that came out of those two weeks, and it is calibrated to free models rather than to an imaginary infinite quota. Small mechanical fixes are always fine, mid-size tasks are fine with scripted verification, but anything that approaches a cross-cutting refactor should go back to a human reviewer. The threshold that mattered most was not model quality but the token price of a wrong guess.

Task shape	Rough token class	Verdict for free models
Typo, lint, and comment fixes	under 5k	always fine
Dependency bump with a green test	5k to 20k	fine, verify by script
Test generation for one small module	20k to 60k	fine, review every diff
Cross-cutting refactor or API redesign	100k and up	hand to a human

When the free tier should not be used

This workflow is not for every team. A repository containing secrets, or an organization whose compliance rules require data to stay inside its own boundary, should not point a hosted free server at its codebase even when the meter looks clean. Anyone whose tasks regularly exceed what a session can hold should look elsewhere, because no estimation script fixes a context window that is simply too small.

The number that predicted the week's budget was not the token total; it was the burn rate per task, and once that rate became visible the experiment stopped being about any single tool and started being about the workflow around it. Meter your next sprint the same way, and if you try the script against the MonkeyCode free server, check the README first because the terms keep moving. The most useful comment you can leave is your own burn rate per task, because real numbers beat any endorsement.

48-Hour Field Notes: I Stopped Reading AI Review Comments and Started Scoring Them

Sam Li — Fri, 28 Aug 2026 03:39:17 +0000

Liquid syntax error: Unknown tag 'endraw'

48-Hour Field Notes: A Free AI Server as My PR Reviewer

Sam Li — Wed, 26 Aug 2026 11:14:21 +0000

Friday, 16:47. CI turned red on a one-line change that a human reviewer had already approved. The bug wasn't subtle — a null check on the wrong variable — and it sat in the diff for three days.

That PR is why I stopped asking whether AI coding servers can write code. The harder question is whether they can review it. The DEV front page has been circling this all week: AI promoted every developer to reviewer, and nobody tested the reviewer. So I spent 48 hours testing mine — a free AI coding server, pointed at a real repo, acting as the reviewer on call.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The server I used was MonkeyCode's free tier: free model access with a 10M token allowance, plus a free server option. I'm not going to quote a leaderboard score, because one 48-hour run isn't a benchmark. I kept field notes instead. Here's what I tried, what broke, and what I'd repeat.

The setup

I built a small Node.js service with five seeded bugs. Not clever bugs — the kind that survive review: a wrong variable in a null check, an off-by-one in pagination, an unhandled promise rejection, a SQL query filtering on the wrong column, and a race condition in a cache write. Each one had a known file:line and a one-line fix. That file was my ground truth.

Then I wrote a drill. It sends a diff to the server's endpoint, records latency and response size, and saves the review. Run it five times, score the reviews against the ground truth, and you get a picture of what the server actually catches.

#!/usr/bin/env bash
# review_drill.sh — send the same diff to an AI coding server N times
set -euo pipefail

ENDPOINT="${ENDPOINT:?set ENDPOINT to your server}"
MODEL="${MODEL:?set MODEL to the model id}"
DIFF="${1:?pass a diff file}"

read -r -d '' PROMPT <<'EOF' || true
Review this pull request diff. List concrete bugs only.
For each: file:line, why it breaks, one-line fix.
Ignore style and naming. If nothing is wrong, say so.
EOF

for i in 1 2 3 4 5; do
  start=$(date +%s%N)
  curl -s "$ENDPOINT" \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg m "$MODEL" --arg p "$PROMPT" \
      --rawfile d "$DIFF" \
      '{model:$m, messages:[{role:"user", content:($p+"\n\n"+$d)}], temperature:0}')" \
    > "run_${i}.json"
  end=$(date +%s%N)
  ms=$(( (end - start) / 1000000 ))
  jq -r '.choices[0].message.content' "run_${i}.json" > "run_${i}.md"
  printf 'run %s: %s ms, %s words\n' "$i" "$ms" "$(wc -w < "run_${i}.md")"
done

The script assumes an OpenAI-compatible chat endpoint; if your server speaks something else, swap the payload. The important part isn't the curl — it's the repetition. One review is an anecdote. Five reviews of the same diff give you a spread.

Scoring is a two-liner against your ground truth:

# score a run: does the review mention each seeded bug's file:line?
while read -r bug; do
  grep -q "$bug" run_1.md && echo "HIT  $bug" || echo "MISS $bug"
done < ground_truth.txt

What I tried

Three scenarios, in order.

First, a small diff: one file, fifteen lines changed. This is the easy case, and it should be the floor. The drill measured how fast the server responded and whether it found the seeded bug inside the changed lines.

Second, a large refactor: six files, a moved function, a renamed variable. This is where reviewers earn their keep. The bug was hidden in a file that looked untouched.

Third, a diff with the test suite attached. I wanted to know whether the server would use the tests as evidence or ignore them and pattern-match on the diff alone.

Sample output — the format the harness records, not a product benchmark:

run 1: small diff     -> 2/5 hits, 1 false positive, 41s
run 2: large refactor -> 1/5 hits, 0 false positives, 88s
run 3: with tests     -> 3/5 hits, 2 false positives, 63s

The numbers are specific to my repo and my afternoon. The shape is what matters: the small diff was fast and noisy, the refactor was slow and quiet, and the tests helped — but only when the server actually read them.

What broke

Three things broke, and none of them were the model being "dumb."

The harness broke first. My first version piped the response through jq without checking for an error object. The server returned a 200 with an empty message, jq printed null, and the script kept going. The review file was empty and the run counted as a success. Lesson: validate the response body before you score it.

The context window broke second. The refactor diff, plus the prompt, pushed past what the server could take in one pass. The response came back, but it only covered the first two files. The seeded bug was in the fifth. The review wasn't wrong — it was blind. You can't fix that with a better prompt; you have to split the diff.

The third break was the quiet one: false confidence. On the run with tests attached, the server found three of five bugs and invented two more. The invented ones sounded plausible. Without a ground-truth file, I would have merged its advice. That's the real risk of a free AI reviewer — not that it's useless, but that it's convincing.

What I'd repeat

I'd repeat the ground truth. Without a seeded bug list, you're grading a reviewer on vibes. The drill is only as honest as the file you score against.

I'd repeat the five-run spread. The first run of any diff is the worst — cold start, prompt quirks, bad luck. By run three, the answers stabilized. If you only run the drill once, you're measuring noise.

I'd repeat the latency log, because it caught the one thing the reviews couldn't: the server got slower as the session went on, and the harness couldn't tell me whether that was the server, my network, or the model. Instrument the client, not the server, and you at least know which side of the wire to blame.

Who should not use this

The drill tests review quality on seeded bugs in one small codebase. It doesn't test code generation, refactoring, or long-running agent tasks. It doesn't test security review, and it doesn't test domain expertise.

Don't send a diff with secrets, customer data, or proprietary code to a free hosted server. Free tiers are for code you'd paste into a public gist. If that's not your code, keep the review local.

Don't use this for compliance-sensitive review. A general model doesn't know your auth model, your data retention rules, or your team's definition of "done." It can catch a null check. It cannot sign off on a control.

And if you need a review in under ten seconds, every time, a free server is the wrong tool. The drill showed me variance, not guarantees.

The takeaway

The trend this week is that AI promoted every developer to reviewer. Nobody tested the reviewer. I spent 48 hours testing mine against a file of seeded bugs, and the three failures taught me more than the hits. The harness is above; the ground truth is your own bug history. Run it against whatever free server you already have access to — MonkeyCode's free tier is where I ran mine — and let the next 48 hours tell you whether the reviewer earns its place in your repo.

Your Git History Knows Which AI Coding Server You Should Pick

Sam Li — Tue, 25 Aug 2026 09:02:24 +0000

The argument started in the team channel at 9:40 on a Tuesday. One engineer pasted a link to a free AI coding server and called it good enough. Another replied that self-hosting is the only responsible choice. By lunch, nobody had mentioned the actual codebase.

That's the pattern I keep seeing. Teams treat the free-versus-self-hosted question as a matter of principle. It isn't. It's a measurement problem.

Your repo has a shape. That shape decides the answer before you ever open a pricing page.

Three dimensions that decide the answer

Token appetite. How many tokens does your average diff consume, plus the instructions, plus the back-and-forth? Context churn. How often does a fresh conversation have to re-learn your codebase? Latency tolerance. Does a slow response break your flow, or just delay a batch job?

A team shipping small, focused pull requests has a different appetite than a team doing weekly mega-refactors. The same free tier can feel unusable in an interactive session and invisible in an overnight test run. Most comparisons skip this part because it requires actual work.

A fit test you can run from your terminal

Here's the reproducible part. The script below reads your git history and estimates the token appetite of your last N commits. It calls no model and needs no API key. It uses one heuristic: roughly four characters per token, a conservative average for mixed code, prose, and symbols.

#!/usr/bin/env bash
# fitcheck.sh — estimate your repo's token appetite from git history
set -euo pipefail

N="${1:-50}"
REPO="${2:-.}"
cd "$REPO"

echo "Analyzing last $N commits in $(basename "$(pwd)")"
echo "---"

total_tokens=0
max_tokens=0
declare -A file_hits

for i in $(seq 0 $((N - 1))); do
  diff_text=$(git show --format="" --unified=3 "HEAD~$i" 2>/dev/null || true)
  chars=$(printf '%s' "$diff_text" | wc -c)
  tokens=$((chars / 4))
  total_tokens=$((total_tokens + tokens))
  if (( tokens > max_tokens )); then
    max_tokens=$tokens
  fi

  while IFS= read -r f; do
    file_hits["$f"]=$(( ${file_hits["$f"]:-0} + 1 ))
  done < <(git show --name-only --format="" "HEAD~$i" 2>/dev/null || true)
done

avg_tokens=$((total_tokens / N))
echo "Average diff tokens per commit: $avg_tokens"
echo "Largest diff tokens: $max_tokens"

repeated=0
for hits in "${file_hits[@]}"; do
  if (( hits > 1 )); then
    repeated=$((repeated + 1))
  fi
done
echo "Files touched more than once: $repeated"
echo "Projected monthly burn (20 commits/day): $((avg_tokens * 20 * 22)) tokens"

Run it like this:

chmod +x fitcheck.sh
./fitcheck.sh 50 /path/to/your/repo

The output is an estimate, not a bill. Tokenizers vary by model, and real prompts add overhead this script deliberately ignores. But the relative shape is what matters. A repo projecting 2 million tokens a month and a repo projecting 200 million are not in the same conversation.

The 4-characters-per-token rule is deliberately conservative. Code is denser than prose, and a unified diff carries context lines that a real assistant would also see. If anything, the script undercounts what a full prompt with instructions and file excerpts would cost. That bias is fine for a fit test. You want the order of magnitude, not the invoice.

Reading the numbers

If your average diff stays under roughly 2,000 tokens and your monthly projection fits inside the free allocation, the free tier is structurally sufficient. The only remaining question is latency, and that needs a live test. If your projection blows past the allocation by an order of magnitude, no amount of enthusiasm fixes the math. You need self-hosting, a paid tier, or a serious change in how you chunk work. And if the same files keep reappearing in commit after commit, you're a candidate for context reuse — but only if the server you pick actually supports it. That's a feature question, not a pricing question.

Fit signal	Verdict	Action
Projection under 60% of free allocation	Good fit	Run a 30-minute live latency test
Projection between 60% and 100%	Borderline	Watch real usage for two weeks
Projection over 100%	Bad fit	Self-host, pay, or split work into smaller units
Same files touched repeatedly	Context reuse candidate	Verify the server caches context before relying on it
Strict data residency or audit needs	Bad fit regardless of math	Self-host or use a compliant paid option

The case that keeps coming up

MonkeyCode keeps showing up in my feeds, so I ran the framework on it. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project is open source, and it offers two things that matter here: free model access with a 10-million-token allocation, and a free server option so you can try it without standing up your own inference host. I'm not going to recite the marketing numbers back at you. Run the script instead.

Take a worked example. A mid-sized service repo with 7,000-token average diffs and 20 commits a day projects to roughly 3.1 million tokens per month. That fits inside a 10-million allocation with room to spare. The verdict is a trial, not a migration. A monorepo with heavy generated code, averaging 90,000 tokens per diff, projects past 40 million. The free tier runs out before the work does. That's not a product failure. It's a fit failure, and the script catches it in under a minute.

Who should not use the free option

Teams with data-residency requirements, because your code crosses a network boundary. Anyone generating code for regulated systems where provenance and audit trails matter. And repos so large that the context window alone would consume the allocation before the first useful answer. Also, if you're wiring an AI server into a CI pipeline, a free server without an SLA is a risk you're pricing, not a cost you're avoiding.

The free-versus-self-hosted debate is usually a proxy for something else. Nobody wants to be the person who picked the wrong tool. The fix is to stop picking and start measuring. Your git history has been keeping score for years. Read it.

Clone the project, run fitcheck.sh against your own repo, and let the numbers pick your side.

The Fit Score: When a Free AI Coding Server Earns Its Place in Your Repo

Sam Li — Tue, 25 Aug 2026 04:46:54 +0000

Picture a team that wired a free AI coding server into its review loop. Two weeks later, the sprint stalls. The quota resets on a Tuesday, mid-refactor, and every "just ask the assistant" habit the team built turns into a queue of blocked tasks. Nobody read the reset policy, because nobody expected a free tier to become load-bearing.

Free is a pricing model, not an architecture decision.

The mistake wasn't the tool. It was treating fit as a property of the product when it's actually a property of the pairing between product and repo. Every week brings another coding model, another free tier, another leaderboard telling you the last one was wrong. The useful question is narrower: does this particular server fit this particular codebase, right now?

I use three axes to answer it, and one script to make the answer explicit.

Data sensitivity comes first. If your codebase contains customer data, internal credentials, or unreleased product logic, every prompt you send to a managed endpoint is a data-transfer decision. Some teams can accept that. Regulated ones cannot, no matter how good the model is.

Latency tolerance is second. A self-hosted model on modest hardware answers in seconds, but it competes with your build for CPU and memory. A managed server shifts that cost elsewhere and adds network round-trips and shared-queue variance. The real question is whether your workflow can absorb a slow tail. Interactive completion, probably yes. A CI gate that blocks merges, probably not.

Workload shape is third. Free tiers are designed for spiky, low-volume use. If your team produces a steady, predictable stream of requests, the free allowance becomes a ceiling you will hit at the worst moment. If your usage is genuinely intermittent, the ceiling rarely matters.

Here is the script I use to turn those axes into a number. It is deliberately crude — the weights encode my own priorities, and data risk dominates because it is the one failure you cannot roll back.

#!/usr/bin/env python3
"""fit_score.py — decide whether a free managed AI coding server fits your repo.

Example:
  python3 fit_score.py --data-sensitivity 4 --latency-tolerance 2 \
      --workload-shape steady --monthly-tokens 4 --ci-integration 3
"""

import argparse

WEIGHTS = {
    "data_sensitivity": 0.30,   # 1 = public code, 5 = regulated data
    "latency_tolerance": 0.25,  # 1 = CI gate, 5 = interactive only
    "workload_shape": 0.20,     # spiky vs steady request volume
    "monthly_tokens": 0.15,     # 1 = tiny usage, 5 = heavy usage
    "ci_integration": 0.10,     # 1 = no CI, 5 = merge-blocking gate
}

def main() -> None:
    p = argparse.ArgumentParser()
    p.add_argument("--data-sensitivity", type=int, choices=range(1, 6), default=2)
    p.add_argument("--latency-tolerance", type=int, choices=range(1, 6), default=3)
    p.add_argument("--workload-shape", choices=["spiky", "steady"], default="spiky")
    p.add_argument("--monthly-tokens", type=int, choices=range(1, 6), default=3)
    p.add_argument("--ci-integration", type=int, choices=range(1, 6), default=2)
    args = p.parse_args()

    workload = 5 if args.workload_shape == "steady" else 2
    raw = (
        WEIGHTS["data_sensitivity"] * args.data_sensitivity
        + WEIGHTS["latency_tolerance"] * args.latency_tolerance
        + WEIGHTS["workload_shape"] * workload
        + WEIGHTS["monthly_tokens"] * args.monthly_tokens
        + WEIGHTS["ci_integration"] * args.ci_integration
    )
    score = round(raw, 2)

    if score <= 2.0:
        verdict = "Low risk. Wire it in, monitor quota, move on."
    elif score <= 3.2:
        verdict = "Borderline. Run a two-week staged trial with a hard rollback."
    else:
        verdict = "Do not default to free. Self-host, pay for a tier, or skip AI assistance."

    print(f"Fit score: {score}/5")
    print(f"Verdict: {verdict}")

if __name__ == "__main__":
    main()

Change the weights to match your constraints. The point is to make the decision explicit instead of vibes-based.

Once the score says "low risk," test the server before you trust it. This probe measures time-to-first-token and total completion time against any OpenAI-compatible endpoint. It is an example harness — adjust the headers and path to the provider you are actually testing.

#!/usr/bin/env bash
# probe.sh — measure TTFT and total time for an OpenAI-compatible endpoint.
# Usage: ENDPOINT=... MODEL=... API_KEY=... ./probe.sh

set -euo pipefail
: "${ENDPOINT:?set ENDPOINT}"
: "${MODEL:?set MODEL}"
: "${API_KEY:?set API_KEY}"

PROMPT='Write a Python function that parses an nginx access log line and returns a dict. Keep it dependency-free.'

curl -sS -w '\nTTFT: %{time_starttransfer}s\nTOTAL: %{time_total}s\n' \
  -H "Authorization: Bearer ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{\"model\": \"${MODEL}\", \"messages\": [{\"role\": \"user\", \"content\": \"${PROMPT}\"}], \"max_tokens\": 300}" \
  "${ENDPOINT}/v1/chat/completions"

Run it ten times, at different hours, and record the tail, not the median. A server that is fast at 2 p.m. and slow at 4 p.m. is still slow at 4 p.m. if that is when your team works.

This is where MonkeyCode enters the evaluation. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that fits the pattern above: it offers free model access and a free server option, and the project currently advertises a 10-million-token free allowance. I have not independently verified the quota, the hardware behind the server, or the permanence of the offer — which is exactly why you should run the probe before building anything on it.

On paper, MonkeyCode targets the quadrant the framework calls low-risk: spiky, low-volume, non-regulated teams that want to evaluate coding models without standing up a GPU box. That is a real niche. The free server removes the setup barrier, and the token allowance covers exploratory use. The same caveats apply as to any free tier: quotas reset, latency varies, and a product that is free today can change its terms tomorrow. Treat it as an evaluation vehicle, not infrastructure.

Who should not use this approach? Teams with regulated data, because a managed endpoint is still an external endpoint. Teams with steady high-volume workloads, because a free allowance is a ceiling, not a plan. Teams that need offline guarantees, because free does not mean air-gapped. And teams that already have a working self-hosted setup — if your local model answers fast enough and your data never leaves the building, a free server is a downgrade dressed as a discount.

If you are already running a probe like the one above, the MonkeyCode repo is worth a look as a data point, not a promise. Run the fit score, run the probe, and let the numbers argue with your assumptions. That is the whole method.

Free AI Coding Server or Self-Hosted? Score the Tradeoffs Instead of Arguing

Sam Li — Sat, 22 Aug 2026 14:39:04 +0000

Picture two teams starting from the same commit this month. Team A signs up for a managed free tier, ships a feature in a day, then hits a quota wall during the demo. Team B rents a GPU box, spends two weeks wiring auth, model routing, and disk cleanup, and ships nothing yet. Both made rational choices. Both optimized the wrong constraint.

The mistake is treating "free" and "self-hosted" as properties of a tool. They are not properties. They are constraints you trade against each other. A free managed server trades data locality and quota headroom for zero operations. A self-hosted box trades your engineering hours for control. The real question is not which option is better. It is which constraint your team can absorb without breaking.

So stop arguing. Score it.

Seven dimensions, one weighted sum

Score seven dimensions. Data sensitivity: how much of your code can legally leave your network. Ops capacity: who owns upgrades, restarts, and disk fills. Latency budget: interactive coding punishes round trips. Cost ceiling: hard zero, or a soft budget that can absorb surprises.

Usage shape: steady trickle or spiky bursts, because quotas punish spikes. Lock-in tolerance: how painful switching is later. Feature velocity: how fast you want new models without touching infrastructure. Each dimension gets a weight from 1 to 5 for how much it matters to you, and each option gets a score from 1 to 5 for how well it satisfies that dimension. Multiply, sum, compare.

That is the whole framework. Seven numbers, one spreadsheet. The arguments disappear once the weights are on the table, because most disagreements are really disagreements about weights, not about tools.

The artifact: a scoring script

Here is the decision script I use. It is deliberately boring: no dependencies, no model calls, just a weighted sum.

#!/usr/bin/env python3
"""decide.py — score AI coding tooling options against your constraints."""
import json, sys

OPTIONS = {
    "managed_free": {
        "data_sensitivity": 1, "ops_capacity": 5, "latency_budget": 3,
        "cost_ceiling": 5, "usage_shape": 2, "lock_in_tolerance": 3,
        "feature_velocity": 5,
    },
    "self_hosted": {
        "data_sensitivity": 5, "ops_capacity": 1, "latency_budget": 5,
        "cost_ceiling": 2, "usage_shape": 4, "lock_in_tolerance": 5,
        "feature_velocity": 2,
    },
    "paid_managed": {
        "data_sensitivity": 2, "ops_capacity": 4, "latency_budget": 4,
        "cost_ceiling": 1, "usage_shape": 4, "lock_in_tolerance": 3,
        "feature_velocity": 4,
    },
}

WEIGHTS = {
    "data_sensitivity": 5, "ops_capacity": 3, "latency_budget": 4,
    "cost_ceiling": 3, "usage_shape": 2, "lock_in_tolerance": 2,
    "feature_velocity": 3,
}

def main():
    if len(sys.argv) > 1:
        with open(sys.argv[1]) as f:
            WEIGHTS.update(json.load(f))
    results = []
    for name, dims in OPTIONS.items():
        total = sum(WEIGHTS.get(k, 0) * v for k, v in dims.items())
        results.append((name, total))
    for name, total in sorted(results, key=lambda x: -x[1]):
        print(f"{name:14} {total}")

if __name__ == "__main__":
    main()

Save it, run it, and you get a ranking. Then override the weights with a JSON file that reflects your actual pain:

$ cat weights.json
{"cost_ceiling": 5, "ops_capacity": 4, "data_sensitivity": 2, "feature_velocity": 4, "usage_shape": 1}
$ python3 decide.py weights.json
managed_free    87
paid_managed    67
self_hosted     66

A startup with no ops and a hard zero budget gets a clear answer: managed free. Now flip the weights for a regulated team with strong infrastructure:

$ cat weights.json
{"data_sensitivity": 5, "latency_budget": 5, "cost_ceiling": 2, "usage_shape": 4}
$ python3 decide.py weights.json
self_hosted     89
paid_managed    78
managed_free    74

Same script, opposite answer. That is the point. The framework does not tell you what to pick. It tells you which constraint you are actually optimizing, so the decision survives contact with reality.

Where MonkeyCode fits in the managed-free row

MonkeyCode is a concrete instance of the managed-free row: free model access plus a free server option, so you do not provision or maintain anything to run a real experiment. As of this writing, the operator advertises a 10 million token allowance on the free path. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The interesting part is not the token count. It is that the free server collapses the cost of trying to nearly zero. When the managed-free row wins on paper, the cheapest way to confirm it is a two-day spike with your real code, not a benchmark. Benchmarks measure the model. A spike measures the workflow: does the tool fit your repo, your review process, your tolerance for weird output?

That is also why the open source part matters. You can read what the server actually does before you trust it with a repository. Trust is not a feature flag; it is something you verify.

Who should not use this approach

Three groups should ignore the managed-free row entirely. First, teams whose code cannot leave their network: regulated finance, classified work, pre-IPO secrets. No score overrides a legal constraint. Second, teams whose usage is predictably spiky and whose demos cannot fail: a quota wall at the wrong moment is worse than a bill. Third, teams that refuse to change workflow: a new tool is a tax on muscle memory, and if nobody will pay that tax, the tool is dead on arrival.

And who should not self-host? Anyone without a named owner for upgrades, security patches, and disk fills. A model server is a service you now operate. If your team cannot staff that, self-hosting is a hobby, not an architecture.

The closing move

Run the script. Change the weights until the result matches your actual pain, not your feed's opinion. Then pick the option that loses the least. If you want to test the managed-free row cheaply, MonkeyCode's free server is a reasonable place to run that two-day spike — and because it is open source, you can check what it does first. The decision is yours; the script just makes it honest.

The Free Tier Isn't Free If It Doesn't Fit

Sam Li — Fri, 21 Aug 2026 13:11:39 +0000

A backend team picks the cheapest AI coding option, hits a hard token cap at 3 PM on release day, and spends the evening pasting stack traces into a free chat window. Another team buys a GPU, self-hosts an open model, and loses two weekends to driver hell before the first useful suggestion appears. Both teams optimized for price per token. Both ignored fit per workflow.

The AI badge on a model card tells you what the vendor measured, not what your workflow needs. The same logic applies to access tiers. A free tier is not a price; it is a constraint set, and constraints are only cheap when they match your usage shape. This is a decision framework for choosing between free hosted access, paid APIs, and self-hosted models, plus a script that turns your habits into a score.

Four dimensions decide the fit.

Volume shape comes first. Bursty users generate a lot in short windows — a prototype weekend, a refactor sprint, a learning session. Steady users feed the model daily, like a CI job. Token caps punish the steady user and barely touch the bursty one. If your monthly burn is under eight million tokens and irregular, a free allowance is a feature, not a compromise.

Data gravity is second. Can your code leave your machine? Proprietary algorithms, regulated data, and client contracts often answer no. That single constraint eliminates every hosted option, free or paid, and forces self-hosting. For everyone else, the question is about comfort, not compliance.

Ops budget is third, and it is the one people forget. Self-hosting shifts cost from money to time. Someone must patch the box, watch the GPU temperature, and restart the service at 2 AM. If your team has zero infrastructure hours to spare, the "free" server is the most expensive option you can choose.

Failure tolerance is fourth. What happens when the provider rate-limits you mid-sprint? If the answer is "we wait," hosted is fine. If the answer is "we miss a deadline," you need a fallback or a local model.

The artifact: a fit-score script.

#!/usr/bin/env python3
'''Estimate which AI coding access model fits your workflow.

Inputs: monthly token estimate, data sensitivity flag,
latency budget in ms, ops hours available per month,
and whether usage is steady (daily) or bursty.
'''

def fit_score(tokens_per_month, sensitive, latency_ms, ops_hours, steady):
    score = {'free_hosted': 0, 'paid_api': 0, 'self_hosted': 0}

    if tokens_per_month < 8_000_000 and not steady:
        score['free_hosted'] += 2
    elif tokens_per_month < 50_000_000:
        score['paid_api'] += 2
    else:
        score['self_hosted'] += 2

    if sensitive:
        score['self_hosted'] += 3
    else:
        score['free_hosted'] += 1
        score['paid_api'] += 1

    if latency_ms < 500:
        score['self_hosted'] += 1
    else:
        score['free_hosted'] += 1
        score['paid_api'] += 1

    if ops_hours < 4:
        score['free_hosted'] += 2
        score['paid_api'] += 1
    elif ops_hours < 20:
        score['paid_api'] += 2
        score['self_hosted'] += 1
    else:
        score['self_hosted'] += 2

    return score

if __name__ == '__main__':
    # tokens/month, sensitive, latency_ms, ops_hours, steady
    print(fit_score(6_000_000, False, 900, 2, False))

The weights are deliberately simple. You can argue with them; that is the point. The script exists to force a conversation about your numbers instead of the vendor's marketing.

Run a realistic case. A solo developer writes six million tokens per month, mostly in bursts, with no sensitive data, a 900 ms latency budget, and two spare ops hours. The output is {'free_hosted': 6, 'paid_api': 3, 'self_hosted': 2}. The free bucket wins on every dimension. A regulated fintech team with eighty million tokens per month, sensitive code, and a 300 ms budget lands at the opposite end: self-hosting is the only defensible choice.

How do you get the token estimate if you have never measured it? A rough proxy works: count the characters in your weekly diff output, divide by four, and multiply by the fraction of that code you actually paste into a model. git log --since='1 month ago' -p | wc -c gives the raw number in one command. It is an estimate, not a meter, but it beats guessing.

Where MonkeyCode sits in this framework.

One option in the free-hosted bucket is MonkeyCode, an open-source project that currently offers free model access with a 10 million token allowance and a free server option, so you do not need to provision your own box to start. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming it is the best option for every team; the framework above is the point. The allowance fits a bursty pattern — a few focused sessions per week — not a pipeline generating code all day.

The tradeoffs look like this:

Dimension	Free hosted	Paid API	Self-hosted
Upfront cost	None	None	GPU + setup time
Token ceiling	Fixed allowance	Pay per use	Hardware bound
Data leaves machine	Yes	Yes	No
Ops burden	None	None	High
Latency control	Provider	Provider	Full
Typical failure	Cap mid-sprint	Bill shock	Driver hell

The free server option removes one objection to hosted tools: you do not need to maintain a box. You still give up data locality and latency control, and the allowance is finite and provider-controlled. Those are real costs, and the framework accounts for them.

Who should not use this approach.

If your code is under a compliance regime, stop reading and self-host. If your monthly token burn is steady and above the allowance, a paid API with predictable pricing beats a free tier that interrupts your day. If your team has no infrastructure skills, do not romanticize the GPU; a hosted option, free or paid, is the rational choice. The free tier is a fit for the bursty, non-sensitive, low-ops developer. It is a trap for everyone else.

The badge on the model card never told you this. The vendor's benchmark never told you this. A ten-line script with honest inputs tells you more than both. If your numbers land in the bursty bucket, run the script, then give the free tier a weekend test — the math will tell you quickly whether it fits.

The MiniMax H3 Leaderboard Is Loud. My Repo Is Louder.

Sam Li — Fri, 14 Aug 2026 11:08:21 +0000

Liquid syntax error: Unknown tag 'endraw'

I Grade Every New Coding Model Against My Own Bug History — Here's the Scorecard

Sam Li — Thu, 13 Aug 2026 03:34:31 +0000

A model shipped this week. The screenshots arrived within hours: impossible refactors, one-shot fixes, declarations that everything before it is obsolete. I stopped arguing with launch-week sentiment a while ago. Instead I do something more boring and more useful: I grade the new release against a scorecard built from my own git history, and the scorecard decides whether the model gets anywhere near my working tree.

This is a different framing from the usual "run a benchmark" advice. I'm not benchmarking. I'm auditing a candidate against incidents I've already lived through — and the grading is weighted, because not all failures cost the same.

Why scorecards beat impressions

When I post-mortemed my own bad model adoptions, the mistakes clustered into three buckets:

Selection effects in what I saw. Public demos are chosen because they succeed. My code has none of that filtering.
Recall masquerading as reasoning. Tasks that look like public GitHub code can be answered from memory. My evaluation tasks must come from private history.
Noisy first impressions. A model that shines on a blank file can still wreck a mature module with house conventions. The second case is what I actually pay for.

A scorecard handles all three: frozen tasks from private PRs, mechanical checks, and weights that reflect blast radius rather than vibes.

Step 1: Mine your own git history for tasks

The fastest way to build a task suite is to recover work you've already verified. I scan for small, self-contained fixes:

# Candidate commits: single-file changes, small diffs, likely bugfixes
git log --since="-6 months" --name-only --oneline \
  | awk '/^[0-9a-f]{7,}/ {sha=$1} /\.[a-z]+$/ {print sha, $0}' \
  | sort | uniq -c | awk '$1 == 1 {print}' > candidates.txt

From that list I hand-pick eight episodes: a null-handling regression, a race in a background worker, a migration that needed a rollback path, a refactor tangled in an internal style guide. Each becomes:

suite/worker-race/
  brief.md        # the task, written once, model-agnostic
  before.patch    # starting state of the code
  check.sh        # mechanical verification, exit 0 = pass

Two rules keep this honest: every task has a known-good resolution (because I merged one), and check.sh never gets edited to accommodate a model. A confusing brief is a defect in the task, not evidence about the model.

Step 2: Weighted scoring, not pass counting

Raw pass rate hides the failures that actually hurt. I assign each task a weight reflecting what a wrong answer would cost in production, then let a script do the arithmetic:

#!/usr/bin/env python3
"""grade.py — read results/*.json, emit a weighted verdict."""
import json, glob, sys

# weight = blast radius if this category fails silently
WEIGHTS = {"correct": 1.0, "scope": 3.0, "revertable": 2.0}

def grade(result):
    score, total = 0.0, 0.0
    for task in result["tasks"]:
        total += sum(WEIGHTS.values())
        if task["tests_pass"]:            score += WEIGHTS["correct"]
        if not task["touched_out_of_scope"]: score += WEIGHTS["scope"]
        if task["diff_reverts_cleanly"]:   score += WEIGHTS["revertable"]
    return score / total

for path in glob.glob("results/*.json"):
    r = json.load(open(path))
    print(f"{r['model']:40s} {grade(r):.2%}")

The deliberate asymmetry: scope discipline is weighted three times higher than correctness. In my incident history, the expensive model behavior was never "the fix was wrong" — tests catch that. It was "the fix was right and it also rewrote two files nobody asked about." A candidate at 91% with surgical diffs outranks one at 95% that wanders. The weighting table is the actual decision document; tune the weights to your own incident log, not to mine.

I also log latency and token counts per task, because "free" and "cheap" claims only mean something measured against your workload's shape.

Step 3: One attempt, frozen conditions

Every (model, task) pair gets exactly one shot at temperature zero. Retries and re-prompting measure my patience, not the model. The runner is a thin loop that applies before.patch, invokes whatever endpoint I'm testing through a generic wrapper, applies the returned diff, runs check.sh, and dumps a JSON record into results/. Swapping in this week's release is one environment variable — the model is the variable; everything else stays frozen so scores are comparable across weeks.

Running this without a line item in the budget

Scoring every new release against paid APIs adds up fast, and self-hosting a day-one model eats the hour you were trying to save. For these grading passes I currently use MonkeyCode, which offers free access to a selection of models along with a free server option for the runner side — a full eight-task grading run costs me the time, not an invoice. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Caveats I'd apply to any free tier, this one included: treat it as an evaluation environment, not a production commitment, and don't assume a hosted endpoint behaves identically to the vendor's reference deployment. If a candidate passes and later graduates to real use, I re-grade it against the production endpoint before cutover — I've seen endpoint differences flip a verdict.

Passing earns a probation period, not access

A strong score buys the model one week as a suggestion-only assistant: it proposes, I read every line, nothing merges itself. Week two it may open draft PRs on low-risk paths. Proximity to CI-gated auto-merge comes later and stays behind required human approval. The scorecard is a necessary gate, never a sufficient one.

Where this breaks down

Eight weighted tasks is a smoke screen, not a ranking system. It eliminates disasters and wanderers; it won't finely separate two good candidates. Grow the suite before making expensive commitments.
Single-shot, temperature-zero grading is unfair to agent-loop models. If your workflow is multi-turn, build a frozen multi-turn variant — but freeze it just as hard.
Small or mostly-greenfield repos have less to lose. This scorecard pays for itself on large, old, convention-heavy codebases.
Data handling comes first. If code can't leave your perimeter, don't send it to any third-party endpoint, free tiers least of all.

The feed will crown another model next week. The scorecard doesn't care — it takes an hour, speaks in weights, and grades against bugs I've actually paid for. If you're assembling your own version and want a zero-cost place to run it, MonkeyCode's free model access and server are a reasonable bench; the git history, the checks, and the weights have to come from you.

Your Feed Says the New Model Is Great. Mine Says Prove It in Under an Hour

Sam Li — Mon, 10 Aug 2026 10:38:24 +0000

Another open-weight release, another week of screenshots. This time it's MiniMax M3 filling my timeline, and the pattern is identical to every release before it: polished demos everywhere, reproducible evidence almost nowhere.

I've already covered why benchmark passes shouldn't earn merge privileges, and how to build a multi-day harness for comparing free coding models properly. But there's a cheaper question that comes first: is this model even worth feeding into that harness? Most aren't. What follows is the screening protocol I use to answer that in under an hour, before I invest a single weekend hour.

The output isn't a ranking. It's a short memo: proceed, park, or pass — with receipts.

Why demos can't answer the only question that matters

Launch-week content almost always answers "can this model do something impressive?" Of course it can. Every frontier-adjacent release can. The questions that actually determine whether a model belongs in my workflow are narrower:

Does it survive contact with my unglamorous tasks — the refactors, the half-documented bugs, the regex archaeology?
Does it fail in ways I can live with, or in ways that quietly corrupt my codebase?
Can I check any of this without spending money or owning serious hardware?

That last one used to be the blocker. It isn't anymore, and I'll come back to why.

The protocol: verify, stress, cost

Phase A — Bring your own tasks, not its demo (~15 min)

Pull three prompts from work you genuinely did recently. Mine change every time, but here's the set I used this week:

A. A 50-line function from an actual project + "make this testable
   without changing its observable behavior; list every behavioral risk"
B. A flaky-test description from our tracker + "propose three hypotheses
   ranked by how cheap each is to falsify"
C. A confusing build error + "explain the failure chain, then give the
   smallest change that fixes the root cause rather than the symptom"

I'm not grading eloquence. I'm watching for three instant-disqualifiers: referencing functions or flags that don't exist in the material I provided, ignoring an explicit constraint, or producing a fix that addresses the example I gave but obviously not the class of problem. Any one of these and the memo says pass. Done. The model just saved me a weekend.

Phase B — Interrogate the weakest answer (~15 min)

First replies are marketing. Second replies are character. Take the shakiest output from Phase A and challenge it directly:

"Point to the exact line where your claim is supported by what I gave you."
"Your hypothesis B contradicts the log I pasted. Which one is wrong, and how would you check?"

What I'm scoring is epistemic behavior, not accuracy. A model that says "you're right, I conflated the two call sites — here's the corrected trace" is a candidate. A model that rephrases the same wrong answer with more confidence is a liability wearing a nicer jacket. In my experience this single exchange predicts long-run usefulness better than any single-shot benchmark.

Phase C — One long task, and an honest look at what the poking costs (~25 min)

Short prompts hide context degradation, so I end with one task that forces the model to hold several files in its head at once — something like "here are three modules; trace where this value can become undefined between entry and the database write."

And here's where the economics of skepticism have genuinely shifted. Disclosure: This article was prepared as part of MonkeyCode's product outreach. It's relevant to Phase C specifically because MonkeyCode offers free model access alongside a free server option, which removes the two classic excuses for skipping this phase — no API budget, no local GPU. When the weights are open and a place to run them costs nothing, "verify the claims yourself" stops being a luxury and becomes a routine chore. That's the healthiest thing about the current open-model moment: nobody has to trust launch-day threads, mine included.

The memo format

After roughly an hour I write five lines. No gut feelings, each line cites something I actually saw:

Line	What qualifies as evidence
Constraint discipline	Followed every explicit instruction across all prompts
Grounding	Never cited code or APIs outside the provided context
Behavior under challenge	Located and corrected its own mistake in one or two turns
Long-context stamina	Cross-file reasoning stayed coherent; nothing dropped
Verdict	proceed / park / pass, plus one sentence why

"Park" is the underrated verdict: nothing disqualifying, nothing compelling, revisit in a month. The default for hype is pass — the burden of proof sits on the model, not on me.

Where this protocol breaks down

An hour can expose a bad model. It cannot certify a good one. "Proceed" means "earns a slot in the real harness," full stop.
If your workflow is one mature, well-tuned setup you already trust, even a passing triage may not clear your switching costs.
Free access tiers and free servers are offers, not entitlements — limits and availability can change. Confirm what exists today before designing habits around it, the same skepticism you'd aim at a leaderboard.
On M3 specifically: everything above was run against my own tasks, not quoted from published numbers. Read the official release notes, then reproduce on your material before repeating anyone's figures — especially figures from strangers on the internet, a category that includes me.

The actual takeaway

The durable skill isn't picking the "best" model each cycle. It's owning a cheap, repeatable filter that answers "is this one good for my work" faster than the hype cycle can move on. Open weights plus free places to run them have collapsed the cost of that filter to about an hour. The checklist is the only missing piece, and now you have one.

If cost or hardware has been your reason for taking model claims on faith, the free server option over at MonkeyCode is a reasonable place to run Phase C yourself — but the protocol works anywhere you can get the model to answer the same three tasks twice.

A Staged Gate for Adopting Free AI Coding Models Without Wrecking Your Repo

Sam Li — Mon, 10 Aug 2026 08:10:53 +0000

In my last two posts I argued for running your own harness before trusting any coding model, and for refusing to merge model output just because a benchmark says it's good. This post is the practical middle step I skipped: how do you decide whether a free-tier model belongs in your daily loop at all?

The wrong way to answer that is vibes. You paste three prompts, one looks impressive, and suddenly the model is writing half your PRs. The right way is a staged gate: cheap tests first, expensive tests only for survivors. Here's the exact sequence I use, plus the scoring artifact you can copy.

The staged gate, in one picture

Stage 0: Task taxonomy     → does this model even fit my work?
Stage 1: Static smoke test → syntax, imports, obvious hallucination
Stage 2: Sandbox execution → does it run? do my tests pass?
Stage 3: Diff review       → would I approve this PR from a junior?
Stage 4: Shadow week       → model proposes, human disposes

A model must pass each stage to reach the next. Most free models fail at Stage 2, which is fine — the whole point of staging is that Stage 2 costs you an afternoon, not a production incident.

Stage 0: Write your task taxonomy before touching any model

This is the step everyone skips, and it's why their conclusions are useless. Write down the five to eight task types that actually make up your week, weighted by frequency. Mine, as an example:

Task type	Share of my week	Failure cost if model is wrong
Boilerplate / CRUD scaffolding	25%	Low
Test writing for existing code	20%	Medium
Refactoring within one file	15%	Medium
Cross-file bug diagnosis	15%	High
Config / CI / YAML wrangling	10%	High (silent breakage)
Unfamiliar-library usage	10%	Medium
Code review / explanation	5%	Low

Your table will differ. The weights matter because a model that's great at boilerplate and terrible at cross-file diagnosis might still be a net win if you restrict it to the first row. Free tiers especially tend to be uneven — strong on common patterns, shakier on long-context reasoning. The taxonomy turns "is this model good?" (unanswerable) into "which rows of my table is this model good for?" (testable).

Stage 1–2: A minimal scoring script

For each row of the taxonomy, pick two or three real tasks from your own git history — not from a public benchmark, because public tasks leak into training data. Then score outputs mechanically. Here's a stripped-down version of the scorer I run (adapt freely; this is illustrative, not a product):

#!/usr/bin/env python3
"""Score model outputs per task type. Usage: score.py results.jsonl"""
import json, sys, subprocess, tempfile, os

def run_tests(code: str, test_file: str, lang: str = "python") -> bool:
    with tempfile.TemporaryDirectory() as d:
        src = os.path.join(d, "solution.py")
        with open(src, "w") as f:
            f.write(code)
        r = subprocess.run(["pytest", test_file, "-x", "-q"],
                           cwd=d, capture_output=True, timeout=120)
        return r.returncode == 0

def main(path):
    rows = {}
    for line in open(path):
        rec = json.loads(line)
        # rec: {"task_type": ..., "code": ..., "test_file": ..., "review_pass": bool}
        t = rec["task_type"]
        exec_ok = run_tests(rec["code"], rec["test_file"])
        score = (0.6 if exec_ok else 0.0) + (0.4 if rec["review_pass"] else 0.0)
        cur = rows.setdefault(t, [])
        cur.append(score)
    for t, scores in sorted(rows.items()):
        print(f"{t:35s} n={len(scores)}  mean={sum(scores)/len(scores):.2f}")

if __name__ == "__main__":
    main(sys.argv[1])

Two deliberate choices here:

Execution is weighted above review (0.6/0.4) because "it compiles and passes tests" is objective, while my review judgment is not. When the two disagree, the task goes into a manual pile.
Scores are per task type, never aggregated. An overall mean would hide exactly the failure modes you need to see. A model scoring 0.9 on scaffolding and 0.3 on cross-file bugs is not a "0.6 model" — it's a scaffolding tool.

Stage 3–4: The human gates

Stage 3 is a diff review with a single question: would I approve this if a junior teammate submitted it? Same bar, no discounts for "the AI did it." If anything, review harder — model errors are systematic, not random, so one weird import usually means a whole class of confident wrongness.

Stage 4 is a shadow week: the model proposes changes, you write or heavily edit everything yourself, and you log how often its proposal was usable. If the usable rate doesn't beat your baseline after a week, the model exits the loop for that task type.

Where free infrastructure fits

The expensive part of this workflow isn't the model calls — it's the iteration. Running Stage 1–3 across six task types and three candidate models means a few hundred executions, and you'll want to rerun it every time a model updates. Paying per-token for exploratory evaluation is how people end up evaluating on three prompts and calling it done.

This is where I've been using MonkeyCode: it offers free access to a set of coding models and a free server option, which maps neatly onto this workflow — free model access covers the candidate pool, and the free server is enough to host the sandbox executor for Stage 2 without standing up paid infrastructure. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I won't quote quotas, model lists, or limits here because they change; check the current offering yourself before designing around it. If your evaluation volume is small, any free tier — theirs or a provider's own — works equally well for Stages 0–3. The workflow is the point, not the vendor.

Decision table: when a free model earns a seat

Outcome across your taxonomy	Verdict
Passes stages 1–4 on ≥2 high-frequency task types	Adopt, restricted to those task types
Passes on low-frequency or low-risk types only	Keep for drafts; never let it touch CI config or cross-file changes
Passes execution but fails diff review repeatedly	Do not adopt; systematic style/logic drift will eat your review time
Fails sandbox execution broadly	Drop it and retest in a few months; models move fast

Limitations, and who should skip this

Sample size is your enemy. Two tasks per type is a smoke screen, not a study. If you can't assemble at least 15–20 real tasks, your results are noise. Be honest about that before acting on them.
Free tiers change without notice. Rate limits, model availability, and server capacity can shift mid-evaluation. Never wire a free tier into CI or anything with an SLA.
This doesn't transfer to regulated or high-blast-radius code. If a wrong suggestion can touch auth, payments, medical logic, or infra-as-code that deletes things, free-model gating is the wrong conversation entirely — you need audited tooling and human authorship guarantees.
Solo maintainers gain the most. Teams with established review culture already have a Stage 3; adding model evaluation is cheap. If you're solo with no tests, build the tests first — the harness is useless without them.

The gate is boring on purpose. Boring is what keeps a confident, free, occasionally-wrong model from becoming a line item in your incident review. If you've run a similar staged evaluation, I'd genuinely like to hear where your models failed — the taxonomy rows where free tiers collapse are more useful shared than hoarded.

The Model Passed Your Benchmark. Now Stop Merging Its Code Blindly

Sam Li — Fri, 07 Aug 2026 03:15:45 +0000

A few weeks ago I wrote about building a reproducible test harness for comparing free AI coding models before you commit. That harness answers one question: which model should I use?

It does not answer the harder follow-up: once a model generates a patch for my real codebase, when is it safe to merge?

This week there was a great discussion on DEV about "understanding over origin" — the idea that it doesn't matter whether code came from a human or a model, only whether someone actually understands it. I agree with the principle, but principles don't survive contact with a busy afternoon. What survives is a checklist with teeth. So here is the pipeline I bolted onto my model harness: every AI-generated patch has to pass through a scripted review gate before I even read it, and the script produces a scorecard that tells me how carefully I need to read it.

The problem with eyeballing diffs

When a model produces a 40-line diff that looks idiomatic, my brain does a dangerous thing: it pattern-matches on style and skips semantics. The code reads like something I'd write, so I approve it like something I'd write. The failures I've actually shipped from AI-generated code were never syntax errors — the tests even passed. They were things like:

A retry loop that retried on the wrong exception type, so real errors got swallowed.
A query filter that was subtly wider than the one it replaced (tests passed because fixtures were too small to notice).
A dependency added for a one-liner the standard library already covers.

All three would have been caught by asking four boring questions before reading the code. So I scripted the questions.

The review gate: a reproducible artifact

The gate is a small shell script. It takes a patch file, applies it to a throwaway worktree, and runs four checks. It never touches my working branch, and it prints a one-line verdict at the end.

#!/usr/bin/env bash
# review-gate.sh <patch-file> <base-branch>
set -euo pipefail

PATCH="$1"
BASE="${2:-main}"
WT=$(mktemp -d /tmp/ai-review.XXXXXX)

echo "== 1. Isolate =="
git worktree add --detach "$WT" "$BASE" >/dev/null
if ! git -C "$WT" apply --check "$PATCH" 2>/dev/null; then
  echo "VERDICT: REJECT (patch does not apply cleanly to $BASE)"
  git worktree remove --force "$WT"; exit 1
fi
git -C "$WT" apply "$PATCH"

echo "== 2. Full test suite (not just the touched package) =="
if ! (cd "$WT" && npm test --silent 2>&1 | tail -5); then
  echo "VERDICT: REJECT (tests fail)"
  git worktree remove --force "$WT"; exit 1
fi

echo "== 3. Diff surface audit =="
git -C "$WT" diff "$BASE" --stat
NEW_DEPS=$(git -C "$WT" diff "$BASE" -- package.json \
  | grep -c '^+.*".*":' || true)
echo "new dependency entries: $NEW_DEPS"

echo "== 4. Behavior-change heuristics =="
# Flag the patterns that have burned me before
SUSPECT=$(git -C "$WT" diff "$BASE" \
  | grep -E '^\+.*(catch|except|retry|timeout|WHERE|filter\()' \
  | wc -l | tr -d ' ')
echo "error-handling/query lines added: $SUSPECT"

if [ "$NEW_DEPS" -gt 0 ] || [ "$SUSPECT" -gt 5 ]; then
  echo "VERDICT: REVIEW CLOSELY (expanded surface area)"
else
  echo "VERDICT: STANDARD REVIEW"
fi

git worktree remove --force "$WT"

Adapt steps 2–4 to your stack. The point isn't my specific greps — it's that the checks run before my opinion does, every time, identically. Step 1 exists because a model's patch often "works" only against the stale context it was shown; applying it to a clean checkout of current main is itself a finding.

Where the free compute fits

This loop has two compute-hungry halves: generating candidate patches, and running the gate (full test suites per candidate get slow). Both halves are exactly where I've been using MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Concretely: I use MonkeyCode's free model access to generate two or three independent candidate patches for the same task, and its free server option to run the gate script against each candidate so my laptop isn't tied up running three test suites in parallel. Generating multiple candidates is the underrated move here — when two independent generations converge on the same approach, my review confidence goes up; when they diverge, the divergence points are exactly where I read line-by-line.

I'm deliberately not quoting quotas, model names, or performance numbers, because those change and you should check the current state yourself. What matters for this workflow is just: the generation step and the sandboxed-test step cost me nothing and don't monopolize my machine.

The decision table

After the gate runs, I use this to decide how much of my own attention to spend:

Gate output	My review depth	Typical action
Patch doesn't apply	None	Re-prompt with fresher context; don't hand-fix
Tests fail	Read the failure only	Regenerate or discard; never patch the patch blindly
Passes, no new deps, low suspect count	Read every changed line once	Merge after reading
Passes, new deps or high suspect count	Read lines + write one adversarial test	Merge only if my new test passes
Two candidates diverge on approach	Read both diffs at the divergence point	Pick one, write down why in the commit message

The last row is the one that pays rent. "Write down why" is my personal guardrail for the understanding-over-origin principle: if I can't explain in one sentence why this diff is correct, it doesn't merge, regardless of how green the tests are.

Limitations, and who shouldn't do this

The heuristics in step 4 are mine, tuned to my past failures. Yours will be different. Start with an empty list and add a pattern every time AI-generated code burns you — the script should grow scar tissue, not ship with mine.
Green tests are a floor, not a ceiling. If your test suite has weak coverage, the gate's value collapses to step 1 and the greps. Fix coverage first.
Don't run this for trivial diffs. A five-line config change doesn't need a worktree and a scorecard; ceremony has a cost too.
This is not a security review. Dependency additions get flagged, not audited. If the model adds a package, that package still needs a real look.
If your repo's tests can't run in a clean checkout (hidden local state, manual env setup), step 1 will fail constantly and teach you nothing. That's actually a useful signal about your repo, but fix it before adopting the gate.

Closing

Model comparison tells you which generator to trust on average. A review gate tells you whether to trust this specific patch. The first question is interesting; the second is the one that decides what your users run. If you're generating candidates with free model access anyway, the marginal cost of gating every patch through a clean-worktree script is about fifteen lines of bash — and one honest commit message at a time, it keeps the understanding on your side of the merge button.

If you've built your own version of this — especially the failure-pattern greps — I'd genuinely like to see what's in your list.