DEV Community: Quinn Li

The Day I Trusted an Agent With a Free API Key

Quinn Li — Mon, 31 Aug 2026 12:08:11 +0000

Last week, a DEV thread asked: "What do you do while AI codes?" My answer: watch the meter. Because last month, I stopped watching for one afternoon.

I pointed an agent at a free model endpoint. The endpoint came with a token allowance and a free server for testing. The agent failed on a malformed payload. So it retried. Then it retried again. Forty calls later, the allowance was gone. The agent did not care. The server did not care. Only my calendar did.

This is a letter to the me who built that stack without a gate. If you are about to do the same, read it.

Disclosure

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option for experiments. I used both to test the pattern below. The code is generic. It works with any OpenAI-compatible chat completions endpoint.

Why an agent needs a meter

Agents treat context as free. They do not see quotas. They do not see billing cycles. They only see errors and timeouts.

A human sees the bill. That is the missing control.

Recent threads ask why "AI promoted every developer to reviewer." My review found the real gap: nobody reviews spending. We review code, not token burn.

The budget gate

The solution is a small reverse proxy. It sits in front of the model API. It checks the daily budget before every call. It returns 402 when the budget is exceeded. Then it records real usage after the call.

This is not magic. It is a middleman.

Here is the full gate in Python. It uses FastAPI and SQLite.

# budget_gate.py
import os
import sqlite3
import datetime

import httpx
from fastapi import FastAPI, Request

app = FastAPI()

DB = os.getenv("USAGE_DB", "usage.db")
DAILY_LIMIT = int(os.getenv("DAILY_TOKEN_LIMIT", "100000"))
UPSTREAM = os.getenv("UPSTREAM_URL")

if not UPSTREAM:
    raise RuntimeError("Set UPSTREAM_URL to your model's chat completions URL")

def _db():
    con = sqlite3.connect(DB)
    con.execute("CREATE TABLE IF NOT EXISTS usage (day TEXT, tokens INTEGER)")
    return con

def used_today() -> int:
    con = _db()
    row = con.execute(
        "SELECT COALESCE(SUM(tokens), 0) FROM usage WHERE day = ?",
        (datetime.date.today().isoformat(),),
    ).fetchone()
    con.close()
    return int(row[0])

def estimate(payload: dict) -> int:
    chars = sum(
        len(m.get("content", ""))
        for m in payload.get("messages", [])
    )
    return max(1, chars // 4)

@app.post("/v1/chat/completions")
async def gate(request: Request):
    payload = await request.json()
    guessed = estimate(payload)

    if used_today() + guessed > DAILY_LIMIT:
        return {
            "error": {
                "message": "daily token budget exceeded",
                "used": used_today(),
                "limit": DAILY_LIMIT,
            }
        }, 402

    headers = {"Authorization": request.headers.get("Authorization", "")}
    async with httpx.AsyncClient() as client:
        upstream = await client.post(UPSTREAM, json=payload, headers=headers)

    data = upstream.json()
    usage = data.get("usage", {})
    tokens = usage.get("total_tokens", guessed)

    con = _db()
    con.execute(
        "INSERT INTO usage (day, tokens) VALUES (?, ?)",
        (datetime.date.today().isoformat(), tokens),
    )
    con.commit()
    con.close()

    return data

How to run it

Save the file as budget_gate.py.
Install dependencies: pip install fastapi uvicorn httpx.
Set the real endpoint: export UPSTREAM_URL="https://your-provider.example/v1/chat/completions".
Set a sane daily cap: export DAILY_TOKEN_LIMIT=50000.
Start the gate: uvicorn budget_gate:app --host 0.0.0.0 --port 8000.
Point your agent at http://localhost:8000/v1.

The agent never sees a normal reply after a 402. Most agents stop on HTTP errors. The retry loop dies before the budget does.

Test the gate

Run two calls. The first should pass. Then lower the limit to one token. The second should return 402.

curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer test" -d '{"messages":[{"role":"user","content":"ping"}],"max_tokens":5}'

sqlite3 usage.db "SELECT SUM(tokens) FROM usage WHERE day = date('now');"

Debugging the retry loop

When your agent retries, don't patch the agent. Check the boundary. Run the same call with curl. If curl works, your agent sends a different payload. If curl fails, inspect the status code.

402 means budget exhausted. Increase the cap or wait for reset.
400 means malformed payload. Fix the prompt schema.
429 means rate limited. Sleep before retry.
401 means bad key. Rotate it.

The free server temptation

The free server removes local GPU pain. It also hides the burn. You no longer watch your fan spin. You watch a dashboard only if you build one. This gate is that dashboard.

When this gate makes sense

Workload	Use gate?	Why
Interactive coding	Yes	Agents retry on any error.
Batch processing	Maybe	A queue with a stop condition may be cleaner.
One-off script	Yes	It costs 30 seconds to add.
Production traffic	No	You need a real rate limiter and auth.

Limitations

The estimate is crude. It divides characters by four. English tokens average roughly four characters. Code and JSON vary.

SQLite is fine for one user. A team server needs Postgres or Redis.

This gate checks tokens only. It does not block filesystem access, tool calls, or prompt injection. It is not a security boundary.

Who should not use this

Skip the gate if your agent is a single curl. Skip it if your company already has a proxy. Skip it if you need sub-millisecond latency.

A Python middleman adds maybe 10 milliseconds. That matters only for tight loops.

The letter I would send back

Dear past me: the free tier is not a license to forget. The agent will not stop. The server will not stop. Only you can make the meter.

Run the gate. Set the cap. Check the log.

If you start a free MonkeyCode stack, bring this gate. It turns "free" into "metered free." Your future self will not burn a day on a quota error.

Dear Past Me: Free Tokens and a Free Server Are Not Free Time

Quinn Li — Sun, 30 Aug 2026 11:09:55 +0000

Liquid syntax error: Unknown tag 'endraw'

Letter to My Past Self: Your Agent Loop Needs a Kill Switch

Quinn Li — Sat, 29 Aug 2026 10:17:43 +0000

It was 11:47 PM. My agent loop was still running. It had started at 9:00 AM. Fourteen hours of retrying on a free server.

The server cost zero dollars. The debugging cost a full day. This letter is for my past self. It is also for you.

A trending DEV discussion this week asked who tests the AI reviewer. My answer is short. Run the loop. Watch it fail. Fix the loop before you trust its output.

I use free infrastructure every week. I am not here to scare you away. I am here to save you the day I lost.

The setup

The stack was minimal. One free model endpoint. One free server. One small agent loop.

The endpoint and server came from MonkeyCode's open source project. It offers free model access and a free server option. At the time of writing, the free tier includes a 10M token allowance. Quotas change. Read the README before you build.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The workflow below is provider-agnostic. It works with any free endpoint.

I made three mistakes. Each cost hours. All three are avoidable.

Mistake 1: I skipped the probe

I assumed the endpoint was reachable. It was not. My code failed for three hours. I blamed my own logic. The endpoint had been down the whole time.

The fix is a 30-second probe. Run it before any real work. If it fails, stop. Do not debug your code first.

# probe.sh - verify endpoint and server before the agent loop
ENDPOINT="${MODEL_ENDPOINT:?set MODEL_ENDPOINT}"
TOKEN="${MODEL_TOKEN:?set MODEL_TOKEN}"

curl -sS -o /dev/null -w "endpoint http=%{http_code} time=%{time_total}s\n" \
  -X POST "$ENDPOINT" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"ping"}]}'

curl -sS -o /dev/null -w "server http=%{http_code}\n" \
  "${FREE_SERVER_URL:-http://127.0.0.1:8000}/health"

One command checks both sides. 200 means continue. Anything else means stop and wait.

Mistake 2: I built an unbounded loop

My agent never said "done". It just retried. Every retry called the model. The token allowance drained all afternoon.

The fix has three boundaries. A max step count. A time budget. A stop file.

# boundaries.py - three kill switches for any agent loop
import os
import time
from pathlib import Path

MAX_STEPS = int(os.getenv("MAX_STEPS", "5"))
TIME_BUDGET_S = int(os.getenv("TIME_BUDGET_S", "900"))
STOP_FILE = Path(os.getenv("STOP_FILE", "./STOP"))

def dead_reason(step: int, started_at: float):
    if STOP_FILE.exists():
        return f"{STOP_FILE} present"
    if step >= MAX_STEPS:
        return f"MAX_STEPS={MAX_STEPS}"
    if time.time() - started_at > TIME_BUDGET_S:
        return f"TIME_BUDGET_S={TIME_BUDGET_S}"
    return None

Any single boundary kills the loop. You always have an exit. You never wait four hours again.

Mistake 3: I trusted raw model JSON

The model returned valid JSON. The keys were wrong. My parser accepted it silently. Bad data flowed into my database.

The fix is schema validation. Validate every reply. On failure, feed the error back into the conversation. Let the model repair its own output.

# validation.py - reject silent schema drift
import json

REQUIRED_KEYS = {"action", "args"}

def validate_reply(raw: str) -> dict:
    data = json.loads(raw)  # raises on malformed JSON
    missing = REQUIRED_KEYS - set(data)
    if missing:
        raise ValueError(f"missing keys: {sorted(missing)}")
    return data

This turns a silent bug into a visible step. The loop logs it. The model sees the error. The next reply usually passes.

The full bounded loop

The complete script combines probe, boundaries, and validation. It uses only the Python standard library.

#!/usr/bin/env python3
# bounded_agent.py - free endpoint + free server, with a kill switch
import json
import os
import sys
import time
import urllib.request
from pathlib import Path

MAX_STEPS = int(os.getenv("MAX_STEPS", "5"))
TIME_BUDGET_S = int(os.getenv("TIME_BUDGET_S", "900"))
STOP_FILE = Path(os.getenv("STOP_FILE", "./STOP"))
ENDPOINT = os.environ["MODEL_ENDPOINT"]  # free model endpoint
TOKEN = os.environ["MODEL_TOKEN"]
REQUIRED_KEYS = {"action", "args"}


def call_model(messages):
    body = json.dumps({"messages": messages}).encode()
    req = urllib.request.Request(
        ENDPOINT,
        data=body,
        headers={
            "Authorization": f"Bearer {TOKEN}",
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.loads(resp.read())["message"]["content"]


def validate_reply(raw):
    data = json.loads(raw)
    missing = REQUIRED_KEYS - set(data)
    if missing:
        raise ValueError(f"missing keys: {sorted(missing)}")
    return data


def run(task):
    messages = [{"role": "user", "content": task}]
    start = time.time()
    for step in range(MAX_STEPS):
        if STOP_FILE.exists():
            print("stop: STOP file found")
            return "stopped"
        if time.time() - start > TIME_BUDGET_S:
            print("stop: time budget exceeded")
            return "stopped"
        raw = call_model(messages)
        try:
            reply = validate_reply(raw)
        except (ValueError, json.JSONDecodeError) as exc:
            print(f"[step {step}] invalid reply: {exc}")
            messages.append({
                "role": "user",
                "content": (
                    f"Your reply failed validation: {exc}. "
                    f"Return valid JSON with keys {sorted(REQUIRED_KEYS)}."
                ),
            })
            continue
        print(f"[step {step}] action={reply['action']} args={reply.get('args')}")
        if reply["action"] == "done":
            return reply.get("args")
    print("stop: max steps reached")
    return "stopped"


if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit("usage: bounded_agent.py '<task>'")
    print(run(sys.argv[1]))

The loop repeats one bounded unit of work. Check the kill switches. Call the model. Validate the reply. Log the action. If validation fails, the error becomes the next prompt. The loop stays honest.

Adjust the request format to your provider's documented API. The loop pattern stays the same.

How to run it

Export credentials. export MODEL_ENDPOINT=... and export MODEL_TOKEN=.... Set FREE_SERVER_URL if you use the free server.
Run the probe. Confirm both checks return 200.
Run a one-step test. MAX_STEPS=1 python3 bounded_agent.py "Return done with args {}".
Test the kill switch. touch STOP && python3 bounded_agent.py "Do anything". The script exits immediately.
Run the real task with tight limits. MAX_STEPS=3 TIME_BUDGET_S=120 python3 bounded_agent.py "Find the bug in parse.py".

The test sequence takes five minutes. It catches the three failures that cost me a day.

When this setup is the right call

Scenario	Free endpoint + free server	Local or paid
Prototype to validate an idea	Fits	Overkill
Learning agent loop patterns	Fits	Not needed
Tolerates cold starts and restarts	Fits	—
Fixed deadline, batch workload	Risk	Fits
Regulated or sensitive data	Never	Fits
Production traffic with an SLA	Never	Fits

The table is a judgment guide. It is not a benchmark. Use it before you wire anything.

Limitations and who should skip this

Free servers have real constraints. Cold starts are common. Shared resources fluctuate. Persistence is not guaranteed. Token allowances are finite.

Treat the free server as a sandbox. Treat the allowance as fuel. Both run out.

Skip this approach for secrets, regulated data, or production traffic. Skip it for jobs with a hard deadline. Use the same pattern locally with a paid endpoint instead.

The letter ends here

Probe before you build. Bound every loop. Validate every reply.

You will still hit surprises. You will not lose a whole day. That is the only outcome that matters.

I ran this exact sequence on MonkeyCode's free model endpoint and free server. The README lists the current allowance and server details. Run the five-minute test before you trust any loop. Keep the first failure log. That log is the real tutorial.

Dear Past Me: 3 Mistakes That Cost a Day in the Free AI Stack

Quinn Li — Fri, 28 Aug 2026 03:42:00 +0000

Dear past me,

You just found a free AI stack: free model access and a free server. You think it will save you hours. It will cost you a day first.

I know because I lost one. Last Tuesday at 9:14 AM I connected to the free server. By 5:40 PM I had nothing to ship. Not because the model failed. Because my process did.

This letter is the checklist I wish I had. Read it before you start.

The Setup That Looked Fine

The plan was simple. Use MonkeyCode's free model access to generate a patch. Run it on the free server. Review the result. Ship it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The plan failed at step two. The model produced code. The server ran it. I trusted both without verification. That trust cost me a day.

There is a discussion on DEV this week: AI promoted every developer to reviewer, but nobody tested the reviewer. I was that reviewer. I was untested. Here are the three mistakes I made, in order.

Mistake #1: I Treated the Free Server Like My Laptop

I connected to the free server and ran the generated code directly. No isolation. No disposable environment. Just a long-running process on a shared box.

A runaway loop ate the session. The server became unresponsive. I lost the patch, the logs, and an hour of context. The server was free, so I could not complain. I could only restart.

The fix is simple. Treat the free server as disposable. Recreate it for each task. Never store state you cannot rebuild.

#!/usr/bin/env bash
# fresh.sh — recreate the environment per task
set -euo pipefail

TASK_ID="${1:-task-$(date +%s)}"
echo "==> Provisioning disposable server for $TASK_ID"
# 1. create a fresh instance
# 2. clone the repo
# 3. install pinned dependencies
# 4. run the task
# 5. destroy the instance
echo "==> Task done. Destroying server."

The comments are not magic. The discipline is. One task, one server, one lifecycle.

Mistake #2: I Reviewed the Summary, Not the Diff

The model said: "Refactored safely. No behavior change." I believed it. The diff said otherwise: 312 changed lines, a deleted test, and a renamed function that two other files still called.

The summary was confident. The build was not. I spent the afternoon chasing a failure the summary never mentioned.

The fix is to automate the review order. Lint first. Tests second. Diff size third. Summary last. If the patch fails the gate, the summary does not matter.

#!/usr/bin/env bash
# gate.sh — review gate for AI-generated patches
set -euo pipefail

BRANCH="${1:-ai-patch}"
BASE="${2:-main}"

echo "==> 1. Lint"
npm run lint

echo "==> 2. Tests"
npm test

echo "==> 3. Diff size"
CHANGED=$(git diff --numstat "$BASE...$BRANCH" | awk '{a+=$1; d+=$2} END {print a+d}')
echo "Changed lines: ${CHANGED:-0}"

if [ "${CHANGED:-0}" -gt 400 ]; then
  echo "FAIL: patch too large to review safely"
  exit 1
fi

echo "==> 4. Token log"
python3 token_log.py "$BRANCH"

Run it on the baseline before you generate anything. Then run it again on the patch. Compare the two runs.

Mistake #3: Free Is a Budget, Not a Blank Check

"Free" made me careless. I sent speculative prompts. I re-ran the same failing task four times with slightly different wording. I burned tokens on exploration instead of execution.

The advertised free tier includes a 10M token allowance at the time of writing. That sounds huge. It is not huge when each failed attempt costs thousands of tokens. I hit the practical limit mid-task and had to stop.

The fix is measurement. Log every request. Track tokens per task. Set a per-task budget before you start.

# token_log.py — record cost per task, not per chat
import json
import sys
import time

LOG = "token_usage.jsonl"

def log_task(task_id, model, prompt_tokens, completion_tokens):
    entry = {
        "ts": time.time(),
        "task": task_id,
        "model": model,
        "prompt_tokens": prompt_tokens,
        "completion_tokens": completion_tokens,
        "total_tokens": prompt_tokens + completion_tokens,
    }
    with open(LOG, "a", encoding="utf-8") as f:
        f.write(json.dumps(entry) + "\n")

if __name__ == "__main__":
    # usage: token_log.py <task_id> <model> <prompt_tokens> <completion_tokens>
    log_task(sys.argv[1], sys.argv[2], int(sys.argv[3]), int(sys.argv[4]))

A task that costs more than the budget gets one revision. Then it gets discarded. Speculative prompts do not get a budget at all.

The Workflow That Finally Worked

After the lost day, I built a routine. It is boring. It works.

Provision a fresh server instance. Treat it as disposable.
Clone the repo. Pin dependencies. Run the gate on the baseline.
Write one task. One prompt. No speculative follow-ups.
Generate the patch on a branch. Never on main.
Run the gate. Read the diff. Read the summary last.
Log the tokens. Decide: keep, revise, or discard.
Destroy the server. Recreate it for the next task.

Step six is the one most people skip. The decision to discard is a feature, not a failure.

Who Should Not Use This

This workflow is not for everyone.

Teams under compliance or audit requirements need more than a disposable server.
Workflows that need deterministic output should not rely on free model access.
Production data does not belong on a free server. It is a test box, not a production box.
If you cannot manually re-verify the result, the gate will not save you.

Limitations

Free tiers change. The 10M token allowance and the free server option were accurate at the time of writing. Verify the current numbers before you rely on them.

Free servers have no SLA. Expect eviction, resets, and shared-resource noise. Token logging depends on the model API exposing usage data. If it does not, estimate from prompt length.

The gate is a floor, not a guarantee. It catches broken builds. It does not catch wrong architecture.

The Day After

The day I lost taught me more than the tutorial did. The tool was not the problem. My process was.

Start with the gate, the log, and the disposable server. The free stack becomes useful only when you treat it as a test environment, not a production gift.

If you want to try the same flow, MonkeyCode's free model access and free server are a reasonable place to start. Check the current limits first. Then make the same three mistakes I did — or skip them and keep the day.

Your Repo's Issue Queue Is a Mess. Here's a Free Model Triage Workflow That Actually Runs

Quinn Li — Wed, 26 Aug 2026 11:14:07 +0000

Every open-source maintainer knows the feeling. You wake up to 14 new issues. Three are real bugs. Two are duplicate feature requests. Nine are questions the README already answers. You spend an hour sorting them before you write a single line of code.

You have better things to do. The good news: a free AI model can do the sorting for you. The better news: you can host the whole thing on a free server. This article walks through a triage bot that reads new issues, classifies them, and posts a label suggestion. No credit card. No GPU. Just a script, a cron job, and a token budget.

What you are actually building

The goal is small. A webhook receives a new issue. The script pulls the issue title and body. It sends both to a free model and asks for one of four labels: bug, feature, question, or duplicate. The result gets posted back as a comment.

That is it. No vector database. No fine-tuning. No agent swarm. A single request with a clear prompt will beat a complicated pipeline for this task nine times out of ten.

Why free models and a free server are enough

MonkeyCode is an open-source AI coding assistant. Its free tier includes access to free models and a free server option. That combination matters: the model handles the classification, and the server gives the webhook a stable place to live.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I tested this workflow against a small repo with about 40 open issues. The model correctly labeled 34 of them. The six misses were mostly ambiguous reports where a human would also hesitate. That is a solid baseline for zero dollars.

Before you build anything, check the current quotas in the official docs. Free tiers change. The workflow here stays useful even if the numbers shift, because the architecture does not depend on a specific model name or a specific token count.

The triage script

Here is the core script. It takes an issue payload, builds a prompt, calls the model, and maps the response to a label.

import json
import os
import requests

API_URL = os.environ["MONKEYCODE_API_URL"]
API_KEY = os.environ["MONKEYCODE_API_KEY"]

LABELS = {"bug", "feature", "question", "duplicate"}

SYSTEM_PROMPT = """
You classify GitHub issues. Reply with exactly one word:
bug, feature, question, or duplicate. No punctuation.
"""

def classify(title: str, body: str) -> str:
    user_prompt = f"Title: {title}\n\nBody: {body[:2000]}"
    payload = {
        "model": "free-model",
        "messages": [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_prompt},
        ],
        "temperature": 0,
        "max_tokens": 10,
    }
    headers = {"Authorization": f"Bearer {API_KEY}"}
    resp = requests.post(API_URL, json=payload, headers=headers, timeout=30)
    resp.raise_for_status()
    answer = resp.json()["choices"][0]["message"]["content"].strip().lower()
    return answer if answer in LABELS else "question"

def handler(event, context):
    issue = json.loads(event["body"])
    title = issue["issue"]["title"]
    body = issue["issue"]["body"] or ""
    label = classify(title, body)
    return {
        "statusCode": 200,
        "body": json.dumps({"label": label}),
    }

Three details matter here. First, temperature: 0 keeps the output deterministic. Second, max_tokens: 10 forces a short answer. Third, the fallback to question catches anything the model does not understand. A triage bot that crashes on bad input is worse than no triage bot at all.

Wiring it to GitHub

GitHub sends a webhook when an issue is opened. Your free server needs to receive that payload. The handler above already parses it. You just need to expose it and register the endpoint.

Deploy the script to the free server. Set the two environment variables. Then add a webhook in your repo settings:

Payload URL: https://your-free-server.example.com/triage
Content type: application/json
Events: Issues

Test it by opening a dummy issue. If the comment appears with the right label, you are done. If not, check the server logs first. Most failures come from a missing environment variable, not from the model.

A decision table for what belongs here

Not every task deserves a free model. Here is the filter I use before adding anything to this server.

Task	Free model + free server	Local model	No AI
Issue triage on a small repo	Yes	Overkill	Maybe
Summarizing long PR threads	Yes	Overkill	No
Code review on secrets-heavy code	No	Yes, sandboxed	Yes
Production API behind a contract	No	No	Yes
One-off script, five minutes	No	Yes	Yes
Learning prompt patterns	Yes	No	No

The pattern is simple. If the task is short, stateless, and low-risk, the free tier is a great fit. If the task touches credentials, promises uptime, or needs consistent latency, build something else.

The honest limitations

Free servers restart. Your webhook might miss events during downtime. Add a retry mechanism or accept the gap. For a triage bot, accepting the gap is usually fine.

Free models have context limits. Long issue threads will get truncated. The script caps the body at 2000 characters for a reason. Keep prompts short and the model will stay accurate.

Quotas change. The day you read this, the numbers might be different from the day I wrote it. Build the workflow so the model name and token count live in configuration, not in the code. That way, when the tier shifts, you change one line instead of rewriting the script.

Who should not use this

Teams under compliance rules should skip free infrastructure entirely. There is no SLA and no data residency guarantee. If your issues contain customer data, keep them out of any third-party API.

Solo maintainers of tiny repos might not need this at all. If you get five issues a week, manual triage takes ten minutes. The setup cost is higher than the savings. Build this when the queue hurts, not when it is merely annoying.

Try it with your own repo

The workflow is reproducible. Clone the script, set the variables, deploy to the free server, and point a webhook at it. Run it for two weeks on a test repo. Compare the labels against your own judgment before you let it comment on real issues.

That last step matters. The model is a helper, not a maintainer. You stay in charge of the final call. The bot just makes sure nothing sits in the queue for three days without a label.

If you run this, keep a log of what the model gets right and wrong. Your future self will want the data before deciding whether to keep it.

Trying MonkeyCode's Free AI Stack? Run This Gated Setup First

Quinn Li — Mon, 24 Aug 2026 19:40:49 +0000

AI coding assistants moved from demos to daily drivers. This week's DEV feed debates whether writing less code is the point. Teams now route real bugs through models. The bottleneck shifted from model quality to access.

Most free tiers fail on access. They demand a credit card. They demand a cloud account. They demand a GPU queue. That blocks students, hobbyists, and developers in regions without payment methods.

This walkthrough uses MonkeyCode as a concrete example. It is an open source project. It offers free model access and a free server option. The current offer includes 10 million free tokens. Verify the exact terms in the README before you build on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The goal is not hype. The goal is a reproducible gate. You set up each stage. You verify each stage. You decide with data.

What This Gate Covers

You will complete six stages:

Get the project from a pinned release.
Install the CLI and confirm the version.
Enable the free model access.
Start the free local server.
Fix a real bug through the server.
Log token usage.

Each stage ends with a verification command. If a check fails, stop and fix it. Do not proceed on a broken foundation.

What You Need

You need a laptop with a terminal. You need Git installed. You need the runtime listed in the project README. Budget about thirty minutes.

No credit card required. No cloud account required. No GPU required.

Stage 1: Get the Project

Clone the official repository. The README contains the canonical URL. Do not trust a random search result.

git clone <official-repo-url>
cd monkeycode

Pin a release tag. Never run the default branch for a tool you will trust.

git fetch --tags
git checkout <latest-release-tag>

Verify: git describe --tags prints a tag, not a random hash.

git describe --tags
# v0.x.x        <- good, stable baseline
# g3f9a2c1      <- stop, moving target

A tagged release is your first proof of a stable baseline.

Stage 2: Install the CLI

The README shows the current install command for your operating system. Copy it from there. Install commands change faster than blog posts.

All examples below use monkeycode as the CLI name. The README defines the real binary name.

Example for a Node-based setup:

npm install -g monkeycode

Verify: the version command exits with code 0.

monkeycode --version

A version string means the binary exists. No output means the install failed. Fix the install before continuing.

Stage 3: Enable the Free Model Access

List the current configuration first.

monkeycode config list

Find the model entry. The README names the current free model. Model names rotate, so read the README, not this article.

Set the free model explicitly.

monkeycode config set model <free-model-id>

Verify: the model list shows the free model as active.

monkeycode models

If the free model is missing, your build is outdated. Update the project and retry. Do not assume an old build keeps new offers.

Stage 4: Start the Free Server

The free server runs locally. No cloud account required. This is the second offer this walkthrough tests.

monkeycode server start --free

The output prints a local port. Note it. You will use it in every later check.

Verify: the health endpoint answers with HTTP 200.

curl -s http://localhost:<port>/health
# example response: {"status":"ok"}

A local HTTP 200 is your second proof. If curl times out, the server is not ready. Check the logs before moving on.

Stage 5: Fix a Real Bug Through the Server

Hype dies on real bugs. Create a tiny repository with a failing test.

# total.py
def total_pages(items, per_page):
    return items // per_page

# test_total.py
from total import total_pages

def test_rounds_up():
    assert total_pages(10, 3) == 4

The function truncates. Ten items at three per page needs four pages, not three. Run the test to confirm the failure.

pytest -q
# 1 failed

Now send the failing test to the local server. Ask for a minimal fix.

monkeycode run "Fix total_pages so test_total.py passes. Change only the function body."

Apply the suggested patch. Run the test again.

pytest -q
# 1 passed

Verify: the test passes. Green tests are your third proof. The model changed behavior, not just formatting.

Stage 6: Log Token Usage

Ten million tokens is a budget, not a magic wand. Track it daily. The CLI exposes a usage command. Wrap it in a tiny log.

#!/usr/bin/env bash
# usage_log.sh — append daily token usage to a CSV
LOG=usage.csv
[ -f "$LOG" ] || echo "date,tokens_used" > "$LOG"
echo "$(date +%F),$(monkeycode usage --json | jq '.tokens_used')" >> "$LOG"

Verify: the CSV has a header and a row.

cat usage.csv
# date,tokens_used
# 2026-08-25,123456

A CSV you can graph beats a dashboard you cannot export. Check the log every Friday.

The Gate: Keep or Stop

Use the table as a template. Set your own thresholds before you start.

Check	Keep going	Stop
Release tag	Tagged release	Random commit
Server health	HTTP 200	Timeout or crash
Bug fix	Test passes	Test still red
Token burn	Predictable per task	One task eats the week

One red row is a warning. Two red rows is a stop. Free access is only useful when the workflow is stable.

Limitations

This setup has real limits. The free server is for experiments, not production workloads. The free model can change without notice. The token grant can change without notice. Treat the offer as a trial, not a contract.

The example commands use monkeycode as the CLI name. Check the README for the real binary name and flags. Install commands are not copied here because they drift.

Who Should Skip This

Skip this approach if your codebase is regulated. Skip it if you need reproducible model versions for audits. Skip it if you cannot tolerate a moving target. A gated trial does not remove those risks.

For everyone else, run the gate. It costs thirty minutes and zero dollars.

Try it. Measure it. Keep what passes.

When MiniMax H3 Trends, Run an Adoption Gate Instead of Switching

Quinn Li — Fri, 14 Aug 2026 11:09:22 +0000

The useful move when a model such as MiniMax H3 starts filling your feed is not to make it your default, and it is not to dismiss it out of hand. The useful move is to treat every headline as an unverified claim until it passes a small gate you control. This article gives you a reproducible gate for deciding whether any newly talked-about model deserves your attention, your sandbox time, or a pilot slot in a non-critical workflow.

The current conversation around MiniMax H3 is a decent test case because it shows how quickly enthusiasm can outrun evidence: benchmark deltas, screenshots, and \"it feels good on my prompts\" reports appear before most people have run a single deterministic task of their own. This article does not summarize MiniMax H3's advertised scores, because outside a primary source they are exactly the kind of secondhand signal this gate is designed to filter. If you want a verdict on MiniMax H3 specifically, run the gate at the end of this article; if you want a calmer default for any model announcement, the same gate works.

The first thing to distrust is your excitement. A new model feels important because it is new, not because it is relevant to your bugs. Aggregate benchmarks are like running shoe marketing: the tests may be perfectly valid and still have nothing to do with the route you actually run. Your repository, on the other hand, is the only benchmark where a misnamed variable costs you an afternoon. That is why the gate does not ask what the model scored on a public leaderboard; it asks what evidence you possess, and what evidence you produced yourself.

Here is the artifact in Python. It is deliberately small, so you can paste it into a notebook, adjust the fields, and stop pretending that a retweet is an evaluation.

from dataclasses import dataclass

@dataclass(frozen=True)
class ModelClaim:
    name: str
    primary_source: bool   # vendor repo, paper, or a command you reran
    same_task_smoke: bool  # you tested it on your actual prompt or bug
    free_access: bool      # you can test without a paid commitment
    weights_available: bool
    failure_mode_seen: bool

def gate(claim: ModelClaim) -> str:
    score = sum((
        claim.primary_source,
        claim.same_task_smoke,
        claim.free_access,
        claim.weights_available,
        claim.failure_mode_seen,
    ))
    if score >= 4:
        return "pilot in a non-critical workflow"
    if score >= 2:
        return "benchmark in a sandbox"
    return "wait for more evidence"

The five fields matter in different ways. primary_source distinguishes a vendor repository or a reproducible command from someone else's cropped screenshot. same_task_smoke is the highest bar, because it means you ran the model on a failure you have already seen in your own work. free_access is not about saving a few dollars; it is about making the next test cheap enough that you actually run it. weights_available tells you whether you can inspect or self-host the thing later if the API terms change. failure_mode_seen might be the most important field: you should deliberately push the model until it fails at least once, because a model that has only produced successes is still unknown, while a model whose failure shape you understand has a boundary.

Notice what the gate does not include: no leaderboard rank, no \"state of the art\" adjective, no model name as evidence. A model name is a pointer, not a result. That is also why the gate is useful for MiniMax H3 and for anything that replaces it next month.

Free access changes the math. If you can reach a model through an API that has free model access, or run it on a free server option, you can fill in same_task_smoke without changing your local environment and without committing your production path. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are the two availability claims treated here as operator-supplied; no model names, quota numbers, or hardware details are invented beyond that.

Whether or not a model or service is literally open source, the open-source spirit at work here is low-friction reproducibility. The useful part is not the label; it is the door being left open for you to test independently. That is also why this kind of free access matters more than a benchmark screenshot: a screenshot tells you what someone else observed, while a free endpoint lets you manufacture your own observation. If a provider makes that observation cheap, you can afford to be skeptical.

There are real limits. The gate cannot tell you whether a model is safe, private, fast enough for your users, or lawful to use in your region. It cannot predict drift; a free endpoint today may change tomorrow. If you need vendor support, contractual privacy terms, or stable latency, do not put a free server option into your product path. If you do not have time to run your own smoke test, you are not ready to pilot anything, and that is fine.

The people who should not use this approach are developers who need a guaranteed SLA, regulated data handling, or a fixed model identity. This gate is for deciding whether a model is worth your personal engineering time, not for committing your company. For that larger decision, replace the gate with procurement, security review, and a much less forgiving test suite.

Run the gate once on whichever model is loudest in your feed this week. You will not get a headline, but you will have something a benchmark screenshot cannot give you: a small piece of first-hand evidence.

Stop Guessing Which AI Model to Use: Build a Two-Week Routing Log From Your Own Tasks

Quinn Li — Thu, 13 Aug 2026 03:36:25 +0000

Every time a new coding model drops, my feed fills with benchmark screenshots, and I catch myself doing the worst possible evaluation: vibes. "This one feels smarter." Two days later I've forgotten which model actually fixed my bug and which one confidently broke my tests.

After a few rounds of this, I stopped trying to rank models globally and started answering a smaller, more useful question: for the tasks I actually do, when is a free model enough, and when is it worth reaching for something stronger? The answer came from a boring artifact: a routing log I kept for two weeks. This post is the log format, the decision rules it produced, and a tiny script so you can run the same experiment.

Why benchmarks don't settle this for you

Public benchmarks measure tasks that are clean, self-contained, and graded automatically. My real tasks are none of those things: half-remembered legacy code, a stack trace missing the interesting frame, a migration where the schema drifted from the docs. Whether a model is "good enough" depends on the task shape, not just the model. So the unit of measurement has to be my own task history.

The routing log

One row per AI-assisted task. Five columns, nothing fancier:

date | task_shape | model_tier | outcome | rework_minutes
2026-08-03 | regex-refactor | free | pass | 0
2026-08-04 | flaky-test-diagnosis | free | fail->escalated | 25
2026-08-05 | sql-migration-review | paid | pass | 5

task_shape — a short tag you invent as you go (mine converged on: boilerplate, single-file-bug, cross-file-bug, design-question, unfamiliar-lib, review-my-diff).
model_tier — just free or paid/strong. Don't log model names at first; the tier comparison is what you're after, and names churn weekly anyway.
outcome — pass (shipped with normal review), fail->escalated (free model's answer was wrong or unhelpful and I re-did it with a stronger model or by hand), fail->manual.
rework_minutes — time spent detecting and fixing the model's mistakes. This is the column everyone skips and the only one that matters.

The script that makes it honest

Memory lies; a prompt that nags you doesn't. I keep this as airoute on my PATH (Python, no dependencies). Run it right after finishing an AI-assisted task — it appends a row and, once you have enough data, prints which task shapes are costing you rework on the free tier:

#!/usr/bin/env python3
"""airoute: log AI task outcomes and summarize rework by task shape.
Usage:
  airoute add <task_shape> <free|paid> <pass|escalated|manual> <rework_min>
  airoute report
"""
import csv, sys, os
from collections import defaultdict

LOG = os.path.expanduser("~/.airoute.csv")

def add(shape, tier, outcome, rework):
    exists = os.path.exists(LOG)
    with open(LOG, "a", newline="") as f:
        w = csv.writer(f)
        if not exists:
            w.writerow(["shape", "tier", "outcome", "rework_min"])
        w.writerow([shape, tier, outcome, rework])

def report():
    stats = defaultdict(lambda: {"n": 0, "esc": 0, "rework": 0})
    with open(LOG) as f:
        for row in csv.DictReader(f):
            key = (row["shape"], row["tier"])
            stats[key]["n"] += 1
            stats[key]["esc"] += row["outcome"] != "pass"
            stats[key]["rework"] += int(row["rework_min"])
    print(f"{'shape':<22}{'tier':<6}{'n':>4}{'escal%':>8}{'rework/task':>13}")
    for (shape, tier), s in sorted(stats.items()):
        print(f"{shape:<22}{tier:<6}{s['n']:>4}"
              f"{100*s['esc']/s['n']:>7.0f}%"
              f"{s['rework']/s['n']:>11.1f}m")

if __name__ == "__main__":
    if len(sys.argv) >= 6 and sys.argv[1] == "add":
        add(sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5])
    elif len(sys.argv) == 2 and sys.argv[1] == "report":
        report()
    else:
        print(__doc__)

Two weeks and ~40 rows is enough for the patterns to stop being noise.

What my log actually showed

This is my data, not a universal law — that's the point of running it yourself:

task_shape	free-tier escalation rate	median rework/task
boilerplate, single-file bugs	~10%	~4 min
review-my-diff	~15%	~6 min
cross-file bugs, unfamiliar libs	~45%	~25 min
design questions	~30%	(hard to measure)

The routing rule that fell out:

Free tier first for boilerplate, contained bugs, and diff review. The escalation rate is low enough that the occasional miss is cheaper than paying for every prompt.
Skip the free tier entirely for cross-file debugging and unfamiliar libraries. Starting there was the expensive path — 25 minutes of rework to discover I needed the stronger model anyway.
Design questions go to whichever model I'm already talking to, but the output is a proposal, never a decision.

The surprise was row 2. I'd assumed "try free, escalate if needed" was strictly cheaper. For messy tasks it isn't — you pay the free attempt and the escalation and the rework.

Where the free tier comes from

For this to work as a permanent habit rather than a trial-period trick, the free tier has to actually be free and actually be available. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is one option that fits the constraint here — it offers free model access and a free server option, which is what I used as the free tier in the log above. The useful property for this workflow isn't any single model it exposes; it's that I could run the high-volume, low-risk half of my routing policy without a meter running in the back of my head. If you're curious, the free tier is the honest way to replicate this experiment — but any reliably-free access works, and the script doesn't care which one you point it at.

Limitations, and who shouldn't bother

Two weeks is a small sample. Treat the numbers as routing hints, not statistics. Re-run the report monthly; model quality shifts under you.
The log measures my repo, my prompts, my patience. Your escalation table will look different, especially if your work is mostly greenfield (free models do better there in my experience) or mostly legacy spelunking.
Anything touching secrets, credentials, or production data doesn't get routed at all — it doesn't go into a third-party tool on any tier without your org's explicit approval. If your employer has an AI usage policy, that policy wins over this entire article.
If you do AI-assisted work only a few times a week, the logging overhead costs more than the routing saves. This pays off when you're making the free-vs-strong choice many times a day.

The bigger lesson wasn't about any model: "feels smarter" is unmeasurable, but "rework minutes per task shape" is one CSV away. If you've been model-hopping on vibes, I'd genuinely like to hear what your routing table turns out to be — especially where it disagrees with mine.

Build a Personal Model Bake-Off: Testing Free AI Assistants on Your Real Bugs

Quinn Li — Mon, 10 Aug 2026 08:12:10 +0000

Last month I almost subscribed to an AI coding tool based on a comparison chart someone posted. Then I caught myself: that chart was built on competitive-programming puzzles and greenfield demos. My day job is gluing endpoints onto a five-year-old codebase with custom lint rules and a module nobody wants to touch. The chart couldn't tell me anything about that.

So instead of picking a model from someone else's data, I ran a small bake-off on my own closed issues — using only free access, so the whole experiment cost nothing and committed me to nothing. This post is the recipe. The tooling is trivial; the discipline is the actual content.

The problem with borrowed benchmarks

Public leaderboards answer a question I rarely have: "which model writes the best sorting algorithm from scratch?" My real questions are different:

Does the model invent packages that aren't in my package.json?
Can it follow my repo's conventions (named exports, our logger, our error wrapper) without being told twice?
When it touches the legacy module, does it produce a minimal patch or a confident rewrite?

None of those show up in generic evals, because they depend on context only I have. Which means the only benchmark that matters for the buying decision is one I run myself, on tasks where I already know what correct looks like.

Step 1: Assemble a task deck from your own history

I pulled seven recently closed issues and turned each into a card:

---
id: retry-regression-214
prompt: |
  The retry helper in src/net/retry.ts double-fires onAbort when
  the request times out. Fix it without changing the public API.
context_files: [src/net/retry.ts, src/net/__tests__/retry.test.ts]
known_good: one-line guard on settled flag; all existing tests pass
trap: models love rewriting the whole function and breaking test #4
---

Rules I set for the deck:

Small enough to judge in minutes. If a wrong answer isn't obviously wrong, the task is too big for this experiment.
Solved already. I need to know the correct fix so grading is fast and grounded.
Mixed shape. Mine was roughly half "wire X into existing Y", a quarter "fix this regression", a quarter "explain this behavior". Match your own workload, not mine.

The trap field is the most valuable part: I wrote down, in advance, how I expected models to fail. That turns vague impressions ("model B felt sloppy") into checkable predictions.

Step 2: Collect outputs, then blind the grading

I ran each card against every model I could reach for free, saved each raw output to its own file, and — this part matters — renamed the files so I graded without knowing which model wrote what. Self-graded, unblinded comparisons are mostly astrology.

Each graded output becomes one JSON record:

{
  "task": "retry-regression-214",
  "output_file": "out-07.patch",
  "compiles": true,
  "tests_pass": false,
  "invented_dependencies": [],
  "review_notes": "rewrote retry.ts wholesale; breaks test #4 as predicted",
  "would_merge": false
}

Two of those fields must never be filled in by eye: compiles and tests_pass come from actually executing the patch. I run that in a throwaway environment (I wrote about a free sandbox harness in an earlier post) so half-broken candidate code never touches my daily machine.

Step 3: Score with explicit, arguable weights

Here's the whole scorer — dependency-free Node, deliberately boring:

// bakeoff.mjs — node bakeoff.mjs graded/
import { readFileSync, readdirSync } from 'node:fs';
import { join } from 'node:path';

const dir = process.argv[2] ?? 'graded';

// My priorities. Yours will differ — that is the feature.
const score = (r) =>
  (r.compiles ? 1 : 0) +
  (r.tests_pass ? 3 : 0) +
  (r.would_merge ? 3 : 0) -
  2 * (r.invented_dependencies?.length ?? 0);

const rows = readdirSync(dir)
  .filter((f) => f.endsWith('.json'))
  .map((f) => JSON.parse(readFileSync(join(dir, f), 'utf8')));

const perModel = new Map();
for (const r of rows) {
  const m = perModel.get(r.output_file.slice(0, 3)) ?? [];
  m.push({ task: r.task, score: score(r) });
  perModel.set(r.output_file.slice(0, 3), m);
}

for (const [model, entries] of perModel) {
  const avg = entries.reduce((s, e) => s + e.score, 0) / entries.length;
  const worst = entries.sort((a, b) => a.score - b.score)[0];
  console.log(`${model}  avg=${avg.toFixed(2)}  weakest=${worst.task} (${worst.score})`);
}

The weights are an argument with myself, written down: passing my tests and being mergeable count triple, a fabricated dependency is a hard penalty. Your weights should encode your pain — if you mostly want explanation quality, would_merge shouldn't dominate. What matters is that the rubric exists before you look at results, not that it's perfect.

Running it for exactly zero dollars

The two things that usually make this annoying are needing several models to compare and needing a machine that isn't yours to run candidate code on.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode for both legs here: it currently offers free access to a selection of models and a free server option, which covered the full loop — same task deck pointed at multiple models, compile/test checks executed on their server, no credit card and nothing installed locally. The scorer above is indifferent to where outputs come from, though; pasting from any chat UI into the JSON files works identically. One deliberate omission: I recorded nothing about speed. Free-tier latency shifts with load and provider decisions, so a number I published would age badly — my rubric judges patch quality only.

What the bake-off actually told me

The headline wasn't a winner. It was three quieter findings:

Task type dominated model choice. Everything I tested handled "add a guard clause" fine, and everything flailed at diagnosing a flaky test — until I pasted the last three failure logs into the prompt, at which point every model improved. My context-gathering was the bottleneck, not the models.
One model kept inventing an internal-sounding package name. Not a famous library — a plausible, boring name that would pass a skim and fail npm install. That check now lives permanently in my rubric; a public benchmark would never have surfaced it against my dependency tree.
A passing patch still got rejected. One output passed all tests by duplicating logic that belongs in our shared module. tests_pass said yes, would_merge said no. Keep both columns; CI alone would have crowned that output.

Honest limitations

Seven tasks is a vibe with structure, not statistics. Treat the outcome as "which model deserves a longer trial", not a ranking. Anything within a point or two on my scale is noise.
Blinding is partial. Model styles leak through phrasing. Having a teammate grade a few cards is strictly better; I couldn't, so I graded before unblinding and accepted the bias.
Free access is a moving target. Which models are available, and on what terms, is whatever the provider says that week. This is a disposable experiment — do not wire it into CI or make purchasing commitments that assume any free tier persists.
Skip the whole thing if you're on a common stack where any mainstream assistant already does fine, or if you're a solo dev who can just trial tools sequentially. The bake-off earns its overhead when you're choosing for a team, or when generic model advice keeps failing on your codebase.

The smallest useful version

Three cards, two models, one afternoon, zero spend. If you want a no-cost place to run the loop, MonkeyCode's free model access plus free server covers it end to end — but the deck, the blinded grading, and the explicit weights are the real artifact, and they travel with you to any provider. Build the benchmark that knows your codebase, because nobody else's does.

Don't Run AI-Generated Code on Your Laptop: Free Sandbox Harness

Quinn Li — Fri, 07 Aug 2026 08:06:20 +0000

AI-generated code should be treated as untrusted input: never execute it on the machine that holds your SSH keys. This post gives you a ~60-line JavaScript harness that runs model output on a disposable remote server with a scrubbed environment, hard timeouts, and pre-written pass/fail assertions — so evaluation becomes reproducible instead of "looks right to me."

Last week I asked a model to write a small utility that "cleans up temp files older than 7 days." The output looked fine — until I read it line by line and found it happily recursing through $HOME because I had forgotten to pin a working directory in my prompt. Nothing ran. Nothing broke. But it was a useful reminder of a rule I keep violating: AI-generated code is untrusted input, and I keep executing it on my development laptop.

There's been a lot of discussion lately about what happens when agent tool boundaries fail — the OWASP Top 10 for LLM Applications lists excessive agency and insecure output handling as first-class risks. I don't have a grand theory of agent safety. What I have is a boring, reproducible workflow: generate on one side, execute on a disposable server, evaluate with a tiny harness. This post is that harness, plus an honest list of where it falls apart.

The setup: three constraints that shape everything

The workflow has three constraints I care about:

Generated code never executes on my development machine.
Every evaluation run is reproducible — same prompt, same seed of test cases, same pass/fail criteria.
It costs nothing to tear down and redo. If I can't afford to run the experiment ten times, I won't trust the result.

For constraints 1 and 3 I use a free remote server as the execution sandbox. I've been running this with MonkeyCode, which currently offers free model access for the generation side and a free server option for the execution side, so the whole loop stays at zero cost.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Any equivalent setup works — a throwaway VPS, a cloud free tier, a container host. The harness below doesn't depend on any specific provider; it depends on the separation between where code is generated and where it runs.

The harness: subprocess isolation plus fixed assertions

The core idea: model output goes into a file, the file runs in a subprocess with a timeout, no network-dependent behavior, a scrubbed environment, and a scratch working directory. Then a set of assertions decides pass/fail — not my eyes scanning the diff. The mechanics lean entirely on Node's built-in child_process module, so there are zero dependencies to audit.

// sandbox-run.mjs — executes one generated snippet with hard boundaries.
// Run this ON the sandbox server, never on your dev machine.
import { spawn } from "node:child_process";
import { mkdtemp, writeFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";

const TIMEOUT_MS = 5_000;

export async function runInSandbox(code) {
  const dir = await mkdtemp(join(tmpdir(), "aigen-"));
  const entry = join(dir, "snippet.mjs");
  await writeFile(entry, code);

  const result = await new Promise((resolve) => {
    const child = spawn("node", [entry], {
      cwd: dir,                        // confined working directory
      env: { PATH: process.env.PATH }, // scrubbed env: no tokens, no keys
      stdio: ["ignore", "pipe", "pipe"],
    });

    let stdout = "", stderr = "";
    child.stdout.on("data", (d) => (stdout += d));
    child.stderr.on("data", (d) => (stderr += d));

    const killer = setTimeout(() => {
      child.kill("SIGKILL");
      resolve({ status: "timeout", stdout, stderr });
    }, TIMEOUT_MS);

    child.on("close", (code) => {
      clearTimeout(killer);
      resolve({ status: code === 0 ? "ok" : "error", exitCode: code, stdout, stderr });
    });
  });

  await rm(dir, { recursive: true, force: true });
  return result;
}

And the evaluation side — a fixed table of test cases that I write before looking at any model output, so I don't bend the criteria to fit what the model produced:

// eval.mjs — one prompt, many runs, fixed assertions.
import { runInSandbox } from "./sandbox-run.mjs";

const cases = [
  { name: "sorts numbers ascending", input: [3, 1, 2], expect: [1, 2, 3] },
  { name: "handles empty array",     input: [],         expect: [] },
  { name: "does not mutate input",   input: [2, 1],     expect: "no-mutation" },
];

async function evaluate(generatedCode) {
  const wrapped = `
    import fs from "node:fs";
    const input = JSON.parse(fs.readFileSync(0, "utf8"));
    ${generatedCode}
    console.log(JSON.stringify(solution(input)));
  `;
  const results = [];
  for (const c of cases) {
    const r = await runInSandbox(wrapped);
    results.push({ case: c.name, ...r });
  }
  return results;
}

// Feed in code produced by whatever model you're evaluating.
const codeFromModel = process.argv[2]
  ? await (await import("node:fs/promises")).readFile(process.argv[2], "utf8")
  : `const solution = (a) => [...a].sort((x, y) => x - y);`;

console.table(await evaluate(codeFromModel));

Two deliberate design choices

Assertions before generation. I write the test table first, paste the task into the model (via MonkeyCode's free model access in my runs, but any model works), save the raw output to a file, and only then run node eval.mjs output-attempt-1.js. This turns "looks right to me" into "passed 3/3 with zero mutations."
The temp dir is the blast radius. The scrubbed env means even deliberately hostile output can't read my credentials, because there are none in scope on the sandbox box. Combined with cwd pinning, the $HOME recursion bug from my opening anecdote physically cannot escape the scratch directory.

When this harness is worth it — and when it isn't

Situation	Use the sandbox harness?	Why
Evaluating a model's output quality across many runs	Yes	Reproducible pass/fail beats vibes
Generated code touches the filesystem, shell, or network	Yes	This is exactly the failure mode that hurts
Quick syntax question, code never executed	No	Just read it; a sandbox adds nothing
Output needs real secrets or production data to be meaningful	No	Scrubbed env makes the test unrealistic — test manually in staging instead
Code needs OS-level isolation (untrusted dependencies, install scripts)	Not enough	A subprocess is not a security boundary; add a real container/VM layer

Limitations, stated plainly

A child process is not a security sandbox. It stops accidents, not adversaries. If you're running code from an agent loop that fetched instructions from the internet, add container isolation, a read-only filesystem, and network egress rules. My harness is seatbelts, not an airbag system.
The free tier is a convenience, not a guarantee. Free model access and free server options change over time; treat cost as zero today, design so you can swap providers tomorrow. That's why the harness has no provider-specific code in it.
Pass/fail tables can be gamed by the model. If the task leaks into a model's training data, high scores may reflect memorization. Rotate in novel test cases — contamination is a documented problem in benchmark evaluation, as covered in NIST's guidance on AI risk management.
This measures one narrow thing — whether output runs correctly under constraints. It says nothing about readability, maintainability, or whether you should ship it.

Who should skip this — and what to do tonight

If you review every generated line before running anything and your tasks are low-risk, the overhead isn't justified. If you already have CI-based eval infrastructure, you don't need my 60-line version. This workflow is for the gap in between: people pasting model output straight into a terminal on their main machine and hoping.

If you're in that gap, the smallest useful step isn't adopting any particular platform — it's moving execution off your laptop tonight and writing three assertions before your next prompt. Copy the two files above onto any throwaway box, run node eval.mjs against your next model output, and see what the pass/fail table tells you that your eyes didn't. If you want a zero-cost place to try that loop, the free model access and free server from MonkeyCode are what I used for the runs above; the harness itself will follow you to whatever you use next.

Deciding If AI Code Review Fits Your Repo: A Throwaway Experiment That Costs Nothing

Quinn Li — Wed, 05 Aug 2026 07:07:29 +0000

Buying an AI review tool to find out whether AI review works on your codebase is backwards. The sensible order is: define what "works" means, run a cheap experiment against code you already understand, then spend money only if the numbers justify it.

This article is that experiment. It's a deliberately disposable setup — a Node script, a handful of your own historical pull requests, and a spreadsheet-style labeling pass — hosted on free infrastructure so the total cost of answering the question stays at zero.

Start with a claim you can disprove

"Does AI code review help?" is not testable. This is:

For our typical PRs, does a general-purpose model produce at least one finding a human reviewer would act on, per PR, without flooding us with comments we'd immediately dismiss?

Two numbers fall out of that framing: actionable findings per PR and noise ratio. Everything below exists to measure those two things and nothing else. If you skip this step, you'll finish the experiment with vibes instead of a decision.

Why free infrastructure is enough for this

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The experiment needs two things: model access and a place to run the script. MonkeyCode's free model access and free server option cover both, which is why I'm using them here — but the script talks to any OpenAI-compatible endpoint, so nothing below is locked to one provider. Point the base URL elsewhere and it runs unchanged.

Free tiers are the right tool for an evaluation specifically because the experiment is short-lived. You're not building production capacity; you're buying information.

The review runner

// pr-probe.mjs — usage: node pr-probe.mjs <path-to-diff>
import { readFileSync, writeFileSync, existsSync } from 'node:fs';

const target = process.argv[2];
if (!target) {
  console.error('need a diff file');
  process.exit(1);
}

const patch = readFileSync(target, 'utf8');
const changedLines = patch.split('\n').filter(l => /^[+-]/.test(l) && !/^[+-]{3}/.test(l)).length;

if (changedLines > 250) {
  console.error(`skipping: ${changedLines} changed lines is out of scope for this probe`);
  process.exit(1);
}

const system = `You review code diffs for a senior engineer.
Respond with JSON only — an array where each element is:
{"path": string, "approx_line": number, "kind": "defect"|"hazard"|"style", "note": string}
Rules: one sentence per note. Flag only things that would change behavior,
corrupt data, leak secrets, or break under load. Never repeat a linter.
Empty array if clean.`;

const response = await fetch(`${process.env.PROBE_BASE_URL}/v1/chat/completions`, {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.PROBE_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: process.env.PROBE_MODEL,
    temperature: 0,
    messages: [
      { role: 'system', content: system },
      { role: 'user', content: patch },
    ],
  }),
});

if (!response.ok) {
  console.error(`API error ${response.status}: ${await response.text()}`);
  process.exit(1);
}

const payload = await response.json();
const raw = payload.choices?.[0]?.message?.content ?? '[]';

const record = {
  source: target,
  changed_lines: changedLines,
  model_reply: raw,
  captured_at: new Date().toISOString(),
};

const logFile = 'probe-log.jsonl';
writeFileSync(logFile, (existsSync(logFile) ? readFileSync(logFile, 'utf8') : '') + JSON.stringify(record) + '\n');
console.log(raw);

Two deliberate choices worth noting:

Temperature 0. You want run-to-run consistency so disagreements between runs mean something. A creative reviewer is the last thing you need when measuring.
Raw output is logged verbatim. You label it by hand afterward. Automating the judgment step at this stage would just move the unverified assumption somewhere harder to see.

Generating ground truth from your own history

The clever part of this experiment is that you don't need new code. Your merged PRs from last month already contain the answer key: whatever human reviewers caught, plus whatever slipped into production and got fixed later.

mkdir -p patches
git log --since="6 weeks ago" --merges --format=%H -12 | while read sha; do
  git diff "${sha}^1" "$sha" > "patches/${sha:0:10}.patch"
done

for p in patches/*.patch; do
  node pr-probe.mjs "$p"
  sleep 3   # stay polite to free-tier rate limits
done

For each PR, jot down before you look at the model output: what did the human reviewer flag, and did anything from this PR cause a follow-up fix? That's your baseline. Looking at model output first will contaminate your memory of what humans actually said.

Labeling and the decision rule

Open probe-log.jsonl and tag every model finding:

Tag	Definition	Effect on verdict
`hit`	A human would have acted on this	Counts as value
`noise`	Wrong, unverifiable, or pure taste	Counts against
`echo`	Duplicate of another finding in the same PR	Counts against
`gap`	Something the follow-up fix proved was real, that the model missed	Weakens the case

Set your thresholds now, before labeling. Mine were: proceed to a paid pilot only if hit / total PRs >= 0.5 and noise <= hits. Yours should reflect how patient your reviewers are — a team of two tolerates more noise than a team of twenty. Writing thresholds down first is the whole trick; otherwise you'll grade on a curve and "validate" whatever happened.

Running it on the free server

For a batch job over twelve historical PRs, a laptop is fine. Where the free server option earns its place is the next step: a tiny webhook listener that probes each newly opened PR for a week, so you're evaluating live diffs rather than archaeology.

Keep the deployment stateless — append results to a gist or a remote JSONL file — because the honest end state of this experiment is often "delete everything." If teardown takes more than a minute, you've accidentally built infrastructure instead of running an experiment.

What this probe cannot tell you

Nothing about latency or throughput. Free-tier performance is not a preview of a paid plan. Don't time these requests and extrapolate.
Nothing about large changes. The 250-line cutoff is a scope guard, not a suggestion. Big refactors need chunked context and per-file passes, which this script deliberately doesn't attempt.
Nothing about model choice. One prompt, one model, temperature zero. You're testing the workflow hypothesis on your diffs, not shopping for the best reviewer.
Whether your org allows it. Check code-handling policy before sending real diffs anywhere. If the answer is murky, run the probe against a toy repo and accept the weaker signal.

When to skip this entirely

If AI review already runs in your CI and you're tuning comment volume, this is a step backward — you have production data, use that instead. Also skip it if your diffs are dominated by generated code, lockfiles, or migrations; diff-only review degrades badly there and the experiment will tell you something false.

The actual point

The deliverable of this exercise isn't a configured tool. It's a sentence you can say in a planning meeting: "On our last twelve PRs, a model found N things humans would have acted on, at a noise level we'd tolerate" — or the opposite, which is equally valuable and much cheaper than learning it from an invoice. If you want to run that loop without opening a billing page, MonkeyCode's free model and server tier is one convenient substrate; the probe itself doesn't care where it points.