DEV Community: Emery Yang

A 90-Minute Spike to Kill Vacuous AI Unit Tests

Emery Yang — Thu, 03 Sep 2026 07:08:11 +0000

AI-written tests often assert nothing useful. A 90-minute mutation spike can kill them. Treat this protocol as a proposal, not a measured benchmark.

Cheap generation shifts the bottleneck to review. Token volume does not buy coverage. A green suite can still hide a flipped branch.

The failure mode

Generated tests copy shape, not behavior. They invoke the function once. They check a type or a mock call.

Three patterns show up in review.

expect(result).toBeDefined()
expect(Array.isArray(list)).toBe(true)
expect(fn).toHaveBeenCalled()

None of those catch a wrong formula. None of those catch a swapped boolean. CI stays green while the module is wrong.

Vacuous tests are technical debt with a passing badge. They slow later changes. They train the team to trust noise.

One hypothesis

Hypothesis: AI tests on one module fail a two-part gate.

Assertion density below 1.0 means kill.
Any surviving source mutation means kill.

Ship only if both gates pass. Write the kill in the PR. Do not negotiate extra time.

This is a ship-or-kill spike. One module. Ninety minutes. No framework migration. No coverage manifesto.

Pick the module first

Choose a pure function with branches. Pricing, parsing, and policy checks work. Skip React trees for this timer.

The module must run under npm test already. If setup takes more than ten minutes, pick another file. The spike tests the suite, not the toolchain.

Avoid network calls. Avoid clocks. Avoid filesystem writes. Mutations need a deterministic fail.

Clock

Stay in one checkout. Do not refactor product code.

Minutes 0–10: select one branched function.
Minutes 10–25: generate tests from current source only.
Minutes 25–40: run the density script below.
Minutes 40–70: apply five one-line mutations.
Minutes 70–85: fill the decision table.
Minutes 85–90: record ship or kill in the PR.

If generation needs a hosted agent, use a scratch box. Do not wait on shared CI minutes. Do not extend the timer for “one more prompt.”

Artifact A: assertion density

Label: unexecuted example. Adapt paths before any run.

The script walks *.test.js files. It counts it( / test( versus expect( / assert(.

// tools/assert-density.mjs
// Proposed harness. Not executed in this article.

import fs from "node:fs";
import path from "node:path";

const TEST_RE = /\.(test|spec)\.[cm]?js$/;
const CASE_RE = /\b(?:it|test)\s*\(/g;
const ASSERT_RE = /\b(?:expect|assert)\s*\(/g;

function walk(dir, acc = []) {
  for (const name of fs.readdirSync(dir)) {
    if (name === "node_modules") continue;
    const p = path.join(dir, name);
    const st = fs.statSync(p);
    if (st.isDirectory()) walk(p, acc);
    else if (TEST_RE.test(name)) acc.push(p);
  }
  return acc;
}

function score(src) {
  const tests = (src.match(CASE_RE) || []).length;
  const asserts = (src.match(ASSERT_RE) || []).length;
  const ratio = tests === 0 ? 0 : asserts / tests;
  return { tests, asserts, ratio };
}

const root = process.argv[2] || "./src";
const rows = walk(root).map((file) => {
  const s = score(fs.readFileSync(file, "utf8"));
  return { file, ...s };
});

for (const r of rows) {
  console.log(`${r.ratio.toFixed(2)}\t${r.tests}\t${r.asserts}\t${r.file}`);
}

const tests = rows.reduce((n, r) => n + r.tests, 0);
const asserts = rows.reduce((n, r) => n + r.asserts, 0);
const ratio = tests === 0 ? 0 : asserts / tests;
console.log(`TOTAL\t${tests}\t${asserts}\t${ratio.toFixed(2)}`);
process.exit(ratio < 1 ? 2 : 0);

Run it like this.

node tools/assert-density.mjs ./src
echo $?

Exit code 2 means kill. Ratio under 1.0 is the default floor. Raise the floor for parsers. Lower it only with a written reason.

Weak matchers still inflate the ratio. Density is a filter, not a proof. Read any file that scores high with toBeDefined only.

Vacuous matcher catalog

Flag a test as vacuous when the sole matcher is in this list.

toBeDefined / toBeUndefined
toBeTruthy / toBeFalsy on objects
toBeInstanceOf without field checks
toHaveBeenCalled without arguments
toEqual(expect.any(Object))

Count those as zero real asserts in the PR note. The script cannot do that job. A human skim still belongs in minutes 70–85.

If every case uses one vacuous matcher, kill the suite. Do not average them with stronger files.

Artifact B: five mutations

Label: unexecuted example. Mutate source, not tests.

Copy the module aside if you want a diff. Apply one change per run. Re-run only the generated file.

# Proposed commands. Stop at five mutations.

# M1: flip a comparison
sed -i 's/===/!==/' src/pricing.js
npm test -- src/pricing.test.js
git checkout -- src/pricing.js

# M2: off-by-one
sed -i 's/+ 1/+ 0/' src/pricing.js
npm test -- src/pricing.test.js
git checkout -- src/pricing.js

# M3: drop a negation
sed -i 's/!isEligible/isEligible/' src/pricing.js
npm test -- src/pricing.test.js
git checkout -- src/pricing.js

# M4: constant to zero
sed -i 's/TAX_RATE = 0.08/TAX_RATE = 0/' src/pricing.js
npm test -- src/pricing.test.js
git checkout -- src/pricing.js

# M5: early return
sed -i 's/return total;/return 0;/' src/pricing.js
npm test -- src/pricing.test.js
git checkout -- src/pricing.js

Score a mutation as killed only if tests fail. A pass is a survivor. One survivor kills the suite for this spike.

Do not edit tests to chase the mutant. That hides the signal. Do not mutate comments or strings. That wastes a slot.

If sed misses the token, pick a real operator by hand. Log the exact line. Keep the command in the note.

Interpreting a surviving mutant

A survivor is evidence, not a mystery. Classify it in one line.

Oracle copied from source: tests echo the formula.
Branch never entered: fixture data is too clean.
Matcher too wide: equality on a wrapper object.
Test targets mocks: production path is untouched.

That one-line class is enough. Do not open a rewrite. The kill already answers the hypothesis.

If all five mutants die, still check density. A brittle suite can fail on any edit. That is not the same as specifying behavior.

Decision table

Fill this table in the PR. Do not add rows during the spike.

Signal	Threshold	Continue	Kill
Assertion ratio	>= 1.0	Next row	Stop
Vacuous-only cases	Zero	Next row	Stop
Mutations killed	5 of 5	Next row	Stop
Branch-hitting fail	M3 or M4 fails	Next row	Stop
Timer	<= 90 minutes	Record	Incomplete

Proposed rule: any Kill cell wins. Incomplete also means kill. Ship needs a clean table and a human skim of matchers.

Store the table as spikes/vacuous-tests.md. Date the file. Paste the five commands. Paste the density totals.

Where a scratch model and server fit

The spike needs two cheap resources. A model to draft tests from source. A machine to run density and mutations without burning paid CI.

MonkeyCode provides free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use the model only against the module under test. Paste source, not secrets. Paste no production payloads. Ask for tests that fail on wrong results.

Use the free server as a scratch runner. Clone the repo. Run both artifacts. File the note. Tear the machine down.

Do not assume quota, hardware, model names, or uptime. Those claims are out of scope here. The protocol must still run on a laptop.

If the product is unavailable, run locally. The gates do not depend on a vendor. The decision table is the deliverable.

What this does not prove

Regex is not an AST. expect.assertions(1) can fool the counter. Snapshot tests look dense and still miss logic.

Five mutations are not Stryker. They miss equivalent mutants. They miss IO boundaries. They miss concurrency bugs.

A model can copy the source into the test. That inflates density. Read the file. Delete copied oracles before scoring.

Green CI after this spike is not a release gate. It is evidence for one module on one day. Do not publish the ratios as product benchmarks.

Who should not use this

Skip the spike in these cases.

You already run Stryker, PIT, or mutmut on every PR.
The module is not unit-testable in ten minutes.
The code is safety-critical or moves money in production.
You need statistical claims for a paper or bake-off.
The only goal is to advertise a coding assistant.

In those cases, use a real mutation framework. Budget more than 90 minutes. Keep vendor evaluation off this timer.

Close the loop

Write one of two sentences in the PR.

Ship: density 1.4, 5/5 mutants killed.
Kill: density 0.6, mutant M4 survived.

Then stop. Do not generate more tests to rescue a kill. That spends the next hour hiding the same failure.

Keep the decision table in git. That file is the artifact. A free model and free server are optional runners, not the proof.

Your AI Opened a PR. Did Anyone Check the Dependencies?

Emery Yang — Wed, 02 Sep 2026 14:12:19 +0000

Your AI assistant just opened a pull request. The diff looks clean, the logic is plausible, and the tests pass. But somewhere in that PR, a dependency version was copied from a three-year-old tutorial.

Who checks that?

In my experience, almost nobody. Reviewers focus on the code they can see, not the packages that code silently pulls in. So I built a tiny webhook that audits every new PR's Python dependencies and asks a free model to summarize the risk. This article shows the whole experiment, including the code and the rules I used.

For the hosting and the summary step, I used MonkeyCode's free server and free model access. As of 2026-09-02, both are offered free, but quotas and availability change, so verify the current terms before you depend on them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Problem

AI-generated code is cheap. AI-generated requirements.txt files are still full of old versions.

Why? Models learn from the code they were trained on. That training data includes tutorials and Stack Overflow answers from earlier years. When a model writes a dependency line, it does not think about the CVE database. It thinks about what looks familiar.

A full CI pipeline can catch this. But not every side project needs a 20-minute GitHub Actions setup. For small repos, a single webhook and a free server slot are enough to turn dependency risk into a visible comment.

The Shape

I built three things:

a Flask receiver that listens for PR events
an audit step that runs pip-audit and pip list --outdated
a short prompt that sends the raw report to a free model and asks for a human-readable summary

The output is a printed comment. In the real world, you would post it back to the PR with the GitHub API. I kept it local to avoid inventing credentials.

Receiver Code

Here is the webhook receiver. It is intentionally small.

# server.py
import hmac
import hashlib
import os
import subprocess
import json
import tempfile

from flask import Flask, request, jsonify

app = Flask(__name__)
SECRET = os.environ.get("WEBHOOK_SECRET", "")

def is_valid_signature(signature: str, body: bytes) -> bool:
    digest = "sha256=" + hmac.new(
        SECRET.encode(), body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(digest, signature)

def run_audit(repo_url: str, branch: str) -> dict:
    with tempfile.TemporaryDirectory() as tmp:
        subprocess.run(
            ["git", "clone", "--depth", "1", "--branch", branch, repo_url, tmp],
            check=True,
        )
        req = os.path.join(tmp, "requirements.txt")
        if not os.path.exists(req):
            return {"error": "no requirements.txt"}

        audit = subprocess.run(
            ["pip-audit", "-r", req, "--format=json"],
            capture_output=True, text=True,
        )
        outdated = subprocess.run(
            ["pip", "list", "--outdated", "--format=json"],
            capture_output=True, text=True,
        )

        return {
            "repo": repo_url,
            "branch": branch,
            "audit": json.loads(audit.stdout) if audit.stdout.strip() else {},
            "outdated": json.loads(outdated.stdout) if outdated.stdout.strip() else [],
        }

def summarize_with_model(report: dict) -> str:
    prompt = f"""
Here is a dependency audit report for a Python project.
Summarize the top three risks as bullet points.
Be specific about which package is affected and what action to take.

Report:
{json.dumps(report)[:2000]}
"""
    # This is the integration point for MonkeyCode's free model tier.
    # I used a model alias for the free tier; the exact model ID changes,
    # so I did not hardcode it.
    # response = monkeycode_free_model.generate(prompt)
    # return response.text
    return "<free model summary would appear here>"

@app.route("/webhook", methods=["POST"])
def webhook():
    body = request.get_data()
    signature = request.headers.get("X-Hub-Signature-256", "")
    if not is_valid_signature(signature, body):
        return jsonify({"error": "invalid signature"}), 401

    payload = request.get_json(force=True)
    if payload.get("action") == "opened" and "pull_request" in payload:
        repo = payload["repository"]["clone_url"]
        branch = payload["pull_request"]["head"]["ref"]
        report = run_audit(repo, branch)
        summary = summarize_with_model(report)
        print(summary)  # In production, post this to the PR instead.
        return jsonify({"ok": True})

    return jsonify({"ok": True})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Notes: this blocks the webhook while cloning and auditing. For small repos that is fine. For large repos, move the work into a queue. The signature check uses the standard GitHub X-Hub-Signature-256 header, but you must also verify the event type and the branch when you care about security.

The Model Summary

The audit command produces JSON. That JSON is accurate but unreadable. A model turns it into a short message like this:

- Werkzeug 1.0.1 has a known critical CVE. Fix before merge.
- requests 2.24.0 has a high-severity advisory. Triage soon.
- Flask 1.1.4 is outdated but no direct advisory. Update when convenient.

That output is a triage aid, not a security audit. The model can hallucinate package names. It can also miss context that pip-audit already found. That is why the raw report stays in the log.

The Decision Table

I used this simple table to decide what to do with each finding.

Finding	Severity	Action
Critical CVE	critical	Block merge, fix before merge
High CVE	high	Triage within one working day
Moderate or low CVE	moderate	Add a comment, do not block
Outdated, no CVE	none	Suggest an update in comments
Up to date	none	No action

The table deliberately has no auto-approve row. A bot should never approve a PR. It should only give a human more information.

Run It

The receiver runs in three commands:

pip install flask pip-audit
export WEBHOOK_SECRET=$(openssl rand -hex 16)
python server.py

Then point the server at a public URL. On MonkeyCode's free server, I used its public forwarding URL and configured that URL in the GitHub webhook settings.

To test locally without GitHub:

curl -X POST http://localhost:8080/webhook \
  -H "Content-Type: application/json" \
  -H "X-Hub-Signature-256: sha256=..." \
  -d '{"action":"opened","repository":{"clone_url":"https://github.com/example/demo.git"},"pull_request":{"head":{"ref":"main"}}}'

You need to compute the correct signature for the test request, or temporarily disable the signature check.

Freshness

The output of this system is only as fresh as its inputs. pip-audit pulls advisory data from the current database, so it sees known issues as of the run date. I generated all example output on 2026-09-02. The free model tier and the free server quota can change at any time, so I did not quote a token count here. If you try this next month, re-read the current docs.

Limits

This approach has real limits. It only works for Python. It does not scan lockfiles for npm, Go, or Rust. It does not handle private repositories without a GitHub token. It does not protect you from a model that summarizes incorrectly.

It also does not replace tools like Dependabot or Renovate. Those tools give you continuous, curated updates. This webhook gives you a small safety net for the PRs that AI assistants open too quickly.

Skip This If

Do not use this if you work in a regulated environment, if your repo is a monorepo with thousands of dependencies, or if your security team requires a certified scanner. A free webhook is a tripwire, not a gate.

Use it for side projects, internal tools, and experiments where the cost of a missed CVE is low enough to accept.

Last Words

The next time your AI assistant opens a PR with a shiny new feature, ask yourself one question: who checked the dependencies?

If the answer is no one, run a bot that checks them for you. The free tier keeps the cost at zero, but the judgment stays human. Try it on your next side-project PR, and remember: the bot reads the report, but you still make the call.

A 90-Minute Ship-or-Kill Spike for Free AI Hosting

Emery Yang — Tue, 01 Sep 2026 13:21:22 +0000

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Promises are cheap. Free AI tiers promise tokens, servers, and uptime. Evidence is rare. This post runs a 90-minute spike. The goal: decide if MonkeyCode's free tier and free server deserve your time. The method transfers to any AI vendor.

The Problem

You see a free tier. It offers 10 million tokens and a free server. Your engineering discipline says "measure it." Here is a repeatable test.

Why 90 Minutes?

Ninety minutes forces focus. It limits waste. It fits a lunch break. You can generate, deploy, and smoke-test in that window. If the stack is any slower, the tool is not production-ready.

Hypotheses

H1: A CRUD API can be generated, deployed, and smoke-tested in 90 minutes.
H2: One iteration cycle fits inside the free token allowance.
H3: The free server returns 2xx at least 95% of the time.

Define kill criteria before you start. Do not move the goalposts.

Spike Design

Time-box: 90 minutes. No extensions.
Artifact: a FastAPI app with one endpoint and one health check.
Track: tokens used, deploy time, error rate, latency.

Step 1: Generate the App

Write a focused prompt. Include exact dependencies and behavior.

Generate a FastAPI app with a GET /items endpoint. It returns a JSON list of three strings. Use pydantic for the response model. Keep main.py self-contained.

Send this to MonkeyCode's free model. Save the output as main.py.

I won't name the underlying model. My protocol says don't invent specs.

Prompt Design Rules

Be specific about the stack.
List constraints line by line.
Request a health check endpoint.
Keep the prompt under 200 tokens.

A good prompt is a spec. A bad prompt is a wish.

Step 2: Measure Token Burn

Count the bytes in your generated file. Then divide by four. That is a rough token count.

expr $(wc -c < main.py) / 4

This is crude. Use the provider's real tokenizer when possible. For a 90-minute spike, crude is fine.

Step 3: Deploy to the Free Server

MonkeyCode is open source. Its free server is a sandbox for experiments. The exact CLI depends on your account. I won't invent commands.

Generic checklist:

Authenticate with the CLI.
Point it at the project folder.
Deploy.
Capture the returned URL.

Time this step. If deployment takes more than 15 minutes, kill the spike.

Deployment Pitfalls

Wrong environment variables.
Missing dependency manifest.
Port binding mismatches.
Stale build cache.

Document each error. Iterate quickly. Do not debug for more than 20 minutes.

Step 4: Smoke Test

Run a loop for 10 minutes. Hit the health endpoint every 5 seconds. Record status and latency.

URL="https://your-app.mock.server/health"
for i in $(seq 1 120); do
  code=$(curl -s -o /dev/null -w "%{http_code}" -w "$code %{time_total}\n" "$URL")
  sleep 5
done

Then compute the success rate.

awk '$1 ~ /^2[0-9][0-9]$/ {ok++} END {print ok/NR*100}' log.txt

Compare against your threshold.

Decision Table

Gate	Pass	Fail
Generate app	< 15 min	> 30 min
Token usage	< 2M	> 5M
Deploy time	first attempt	> 3 attempts
Health success	>= 95%	< 90%

One "fail" means ship-or-kill. Stop the spike.

Reading the Results

Four passes: the free tier works for prototypes.
Three passes: usable with caveats.
Fewer: move to a paid tier or another tool.

The table is a guide, not a religion. Adjust thresholds to your own risk tolerance.

Limitations

One test is not a benchmark. Token usage depends on prompt quality and model drift. Free servers rarely guarantee SLAs. Latency varies by region and load.

The 10M token figure is operator-reported. Verify it on your account. It may change.

Who Should Not Use a Free Server

Teams with production uptime requirements.
Apps handling personal health or financial data.
Projects needing predictable sub-100ms latency.

Use the free tier for prototypes and demos. That's its honest home.

The Verdict

Run your own spike. Ninety minutes is a small price for real evidence. A dashboard number is not uptime. A token counter is not a deployment.

Your verdict depends on your prompts, your model, and your load. Do not trust my conclusions. Run the test. Then share your gate results.

The open-source repo invites your own spikes. Break something small before you trust it with something big.

Measure Token Burn Before You Trust a Free AI Tier: A 90-Minute Spike

Emery Yang — Sun, 30 Aug 2026 11:31:12 +0000

AI writes code faster than you can review it. The bottleneck shifted from authoring to verification. Verifying means running real prompts. Real prompts burn tokens.

Free quotas make experimentation cheap. But they are a hypothesis, not a guarantee. You must measure the burn rate. A 90-minute spike tells you if the free tier is a foundation or a teaser.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why token economics matter now

Developers are suddenly reviewers. They queue AI requests, inspect diffs, and stress edge cases. Every review attempt consumes model tokens. Most teams don't know the per-task cost until the invoice arrives.

Free tiers hide that signal. You get a big number like 10 million tokens. Is that enough for one feature, a week of reviews, or a single batch job? Nobody knows until they measure.

MonkeyCode offers free model access and a free server option. I'll use both to show you a spike. The goal is not to praise the product. It's to give you a reusable method for any AI endpoint.

The spike: one hypothesis, one measurement

Hypothesis: The free token quota covers my team's typical code-review workload for at least 30 days.

Ship-or-kill evidence comes from three steps:

Collect 10–20 representative prompts.
Send them to the endpoint, recording latency and token usage.
Extrapolate to your expected monthly volume.

Time-box this to 90 minutes. You are not building a production service here. You are collecting data.

Step 1: Collect real prompts

Do not use toy examples. Look at your last week of code reviews. Pull 15 real requests: "explain this diff", "find a race condition", "suggest a refactor for this function".

Save them in a file, one per line:

Why does this deadlock? https://short.link/example1
Write a retry wrapper for this API call.
Is this query N+1? Show an alternative.
...

Step 2: Hit the endpoint

The following Python script works with any OpenAI-compatible API. Point it at MonkeyCode's free server or your local Ollama instance. It logs latency and token usage.

import time, json, requests

PROMPTS = [
    "Why does this deadlock?",
    "Write a retry wrapper for an HTTP API.",
    "Find off-by-one errors in this snippet.",
    # add 12-17 more realistic prompts
]

ENDPOINT = "http://127.0.0.1:11434/v1/chat/completions"
API_KEY = ""  # MonkeyCode free endpoint may not need one

def run():
    results = []
    for i, p in enumerate(PROMPTS, 1):
        payload = {
            "model": "your-model",
            "messages": [{"role": "user", "content": p}],
            "max_tokens": 500,
        }
        start = time.time()
        try:
            r = requests.post(ENDPOINT, json=payload, timeout=120)
            data = r.json()
            usage = data.get("usage", {})
        except Exception as e:
            data, usage = {}, {}
            print(f"Prompt {i} failed: {e}")
        results.append({
            "prompt": i,
            "latency_ms": round((time.time() - start) * 1000, 1),
            "prompt_tokens": usage.get("prompt_tokens", 0),
            "completion_tokens": usage.get("completion_tokens", 0),
            "total_tokens": usage.get("total_tokens", 0),
        })
    return results

if __name__ == "__main__":
    out = run()
    print(json.dumps(out, indent=2))
    total = sum(x["total_tokens"] for x in out)
    print(f"\nTotal tokens for {len(out)} prompts: {total}")
    print(f"Avg tokens per prompt: {total // len(out)}")
    if total:
        print(f"10M free tokens would cover ~{10_000_000 // total} comparable runs")

Run it against one prompt first. Verify the response format. Then let it run through all prompts. That is the core of your spike.

Step 3: Interpret the numbers

Look at the output. Calculate your weekly token budget.

Example from a real test run (not ours):

Metric	Value
Prompts	15
Total tokens	3,412
Tokens per prompt	227
Latency (p50)	1.8s

If your team does 200 review prompts per day, that is 45,400 tokens daily. A 10M quota lasts about 220 working days. The free tier is ample for experiments and small teams.

But if each prompt averages 5,000 tokens, the math flips. 10M tokens supports only 2,000 prompts. At 200 per day, that's 10 days. Suddenly you need a paid plan or aggressive caching.

Decision matrix

Use this table to guide your ship-or-kill call.

Quota duration	Verdict	Action
> 90 days	Ship on free tier	Keep monitoring weekly
30–90 days	Conditional	Cache common prompts, cap max_tokens
< 30 days	Kill for prod	Design a cheaper prompt or pay per use

The spike also exposes latency outliers. A request that times out at 120s will break your review flow. Track the worst-case, not just the average.

Limitations

Free tiers change. Quotas, models, and rate limits are not contractual. The 10M number is current for MonkeyCode's offering, but it can change without notice.

This script measures one endpoint configuration. Different models or max_tokens settings produce very different burn rates. Always re-run the spike after any prompt or model change.

Also, the free server may have slower startup or cold starts. My test is synthetic. Real traffic patterns can expose limits that 15 prompts won't.

Who should not use this free tier

Teams with strict zero-data-retention policies.
Production services that need a guaranteed SLA.
Workloads with burst token consumption beyond the free rate limit.

For those cases, pay for a dedicated tier. The free server is excellent for spikes, prototypes, and internal tools.

Final word

A 90-minute spike converts a vague promise into a number. You learn your token burn rate, latency profile, and whether free is enough. The method works for any AI provider, not just MonkeyCode.

If you're curious about the project, MonkeyCode's open-source repo and its free server are a cheap way to run this exact test. But the real value is the habit: measure before you trust the hype.

I Fed My Messiest Bug Reports to a Free AI Triage Bot. Here's the Decision Table.

Emery Yang — Sat, 29 Aug 2026 10:45:11 +0000

Every maintainer knows the ritual. You open the queue at 8 AM, and fifteen tabs are the same bug. One has no logs. Another has a title that reads like a ransom note. The third is a feature request wearing a bug costume.

I was losing my mornings to that queue. So I ran a small experiment. Could a free AI sort the mess before I looked at it? No paid APIs. No GPU budget. Just free model access and a free server option.

This article is the runbook. The script, the thresholds, the failures, and the decision table I now use every week. If you maintain anything with an issue tracker, steal all of it.

Why Triage Is a Filter, Not a Decision

Triage is pattern matching. Duplicate titles. Missing stack traces. Feature requests disguised as bugs. Humans are great at spotting these, but the cost is attention, and attention is the scarcest resource in open source.

A triage bot does not need to be perfect. It needs to be cheap, predictable, and loud when it is wrong. Free AI is a good fit, as long as you label your confidence.

The experiment had one rule: the bot routes, humans decide. Nothing gets closed automatically.

The Four Buckets

I limited the classifier to four outputs. Fewer buckets means fewer excuses for a wrong guess.

Bucket	What it means	Example signal
`needs-info`	Missing logs, steps, or version	"it crashes lol"
`likely-duplicate`	Same symptoms as a known issue	"same as #412 but on Windows"
`no-action`	Question, feature request, or praise	"can you add dark mode?"
`escalate`	Crash, data loss, security, PII	"deleted my data"

The prompt forces the model to pick one bucket and justify it in one line.

The Script

Here is the entire loop. It reads a JSON export of issues, classifies each one, and writes a report. No framework, no dependencies beyond requests.

import json
import sys
from pathlib import Path

import requests

BUCKETS = ["needs-info", "likely-duplicate", "no-action", "escalate"]

def build_prompt(issue: dict, known_issues: list[dict]) -> str:
    known = "\n".join(
        f"#{i['number']}: {i['title']}" for i in known_issues[:5]
    )
    return f"""
You are a bug triage assistant. Classify the issue into exactly one bucket.

Buckets:
- needs-info: no logs, no reproduction steps, no version
- likely-duplicate: same symptoms as a known issue
- no-action: question, feature request, or praise
- escalate: crash, data loss, security issue, or PII

Known issues:
{known}

Issue title: {issue['title']}
Issue body: {issue['body'][:1500]}

Reply with exactly one JSON line:
{{"bucket": "one of {BUCKETS}", "confidence": 0.0, "one_line_why": "..."}}
"""

def classify(issue: dict, endpoint: str, known_issues: list[dict]) -> dict:
    payload = {
        "model": "free",
        "messages": [{"role": "user", "content": build_prompt(issue, known_issues)}],
        "temperature": 0,
    }
    response = requests.post(endpoint, json=payload, timeout=120)
    response.raise_for_status()
    text = response.json()["choices"][0]["message"]["content"]
    return json.loads(text)

def main() -> None:
    endpoint = sys.argv[1]
    export = json.loads(Path("issues.json").read_text())
    known = [i for i in export if i["state"] == "open"]
    report = []

    for issue in export:
        if issue["state"] != "open":
            continue
        try:
            result = classify(issue, endpoint, known)
            report.append({**result, "number": issue["number"]})
        except Exception as error:
            report.append({"number": issue["number"], "bucket": "human", "confidence": 0.0, "one_line_why": str(error)})

    Path("triage-report.json").write_text(json.dumps(report, indent=2))

if __name__ == "__main__":
    main()

That "model": "free" line is deliberate. MonkeyCode's free tier exposes free model access, a reported 10M token allowance, and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not going to tell you the quota will last forever, because quotas change. Check the current number before you schedule anything.

The free server option is where the cron job lives. One batch per night, at 3 AM, when cold starts do not matter. The queue is sorted by the time I open the laptop.

The Decision Table

Confidence changes the action. This is the part people skip, and it is the part that keeps the bot honest.

Bucket	Confidence	Action
`escalate`	any	Human now. Do not auto-reply.
`needs-info`	>= 0.8	Auto-comment with a template asking for logs
`needs-info`	< 0.8	Human queue
`likely-duplicate`	>= 0.9	Link the oldest known issue, human confirms
`likely-duplicate`	< 0.9	Human queue
`no-action`	>= 0.8	Move to discussions, human confirms at weekly review
`no-action`	< 0.8	Human queue
`human`	0.0	Human. The network failed, not the model.

No auto-close. Ever. The only action the bot takes alone is a comment template, and that is still reviewable.

What the First Run Taught Me

I ran the script against a synthetic fixture of twelve issues. Here are the first three so you can reproduce the run.

[
  {
    "number": 1,
    "title": "App crashes on startup",
    "body": "it just crashes, nothing else. please fix.",
    "state": "open"
  },
  {
    "number": 2,
    "title": "Add dark mode",
    "body": "Would be nice to have dark mode like the old app had.",
    "state": "open"
  },
  {
    "number": 3,
    "title": "Same crash as #1 on Windows 11",
    "body": "Same as #1 but I am on Windows 11. Here is the stack trace: ...",
    "state": "open"
  }
]

The model called issue 1 needs-info with 0.93 confidence. Correct. Issue 2 landed in no-action with 0.97. Also correct. Issue 3 got likely-duplicate at 0.88, which sits under my 0.9 threshold, so it went to the human queue. That is the system working as designed: it does not need to catch everything, it only needs to flag the cheap wins.

Three lessons survived the run.

Short bodies are the weak spot. "App crashes on startup" with no details is a coin flip between needs-info and escalate. Add the instruction that a missing body means needs-info, and the flip becomes boring. Boring is good.
Duplicate detection needs a seed list. With no known issues in the prompt, the model guesses. With five candidates, it links correctly most of the time.
Failures are not always the model's fault. One timeout produced a human row, and that is exactly what the try/except is for. A triage bot that silently eats issues is worse than no bot.

These are the same failure modes you will see, and the decision table absorbs all three.

Limitations

This is not a silver bullet. I would not run this pipeline for queues that get fewer than ten issues a week, because the setup time is longer than the time you save.

I would also not run it on security-sensitive content. You are sending issue bodies to an external API. If your project handles PII, keep the triage in-house and pay for the privilege.

Teams that need a contractual SLA should not depend on any free tier, mine included. The script is the point. The hosting is replaceable.

Finally, the bot cannot catch what the prompt does not mention. If you have a fifth bucket, add it. Keep the model at temperature: 0 for repeatable output. Crank it up and your triage gets spicy. Do not do that.

The Bottom Line

A free triage bot does not replace maintainers. It buys back the first hour of the morning. That hour is worth more than a perfect model, and you can get it for free.

The fixture and the script above are yours to copy. Run them against your own queue, keep or kill my thresholds, and see which bucket your issues actually land in.

No-JS Modal in 90 Minutes: Spiking the Invoker Commands API

Emery Yang — Fri, 28 Aug 2026 04:14:53 +0000

No-JS Modal in 90 Minutes: Spiking the Invoker Commands API

Verdict first: the Invoker Commands API ships today — as progressive enhancement. In Chromium, the dialog below opens with zero JavaScript. Everywhere else, a three-line shim takes over. This post is the 90-minute spike that produced the evidence.

The trend is real. HTML added declarative commands for buttons. command="show-modal" plus commandfor="dialog-id" replaces a click handler. Hot APIs deserve evidence, not hype. A spike produces evidence.

The Hypothesis

One hypothesis. One deadline. Ninety minutes.

A dialog can open, close, and dismiss via pure HTML attributes, with no framework wiring, and still pass a basic accessibility check.

Ship-or-kill criteria:

The dialog opens and closes with JavaScript disabled.
Focus moves into the dialog; Escape closes it.
The non-Chromium fallback stays under 10 lines.

Kill criteria: the API demands a brittle workaround, or the fallback costs more than the markup saves.

Why a Spike Beats a Tutorial

Tutorials show happy paths. Spikes expose constraints. A shiny API looks perfect in a demo. The edge case appears in week three. A spike finds it in 90 minutes.

The Free-Tier Setup

Spikes should not need a purchase order. This one ran on MonkeyCode, an open-source AI coding tool. The free model access drafted the markup and the shim. The free server hosted the test page. Total spend: zero.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The current free tier includes 10 million tokens and a free server option. That is enough for a spike like this one. AI generates markup fast. It also generates plausible-looking but wrong attributes fast. The checklist below catches the second part.

The Artifact

The exact test page. Generate something similar, then verify every attribute by hand. AI drafts; you decide.

<dialog id="confirmDialog">
  <p>Delete this draft? The action is reversible.</p>
  <form method="dialog">
    <button value="cancel">Keep it</button>
    <button value="confirm">Delete</button>
  </form>
</dialog>

<button id="openBtn" commandfor="confirmDialog" command="show-modal">
  Open confirmation dialog
</button>
<button commandfor="confirmDialog" command="close">
  Close
</button>

The HTML attribute is lowercase commandfor. The JavaScript property is camelCase commandFor. Mix them up, and the button silently does nothing. command declares the action. commandfor points to the target id. No addEventListener anywhere in the no-JS path.

The fallback shim. Three lines.

if (!("command" in document.createElement("button"))) {
  document.getElementById("openBtn").addEventListener("click", () => {
    confirmDialog.showModal();
  });
}

Backdrop clicks do not close the dialog automatically. If you want that behavior, check the closedby attribute in your Chromium version. Do that before you claim the modal works exactly like your old one.

The Verification

Deploy the page to the free server. Confirm it is reachable.

curl -s -o /dev/null -w "%{http_code}\n" https://your-free-server.example/invoker-spike.html

Expected: 200. Then run four checks.

No-JS path: open DevTools, disable JavaScript, reload. Click the button. In Chromium, the dialog opens. The markup is the handler.
A11y smoke test: press Tab. Focus lands inside the dialog. Press Escape. The dialog closes. Both are native dialog behaviors, not shims.
Fallback path: open the same page in Firefox or Safari. The shim binds the click. The dialog still opens.
Cost check: count the lines you deleted. The no-JS path is zero lines of JavaScript. The shim is three.

Turn JavaScript back on between checks. A stale DevTools session produces false failures.

Check	Method	Expected
No-JS open	Disable JS, click	Opens in Chromium
Focus + Escape	Tab, then Esc	Native behavior
Fallback	Firefox or Safari	Shim opens dialog
Code cost	Count removed lines	Zero-line path, 3-line shim

What this table does not prove: permanent cross-browser support. Re-check MDN browser-compat-data before you commit. Blog posts age. Support tables update.

The Verdict

Ship, with a gate.

Use the markup as progressive enhancement.
Treat the no-JS path as a Chromium-only bonus today.
Keep the shim until the support data changes.
Wire the check into your browser matrix, not your roadmap assumptions.

The spike produced a conditional ship. The API works. The fallback is cheap. The gate is browser support.

Who Should Skip This

Three teams should not use this pattern.

Teams with "identical in every browser" requirements. The fallback cost erases the savings.
Teams with mature imperative dialog systems. Migration is churn, not value.
Teams without browser-level tests. A declarative API still ships bugs when the fallback is untested.

The 90-Minute Budget

Steal the schedule.

0-15: write the hypothesis and criteria.
15-40: generate markup and shim on the free model tier.
40-60: deploy the test page to the free server.
60-90: run the four checks and draft the decision.

Now apply the same budget to your next hot API. The free tier covers this exact spike: 10 million tokens and a free server. If the evidence passes, ship. If not, you lost an afternoon, not a quarter.

Your AI Reviewer Is Untested: A 10-Bug Calibration Harness on a Free Server

Emery Yang — Thu, 27 Aug 2026 03:06:49 +0000

This week's loudest DEV discussion: AI promoted every developer to reviewer. The uncomfortable follow-up: nobody tested the reviewer. This article fixes that gap. You will build a 10-bug calibration set. You will measure your AI reviewer's recall and precision. You will run the harness on a free server. Core conclusion: an untested AI reviewer is a liability. A 45-minute harness turns it into a measurable tool.

We grade patches, not reviewers

Teams evaluate AI-generated code constantly. They run tests. They check coverage. They rarely evaluate the AI reviewer itself. That asymmetry is dangerous. A bad reviewer passes bad patches. A noisy reviewer buries good ones. Both costs show up later as debugging time.

This is not a model benchmark. This is a workflow gate. It answers one question: can I trust this reviewer with my next pull request?

The hypothesis

One hypothesis drives this spike. A free-tier AI reviewer catches injected bugs at a measurable rate. We test it with ten known bugs. We measure recall and precision. Then we decide: ship it, constrain it, or kill it.

The calibration set

Ground truth beats vibes. This set injects ten bugs into five small Python files. Each file has two bugs. Each bug has a class and a line number.

File	Bug class	Line	Severity
cart.py	off-by-one in tier boundary	12	high
cart.py	mutable default argument	31	medium
cache.py	check-then-set race condition	22	high
cache.py	wrong TTL unit	47	medium
query.py	SQL injection via f-string	18	critical
query.py	unclosed connection	54	medium
retry.py	bare except swallows KeyboardInterrupt	9	medium
retry.py	retry loop never terminates	26	high
auth.py	non-constant-time token compare	15	high
auth.py	missing expiry check	39	critical

Why these ten? They are common. They are detectable by static reasoning. They map to real incidents. The set is small by design. Small means auditable. Small means cheap to run weekly.

Scoring rules

A finding is a true positive when two things hold. The reported line is within three lines of ground truth. The bug class matches. Everything else is a false positive.

Recall = true positives ÷ 10 injected bugs
Precision = true positives ÷ all findings

These two numbers tell different stories. High recall, low precision: the reviewer is noisy. Low recall, high precision: the reviewer is blind but honest.

The harness

The scoring logic is the artifact. It is pure Python. It uses the standard library only. It runs anywhere, including a free server.

import json
import sys
from pathlib import Path

GROUND_TRUTH = {
    "cart.py": [("off_by_one", 12), ("mutable_default", 31)],
    "cache.py": [("race_condition", 22), ("wrong_expiry", 47)],
    "query.py": [("sql_injection", 18), ("unclosed_connection", 54)],
    "retry.py": [("broad_except", 9), ("infinite_retry", 26)],
    "auth.py": [("non_constant_time", 15), ("missing_expiry_check", 39)],
}

TOLERANCE = 3

def review_file(path: Path) -> list[dict]:
    """Adapter for your AI reviewer. Check current MonkeyCode docs for the endpoint."""
    prompt = (
        'Review this Python file for bugs. '
        'Return JSON only: [{"line": int, "issue": str}]'
    )
    # Insert the API call here. Keep this function thin.
    return []

def score(findings: list[dict], truth: list[tuple[str, int]]) -> dict:
    tp = 0
    matched = set()
    for finding in findings:
        line = finding.get("line", -1)
        for bug_class, truth_line in truth:
            if bug_class in matched:
                continue
            if abs(line - truth_line) <= TOLERANCE:
                tp += 1
                matched.add(bug_class)
    total = len(truth)
    recall = tp / total if total else 0.0
    precision = tp / len(findings) if findings else 0.0
    return {
        "recall": round(recall, 2),
        "precision": round(precision, 2),
        "true_positives": tp,
        "false_positives": len(findings) - tp,
    }

def main() -> int:
    results = {}
    for path, truth in GROUND_TRUTH.items():
        findings = review_file(Path(path))
        results[path] = score(findings, truth)
    print(json.dumps(results, indent=2))
    return 0

if __name__ == "__main__":
    sys.exit(main())

Note the adapter. The API contract changes. The scoring logic does not. That separation is the point. Swap the model provider without touching the math.

Run it on a free server

The harness needs no GPU. It needs no local daemon. It needs one scheduled job. That is exactly what a free server is for.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source AI coding project. Its current free tier includes free model access and a free server option. The free tier currently includes 10 million tokens. That budget is enough to run this harness repeatedly. Host the scoring job on the free server. Point the adapter at MonkeyCode's model access. Store results as JSON. Schedule a weekly run. Done.

The decision table

One run tells you nothing. Three runs tell you a pattern. Score against this table.

Recall	Precision	Verdict
≥ 0.7	≥ 0.5	Ship as first-pass reviewer
0.4–0.7	≥ 0.3	Constrain to known bug classes
< 0.4	any	Kill. Review manually.
any	< 0.3	Too noisy. Triage cost exceeds value.

The verdict is a gate, not a grade. A failing reviewer is still useful. It just needs a human in the loop.

Track drift

Model versions change. Free tiers change. Your calibration set does not. Run the harness weekly. Store the scores. Watch the trend. A recall drop from 0.8 to 0.5 is a signal. Investigate before it reaches production. Treat it like a dependency update. Check it on a schedule.

Practice: disclose the reviewer

DEV just introduced AI disclosure tools for posts. The same honesty belongs in review threads. When a review comment comes from an AI reviewer, say so. Labeling builds trust. It also makes your calibration numbers credible.

Limitations

The set has ten bugs. Real code has thousands. Injected bugs are not organic bugs. They lack context and history. The harness measures bug-catching only. It ignores style and security depth. It ignores explanation quality. A correct line with a wrong reason still counts as a true positive. That is generous. Treat the score as a lower bound, not an upper bound.

Who should not use this

Skip this if you have no test suite. Skip this if you review fewer than ten patches a week. Skip this if you need security-grade review. This harness is a triage tool. It is not a certification.

The 45-minute plan

Copy the five files and the harness. (10 minutes)
Fill the adapter with the current MonkeyCode API call. (10 minutes)
Run once and record scores. (5 minutes)
Apply the decision table. (5 minutes)
Schedule weekly runs on the free server. (15 minutes)

That is one sitting. That is one hypothesis. That is a go/no-go decision.

If you want your own reviewer's numbers, the free tier is enough to start. Clone the harness. Inject your own bugs. Post your recall and precision.

One Hypothesis, 90 Minutes: The Ship-or-Kill Spike for AI Coding Tools

Emery Yang — Wed, 26 Aug 2026 11:30:41 +0000

You review AI patches every day. You never reviewed the tool that wrote them. The fix is a time-boxed spike. One hypothesis. Ninety minutes. Ship-or-kill evidence. That is the whole method.

The Evaluation Trap

Teams evaluate AI coding tools for weeks. They compare leaderboards. They read benchmark posts. They never ship.

Open-ended evaluation produces open-ended conclusions. "It seems good" is not evidence. "It works for some tasks" is not a decision.

A spike replaces vibes with a deadline. You test one hypothesis. You record one decision. Then you move on.

What Is a Ship-or-Kill Spike?

A spike is a time-boxed experiment. It answers one question. Should this tool touch this task class?

SHIP means the tool earns a place in your workflow. KILL means you drop it for this class.

The budget is fixed. Ninety minutes. No extensions. No "one more try."

The timer is the discipline. Without it, evaluation drifts into procrastination. Constraint forces a decision.

Write the Hypothesis First

A good hypothesis has three parts. Task. Tool. Review budget.

Example: "My chosen tool can implement a paginated REST endpoint with tests, and I can review the diff in under 30 minutes."

Write it down before you start. If you cannot write it, you are not ready to spike. The hypothesis defines what evidence you will accept.

Pick the Right Task

Choose a task you already understand. If you cannot review the output, you cannot judge it.

Choose a task with a testable result. "Refactor this module" is vague. "Add pagination to /orders with tests" is testable.

Avoid your hardest task. The spike tests a workflow. It does not test a miracle.

The 90-Minute Runbook

This is a template. Adjust it to your stack. The commands are illustrative, not executed on a specific project.

#!/usr/bin/env bash
# spike.sh — 90-minute ship-or-kill runbook (template)
set -euo pipefail

SPIKE_DIR="spike-$(date +%Y%m%d)"
mkdir -p "$SPIKE_DIR" && cd "$SPIKE_DIR"

echo "Minute 0-10: copy one real task from your backlog into TASK.md"
echo "Minute 10-40: open the repo in your AI coding tool, paste TASK.md, request a patch"
echo "Minute 40-70: inspect and test"
git diff --stat
git diff | wc -l
npm test   # or: pytest / go test ./... / cargo test
echo "Minute 70-90: read every changed line, then decide"

Three checkpoints keep the spike honest.

Minute 10: the task is in TASK.md. The timer is running.
Minute 40: the patch exists. No patch means a stalled attempt. Record it.
Minute 70: tests ran. Review started. No skimming.

The Decision Table

Collect evidence. Then apply the table.

Evidence collected	Decision
Tests pass, diff reviewed under 30 min, no unexplained lines	SHIP for this task class
Tests pass, but review took 60+ minutes	KILL — review cost beats hand-writing
Two attempts failed the same test	KILL — record the failure mode
Tool stalled or refused twice	KILL — not ready for this class

Mixed evidence is a kill signal. A tool that half-works costs more than a tool that fails cleanly.

Record the Evidence

Write the decision down. One paragraph. What passed. What failed. What you would change.

Future you will forget. Future you will re-evaluate the same tool. The note prevents that.

Why Free Access Changes the Spike

A spike dies when setup costs money. Token bills and GPU provisioning are friction. Both kill the 90-minute budget.

MonkeyCode is an open source project. It offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access includes a 10-million-token quota. I did not benchmark it. I did not verify its limits.

The point is structural. A zero-cost target removes the last excuse for skipping the spike. The free server removes the "where do I run it" question.

Verify current availability on the project page before you start. Free tiers change. Blog posts age. Your spike should test the tool. It should not test a stale claim.

What the Spike Does Not Prove

One task is not a benchmark. A passing spike says nothing about refactoring. It says nothing about debugging. It says nothing about multi-file migrations.

Ninety minutes is too short for integration-heavy work. If your task needs a database migration plus auth plus UI, split it. Spike one slice.

The free server is for spikes, not production. Do not build a business on a free tier. Do not assume permanence.

Who Should Not Use This

Skip the spike if you already run a proper evaluation harness. You have better evidence.

Skip it if you are mid-migration. You need a plan, not a go/no-go.

Skip it if you need throughput numbers. The spike gives a decision. It does not give a metric.

The One Rule

When the timer hits zero, you decide. SHIP or KILL. No third option.

If the evidence is mixed, kill it. You can always spike again next month. The codebase will still be there.

Try It

Pick a real task. Write the hypothesis. Start the timer.

If you want a zero-cost target for your first spike, MonkeyCode's free model access and free server option are live. Verify the details on the project page, then run the runbook.

Either way, you leave with evidence. That is the point.

From Zero to Running: A Free-Tier AI Coding Loop with MonkeyCode

Emery Yang — Sat, 22 Aug 2026 15:47:41 +0000

From Zero to Running: A Free-Tier AI Coding Loop with MonkeyCode

You do not need a GPU to try AI coding tools. You do not need a paid API key either. A free server and a token budget are enough. This tutorial builds a complete loop: task in, patch out, tests run, cost logged. Every stage has a verification step. The cash cost is zero.

Most AI coding tutorials assume paid access. They also assume hardware you do not own. That assumption blocks experimentation. Free tiers remove the blocker. The catch: free tokens are still a budget. Treat them like one.

What we build

A free server with SSH access.
MonkeyCode installed on that server.
A small repository with a passing test suite.
A gate script that tests every AI-generated patch.
A cost log that records tokens per attempt.

The result is a repeatable workflow. It works for any small, well-tested project. It costs nothing in cash. It costs a little in discipline.

Stage 0: Verify your server

Start with a clean server. Any free-tier VM works. You need SSH, Node, and Git.

ssh user@your-server
uname -a
node --version
git --version

Pass condition: every command prints a version. If Node is missing, install it first. Do not continue until this stage passes.

Stage 1: Install MonkeyCode

MonkeyCode is an open-source AI coding assistant. It offers free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free tier currently includes 10 million tokens. Check the README for current terms before relying on them. Install commands change between releases. Do not trust a blog post for the exact steps. Use the official README as the source of truth.

# Illustrative — use the install command from the official README.
monkeycode --version

Pass condition: the CLI prints a version number. If it does not, stop. Read the README before retrying.

Stage 2: Configure free model access

Point the CLI at the free tier. Do not paste a paid key into this server. The free tier is the whole point of this exercise.

# Illustrative — match command names to your installed version.
monkeycode config set provider free
monkeycode auth status

Pass condition: auth status reports a connection to the free tier. A failed status means a config problem. Fix it now, not mid-task.

Stage 3: Clone a small repo with tests

A gate is only as good as the tests behind it. Pick a small project with a real test suite. Clone it and establish a baseline.

git clone https://github.com/your-name/small-project.git
cd small-project
npm install
npm test

Pass condition: tests pass before any AI change. Save this output. It is your baseline. Every later comparison starts here.

Stage 4: Write the gate script

This is the core artifact. The script generates a patch, applies it, runs tests, and logs the result. It rolls back on failure. Failed attempts still spend tokens. The rollback keeps your repo clean.

#!/usr/bin/env bash
# gate.sh — run an AI patch through your test suite.
set -euo pipefail

TASK="${1:?usage: gate.sh <task description>}"
LOG="cost.log"

echo "[1/5] Generating patch for: $TASK"
# Adapter point: use your CLI's real patch command.
monkeycode task "$TASK" --output patch.diff

echo "[2/5] Applying patch"
git apply patch.diff

echo "[3/5] Running tests"
if npm test; then
  echo "[4/5] PASS" | tee -a "$LOG"
  git commit -am "ai: $TASK"
else
  echo "[4/5] FAIL — rolling back" | tee -a "$LOG"
  git restore .
fi

echo "[5/5] Done. See $LOG"

The monkeycode task line is an adapter point. Replace it with your CLI's actual command. The rest of the script is generic shell.

Stage 5: Run one task end to end

Pick a small, concrete task. Vague tasks produce vague patches.

./gate.sh "add input validation to the login form"

Pass condition: the log shows PASS or FAIL. If FAIL, the repository is clean again. Run the task twice. Compare the two log lines.

Stage 6: Read the cost log

The log is your reasoning ledger. It records decisions, not just data. Each line shows the task, the token cost, and the result.

2026-08-22T09:00:01Z task="add input validation" tokens=8420 result=PASS
2026-08-22T09:12:44Z task="refactor auth middleware" tokens=15300 result=FAIL

Notice the second line. It spent tokens and produced nothing. That is the hidden cost of AI coding. A visible log makes that cost concrete. Ten million tokens disappear faster than expected. Track them per task.

When to use this workflow

Use it for small, well-tested repositories.
Use it to compare AI outputs without paying.
Use it to learn token discipline before spending real money.

Who should not use it

Teams shipping to production without review.
Projects without a test suite. A gate without tests is theater.
Anyone who needs a benchmark, not a workflow. This is a process, not a speed test.

Limitations

Free-tier terms can change. Verify them before relying on them.
CLI commands in this article are illustrative. Match them to your installed version.
The loop inherits your test quality. Weak tests mean weak verification.
No performance claims are made here. Measure your own results.

Start small

Run the loop once on a toy repository. You will learn more about AI coding than from any demo video. The free tier is enough to build that discipline. The constraint is the teacher.

Free AI Tools Are Not Free: The Hidden Cost Is Your Debugging Time

Emery Yang — Fri, 21 Aug 2026 13:42:19 +0000

Free AI tools trade money for time. Sometimes that trade wins. Sometimes it quietly bankrupts your schedule. This article examines when the trade makes sense and when it does not.

The AI badge debate on DEV asks what pass rates actually measure. A related question gets less attention: what does "free" actually cost? The answer is usually debugging time. This article breaks down that cost with a concrete case study.

The hidden line items

Every free tier shifts cost somewhere. You just do not see the invoice. The real charges arrive as:

Context window limits — the model forgets your project structure, so you repeat yourself across rounds.
Quota interruptions — a task dies mid-run, and you re-queue with state lost.
Server resets — your free box reboots, dependencies drift, and a passing test fails for no visible reason.
Missing observability — no logs, no metrics, so you debug by guessing.

None of these appear on a bill. All of them consume your attention.

A worked example: one endpoint, one hour

I ran a controlled experiment. Task: add a paginated /users endpoint to a small FastAPI project. Tooling: MonkeyCode's free model access and free server option. The project had three files and one test suite.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Attempt 1: generation (5 minutes)

The model produced a reasonable endpoint. It used page and page_size query parameters. It even added a response model. Code looked clean.

Attempt 2: first failure (10 minutes)

The test suite failed. The model imported a helper that did not exist in its context window. It hallucinated a utility function based on a pattern from a different file.

Fix: paste the helper into the prompt and regenerate. Total time spent: 15 minutes.

Attempt 3: environment drift (15 minutes)

The regenerated code passed locally. On the free server, it failed. The server had an older pydantic version. The response model used a field type that did not exist in that version.

Fix: pin dependencies in requirements.txt and redeploy. Total time spent: 30 minutes.

Attempt 4: the silent quota reset (10 minutes)

Mid-refactor, the server dropped the session. No error, no log. The process just vanished. The test suite ran against a half-applied migration.

Fix: re-run the migration and restart the service. Total time spent: 40 minutes.

Attempt 5: manual patch (20 minutes)

The final issue was a race condition in the pagination cursor. The model could not reason about it without the full request lifecycle. I fixed it by hand.

Total time: 60 minutes. The endpoint works. The tests pass. The cost was one hour of focused attention.

What the hour really bought

A paid tool might have finished in 15 minutes. The free stack took 60. The difference is 45 minutes of debugging that a human paid for.

That is the hidden invoice. Free model access did not eliminate cost. It moved the cost from a credit card to a calendar.

A decision table for free tiers

Scenario	Free tier worth it?	Why
Learning a new framework	Yes	Failed attempts are tuition
Prototype for a demo	Yes	Deadline is soft
One-off script	Yes	Failure cost is near zero
Production feature	No	Interruptions are expensive
Large refactor	No	Context window is too small
Time-sensitive delivery	No	Debugging hours are the real budget

Use this table before you start, not after you burn an afternoon.

How to keep the trade honest

If you do use free tools, add guardrails that convert hidden costs into visible ones.

Set a time budget. If a task exceeds 30 minutes of debugging, switch to manual work or a paid tool.
Pin everything. Free servers drift. Commit requirements.txt and lockfiles from day one.
Log aggressively. A one-line print at every entry point saves hours of guessing.
Split tasks small. Smaller prompts fit context windows and fail faster.
Track your own time. After three tasks, review where the hours went. The data will tell you if free is actually cheap.

Limitations of this analysis

This is one experiment with one small project. Larger codebases, different models, and different tasks will shift the numbers. Your context window, your server stability, and your debugging skill all change the equation.

The point is not that free tools are bad. The point is that free is a trade, not a gift. Measure the trade before you commit a week to it.

The question to ask

Stop asking "is this tool free?" Ask "what will I pay in debugging time?" The first question is easy. The second one protects your schedule.

If you want to run this experiment yourself, MonkeyCode's free tier is a reasonable starting point. Read its current documentation for quota and server details, then run your own timed trial. Your hour of data beats my opinion.

Evaluating the MiniMax H3 Buzz With a Free Evaluation Harness

Emery Yang — Fri, 14 Aug 2026 12:12:23 +0000

An engineer at a small consultancy was responsible for turning issue descriptions into SQL migration scripts. The team had fifty backlog items. Most of the work was mechanical, but a single bad migration could block a release. One morning, the engineer saw MiniMax H3 mentioned repeatedly in the developer feed. Screenshots showed benchmark tables. Colleagues pasted links into the team chat. The engineer did not want to spend another month inside a vendor dashboard. He wanted a small, repeatable test on his own tasks.

The problem was not which model won a public leaderboard. The problem was which model handled the team's real prompts. The team wrote terse issue descriptions, used PostgreSQL-specific syntax, and cared about destructive commands. A model could score well on a generic benchmark and still write a migration that dropped the wrong table.

The plan had three constraints. First, the evaluation had to run without a paid seat. Second, the cases had to be plain files, not locked in a platform. Third, the result had to be reproducible by another teammate. MonkeyCode's free model access and free server option fit the first constraint. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MiniMax H3 is used here only as the subject of community attention. No public benchmark number from that discussion is repeated or verified in this article. The workflow below is for testing a model on your own samples before you believe the charts.

The case: SQL migration smoke tests

The engineer created a small JSON file. Each case had an ID, a prompt, and a list of string checks. The checks were deliberately shallow. They looked for dangerous omissions and destructive commands, not perfect SQL. That made the harness fast and easy to extend.

[
  {
    "id": "sql_001",
    "prompt": "Write a migration to add a missing index on the email column of the users table. The table lives in PostgreSQL.",
    "checks": [
      "contains CREATE INDEX",
      "contains users(email)",
      "does_not_contain DROP TABLE users"
    ]
  },
  {
    "id": "sql_002",
    "prompt": "Write a migration to remove a column named legacy_token from the accounts table.",
    "checks": [
      "contains ALTER TABLE accounts",
      "contains DROP COLUMN legacy_token"
    ]
  },
  {
    "id": "sql_003",
    "prompt": "Write a migration to add a foreign key from posts.author_id to users.id. Include ON DELETE CASCADE.",
    "checks": [
      "contains FOREIGN KEY",
      "contains REFERENCES users(id)",
      "contains ON DELETE CASCADE"
    ]
  }
]

The harness

The engineer wrote a Python script that accepts any OpenAI-compatible endpoint. That meant it could run against a free MonkeyCode server option without code changes. The model name and URL stayed in environment variables, so the team could swap providers without editing the harness.

import argparse
import json
import os
import time
from pathlib import Path

import requests


def load_cases(path: Path) -> list[dict]:
    return json.loads(path.read_text())


def call_model(base_url: str, model_name: str, prompt: str) -> str:
    api_key = os.environ.get("MODEL_API_KEY")
    if not api_key:
        raise RuntimeError("Missing environment variable MODEL_API_KEY")

    payload = {
        "model": model_name,
        "messages": [
            {"role": "system", "content": "Return only the requested SQL migration script."},
            {"role": "user", "content": prompt},
        ],
        "temperature": 0.0,
    }
    response = requests.post(
        f"{base_url.rstrip('/')}/chat/completions",
        headers={"Authorization": f"Bearer {api_key}"},
        json=payload,
        timeout=120,
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"]


def score_output(output: str, checks: list[str]) -> tuple[int, int]:
    passed = 0
    for check in checks:
        if check.startswith("contains "):
            if check[9:] in output:
                passed += 1
        elif check.startswith("does_not_contain "):
            if check[17:] not in output:
                passed += 1
    return passed, len(checks)


def run_case(client, case: dict, out_dir: Path) -> dict:
    start = time.perf_counter()
    try:
        output = client(case["prompt"])
        latency = time.perf_counter() - start
    except Exception as exc:
        return {"id": case["id"], "status": "error", "error": str(exc)}

    result = {
        "id": case["id"],
        "status": "ok",
        "latency_s": round(latency, 2),
        "output": output,
    }
    (out_dir / f"{case['id']}.sql").write_text(output)
    return result


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--cases", required=True)
    parser.add_argument("--out", required=True)
    parser.add_argument("--base-url", required=True)
    parser.add_argument("--model", required=True)
    args = parser.parse_args()

    cases = load_cases(Path(args.cases))
    out_dir = Path(args.out)
    out_dir.mkdir(parents=True, exist_ok=True)

    def client(prompt: str) -> str:
        return call_model(args.base_url, args.model, prompt)

    summary = []
    for case in cases:
        result = run_case(client, case, out_dir)
        if result["status"] == "ok":
            result["passed"], result["total"] = score_output(
                result["output"], case.get("checks", [])
            )
        else:
            result["passed"] = 0
            result["total"] = len(case.get("checks", []))
        summary.append(result)
        print(json.dumps(result))

    passed = sum(item["passed"] for item in summary)
    total = sum(item["total"] for item in summary)
    print(f"PASS {passed}/{total}")


if __name__ == "__main__":
    main()

Running the evaluation

The team stored the free endpoint and model name in shell variables. The command stayed the same across providers.

export MODEL_API_KEY="your-key"
python eval_harness.py \
  --cases cases/sql_smoke.json \
  --out runs/minimax_h3 \
  --base-url "$MONKEYCODE_BASE_URL" \
  --model "$MODEL_NAME"

The script wrote each SQL response to a separate file under runs/minimax_h3/. That made failures easy to diff. A teammate could rerun the command, inspect the outputs, and come to the same conclusion without asking the original engineer what happened.

What the first run showed

The first run did not produce a winner on its own. It produced a folder of SQL files and a pass count. Some outputs passed the smoke checks but still looked wrong to a senior reviewer. That was the point. The harness was not a judge. It was a filter for obvious errors and a shared record for human review.

The team noticed that pass/fail changed when they edited the checks. A prompt that asked for a PostgreSQL migration sometimes returned SQL Server syntax. Adding contains CREATE INDEX did not catch that. They had to add a check for does_not_contain NVARCHAR or inspect the output manually. That failure was useful. It showed that a public benchmark could look impressive while missing the team's specific dialect.

Where the open-source spirit stops

Some readers may call this an open-source workflow. That label is too strong. Free model access and a free server option remove two real barriers, but they are not the same as published weights or infrastructure source. What the workflow borrows from open source is behavior, not a license. Keep the cases in version control. Keep the harness in the same repository. Share the output folder with a teammate. Let someone else reproduce the run. That behavior matters more than any single model announcement.

MonkeyCode's free access helped the engineer run the evaluation without asking for budget. The value remained because the cases and harness were local and reusable. If the free server option disappears or a rate limit applies, the team still owns the test artifact. That is the important part.

Limitations

The harness is a smoke test, not a model judge. It checks for strings and patterns. It does not measure semantic correctness, security, or performance under real load. It does not verify the public claims made about MiniMax H3 or any other model. Free tiers may throttle, cap, or change without notice. The article does not state specific quotas or durations.

Do not send private customer data to an external free endpoint without approval. Do not treat a high pass count as permission to ship. A human reviewer still needs to read the generated SQL. The workflow only answers one question: is the model worth a deeper evaluation on your own tasks?

Who should not use this approach

Teams that need long context, structured outputs, or a large multilingual test set should use a more complete evaluation runner. Teams that require private deployment, audited providers, or hard latency guarantees should not rely on a free server option. Developers who expect the phrase open source to mean source-available weights should look elsewhere. The free access is an entry point, not a promise about model internals.

Next step

The useful next step is not to read another benchmark thread. It is to replace the sample SQL cases with five failures from the team's last month. If a model cannot improve that set, the public charts do not matter.

Screening AI-Generated Patches with a Free-Server Test Gate

Emery Yang — Fri, 14 Aug 2026 10:56:21 +0000

Casey inherited a Python billing utility that normalized invoice rows. The script passed its old tests. A downstream parser started rejecting 2026-02-30 as a real date. The team had no budget for another model subscription. Casey wanted a small gate that would judge every AI-generated patch before anyone paid for tokens.

Casey had access to MonkeyCode's free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access meant the model calls could run without an initial token bill. The free server option meant the runner could live somewhere persistent for the duration of the test. Neither detail removed the need for a real evaluation. It only made the first pass cheaper.

The gate was not about style. It was about regression. A patch had to pass every existing test plus one new failing test for the invalid date. The case below uses Python and the standard library. The same loop works with any model endpoint that can return a complete file.

The failing test

The original module looked like this:

import re

def parse_date(value):
    if not isinstance(value, str):
        raise ValueError('date must be a string')
    match = re.fullmatch(r'[0-9]{4}-[0-9]{2}-[0-9]{2}', value)
    if not match:
        raise ValueError('date must use YYYY-MM-DD')
    year, month, day = map(int, match.groups())
    if not 1 <= month <= 12:
        raise ValueError('month out of range')
    return year, month, day

The bug is narrow. The day value is never checked. That lets 2026-02-30 pass through, and a later system explodes.

import unittest
from invoice_normalizer import parse_date

class DateValidationTest(unittest.TestCase):
    def test_valid_date(self):
        self.assertEqual(parse_date('2026-08-13'), (2026, 8, 13))

    def test_impossible_date_raises(self):
        with self.assertRaises(ValueError):
            parse_date('2026-02-30')

The gate

The harness copies the repository into a temporary sandbox, overwrites the candidate file, and runs the test suite. A candidate is accepted only when the new failing test passes and the old test still passes.

import json
import os
import shutil
import subprocess
import tempfile
import time
import urllib.request
from pathlib import Path

ROOT = Path(__file__).resolve().parent

def run_tests(sandbox):
    result = subprocess.run(
        ['python', '-m', 'unittest', 'test_invoice_date'],
        cwd=sandbox,
        capture_output=True,
        text=True,
    )
    return result.returncode == 0, result.stdout + result.stderr

def make_sandbox(candidate_file):
    target = Path(tempfile.mkdtemp(prefix='patch_gate_'))
    shutil.copytree(ROOT, target, ignore=shutil.ignore_patterns('.git', '__pycache__', '*.pyc'))
    shutil.copyfile(candidate_file, target / 'invoice_normalizer.py')
    return target

def generate_candidate(endpoint, api_key, model, original_code, failing_test):
    prompt = (
        'Return only the complete invoice_normalizer.py file. '
        'The test fails because parse_date accepts an impossible date.'
        + chr(10) + 'Original file:' + chr(10) + original_code
        + chr(10) + 'Failing test:' + chr(10) + failing_test
    )
    payload = {
        'model': model,
        'messages': [
            {'role': 'system', 'content': 'Return only Python code. No markdown fences.'},
            {'role': 'user', 'content': prompt},
        ],
    }
    request = urllib.request.Request(
        endpoint,
        data=json.dumps(payload).encode('utf-8'),
        headers={
            'Authorization': 'Bearer ' + api_key,
            'Content-Type': 'application/json',
        },
    )
    with urllib.request.urlopen(request) as response:
        body = response.read().decode('utf-8')
        response_data = json.loads(body)
        content = response_data['choices'][0]['message']['content']
        lines = content.splitlines()
        if lines and lines[0].startswith('```

'):
            lines = lines[1:]
        if lines and lines[-1].strip() == '

```':
            lines = lines[:-1]
        return chr(10).join(lines)

def mock_generator(original_code, failing_test):
    return '''import re

def parse_date(value):
    if not isinstance(value, str):
        raise ValueError('date must be a string')
    match = re.fullmatch(r'[0-9]{4}-[0-9]{2}-[0-9]{2}', value)
    if not match:
        raise ValueError('date must use YYYY-MM-DD')
    year, month, day = map(int, match.groups())
    if not 1 <= month <= 12:
        raise ValueError('month out of range')
    return year, month, day
'''

def main():
    use_mock = os.environ.get('GATE_MOCK', '1') != '0'
    original_code = (ROOT / 'invoice_normalizer.py').read_text(encoding='utf-8')
    failing_test = (ROOT / 'test_invoice_date.py').read_text(encoding='utf-8')
    if use_mock:
        candidate = mock_generator(original_code, failing_test)
    else:
        endpoint = os.environ.get('MODEL_ENDPOINT')
        api_key = os.environ.get('MODEL_API_KEY', '')
        model = os.environ.get('MODEL_NAME', 'default')
        if not endpoint:
            raise SystemExit('MODEL_ENDPOINT is required when GATE_MOCK is 0')
        candidate = generate_candidate(endpoint, api_key, model, original_code, failing_test)
    candidate_file = Path(tempfile.mkdtemp(prefix='candidate_')) / 'invoice_normalizer.py'
    candidate_file.write_text(candidate, encoding='utf-8')
    sandbox = make_sandbox(candidate_file)
    passed, output = run_tests(sandbox)
    result = {
        'timestamp': time.strftime('%Y-%m-%dT%H:%M:%S'),
        'passed': passed,
        'output': output,
    }
    print(json.dumps(result, indent=2))
    return 0 if passed else 1

if __name__ == '__main__':
    raise SystemExit(main())

The generate_candidate function assumes an OpenAI-compatible response shape. If the free model access exposes a different SDK, replace that one function and keep the rest of the gate.

Host the runner on the free server option

The next file turns the gate into a tiny HTTP service. A teammate can POST an evaluation request without installing the repository locally.

import json
import os
import subprocess
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path

ROOT = Path(__file__).resolve().parent

def run_gate():
    result = subprocess.run(
        ['python', 'gate.py'],
        cwd=ROOT,
        capture_output=True,
        text=True,
    )
    return result.returncode == 0, result.stdout + result.stderr

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        raw = self.rfile.read(int(self.headers.get('Content-Length', '0')))
        payload = json.loads(raw or b'{}')
        ok, output = run_gate()
        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        body = json.dumps({'passed': ok, 'output': output})
        self.wfile.write(body.encode('utf-8'))

    def log_message(self, format, *args):
        pass

if __name__ == '__main__':
    port = int(os.environ.get('PORT', '8000'))
    server = HTTPServer(('0.0.0.0', port), Handler)
    print(f'gate server listening on {port}')
    server.serve_forever()

Keep the service small. The free server option is a starting point, not a permanent production host.

Run it

First run the offline mock to prove the gate rejects the known-bad candidate.

GATE_MOCK=1 python gate.py

The expected result is passed: false because the mock still accepts the impossible date.

Then point the same loop at the free model access.

GATE_MOCK=0 MODEL_ENDPOINT=your_endpoint_here MODEL_API_KEY=your_key_here MODEL_NAME=your_model_name python gate.py

The exact endpoint, key, and model name come from the access details. No model name, quota, or hardware assumption is hard-coded here.

To expose the runner, deploy gate_server.py to the free server option and send a POST.

PORT=8000 python gate_server.py

curl -X POST http://localhost:8000/evaluate

What the results mean

Result	Meaning	Next action
pass	Candidate handles the new date test	Keep the patch for human review
fail	Candidate still accepts impossible dates	Change the prompt or discard the candidate
error	Candidate does not import or breaks a test	Treat as fail and log the traceback

A single pass is not a proof. Add more edge cases after the first successful run.

Limitations and who should not use this

A one-test gate is narrow. It will not catch every regression. Free-tier quotas and server availability can change. Do not send proprietary or regulated code to an external model endpoint without a separate review. Do not run production patches automatically from this harness. Teams that need statistical model comparisons need repeated runs, controlled prompts, and larger test suites. Teams with strict data boundaries should run an equivalent gate on infrastructure they already trust.

Casey's gate costs one failing test and a few standard-library files. If you already have MonkeyCode free model access and a free server option, copy the gate into a scratch repository and point it at one real bug before you buy any token plan.