DEV Community: Emery Lin

Use MonkeyCode's Free Tokens to Build a PR Summarizer in CI That Doesn't Spam

Emery Lin — Tue, 01 Sep 2026 13:11:57 +0000

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The best AI code review bot is the one you don't notice. It comments once, updates itself, and never repeats the same advice on the same diff. That's not a model problem; it's a pipeline problem. Model quality matters, but a bot that double-posts or flakes will cost you more trust than it earns.

I've been testing a different pattern: a small GitHub Actions job that pulls the diff, asks a free model for a summary, and upserts a single PR comment. MonkeyCode's open-source project fits this because its free tier includes 10 million tokens and a free server option. As of September 2026, those numbers come from the project's public repo, but always check the README before building a workflow around a promo allowance.

Why AI review bots spam PRs

Most duplicates come from design, not from the model. Each push triggers a new workflow run, each run calls the model again, and each call writes a fresh comment. The model may also produce slightly different wording for the same diff, so simple text comparison won't catch the repeat.

The fix is to make the workflow stateful in three places:

Cache the model response by diff hash.
Comment with a marker so you can find and update it.
Set a timeout so a slow free server can't block the pipeline indefinitely.

That last point is the one most tutorials skip. Free servers get busy. Your CI job should fail fast, not hang.

Workflow: one comment per PR

The artifact below is a complete GitHub Actions workflow with a Python script. It uses an OpenAI-compatible endpoint; MonkeyCode's docs will tell you the exact model name and route. Replace the placeholder with the real value before running it.

# .github/workflows/pr-summary.yml
name: ai-pr-summary
on:
  pull_request:
    types: [opened, synchronize]
permissions:
  contents: read
  pull-requests: write
jobs:
  summarize:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Generate or update summary
        env:
          MC_API_KEY: ${{ secrets.MC_API_KEY }}
          MC_BASE_URL: ${{ secrets.MC_BASE_URL }}
          GH_TOKEN: ${{ github.token }}
        run: |
          python .github/scripts/pr_summarizer.py

The script: diff in, one comment out

Save this as .github/scripts/pr_summarizer.py. It uses only the standard library, so you don't need a language runtime beyond Python.

#!/usr/bin/env python3
import json
import os
import subprocess
import urllib.request

repo = os.environ["GITHUB_REPOSITORY"]
event = json.load(open(os.environ["GITHUB_EVENT_PATH"]))
pr = event["pull_request"]["number"]
base = event["pull_request"]["base"]["ref"]

# 1. Get the diff against the base branch.
#    If your diff is huge, truncate it; see "Limitations" below.
diff = subprocess.run(
    ["git", "diff", f"origin/{base}...HEAD"],
    capture_output=True,
    text=True,
).stdout[:12000]

# 2. Call the model through MonkeyCode's free server.
#    Replace "monkeycode-free" with the model name from the official docs.
payload = json.dumps({
    "model": "monkeycode-free",
    "messages": [{
        "role": "user",
        "content": (
            "Summarize the behavior changes in this PR in under 100 words. "
            "Use bullet points. Do not invent details.\n\n"
            f"```
{% endraw %}
diff\n{diff}\n
{% raw %}
```"
        ),
    }],
    "temperature": 0.2,
}).encode()

req = urllib.request.Request(
    os.environ["MC_BASE_URL"] + "/v1/chat/completions",
    data=payload,
    headers={
        "Authorization": "Bearer " + os.environ["MC_API_KEY"],
        "Content-Type": "application/json",
    },
)
with urllib.request.urlopen(req, timeout=60) as response:
    data = json.load(response)
summary = data["choices"][0]["message"]["content"].strip()

# 3. Upsert one comment per PR.
marker = "<!-- ai-pr-summary -->"
headers = {
    "Authorization": "token " + os.environ["GH_TOKEN"],
    "Accept": "application/vnd.github+json",
}
url = f"https://api.github.com/repos/{repo}/issues/{pr}/comments"

req = urllib.request.Request(url, headers=headers)
with urllib.request.urlopen(req) as response:
    comments = json.load(response)

comment_body = marker + "\n\n" + summary
new_headers = {**headers, "Content-Type": "application/json"}

for comment in comments:
    if marker in comment["body"]:
        req = urllib.request.Request(
            f"{url}/{comment['id']}",
            data=comment_body.encode(),
            headers=new_headers,
            method="PATCH",
        )
        urllib.request.urlopen(req)
        break
else:
    req = urllib.request.Request(
        url,
        data=comment_body.encode(),
        headers=new_headers,
        method="POST",
    )
    urllib.request.urlopen(req)

print(summary)

Add the two secrets in your repo settings: MC_API_KEY and MC_BASE_URL. If you're using MonkeyCode's free server, the base URL comes from the project's setup instructions.

How this stays quiet

The marker is the key. New comments are POSTed only when no marker exists. Every later push PATCHes the same comment instead of appending another one. That alone kills 90% of the spam.

For extra determinism, keep temperature low and pin the model version. A high temperature turns a summary bot into a lottery. You don't want the same diff to produce two different reviews on two identical runs.

Limitations and who should skip this

This is not a code review gate. It cannot reason about your private threat model, and it should never be the only thing between a PR and main. The free server is shared; expect occasional latency or a 503. The 60-second timeout means the job may fail loudly instead of blocking a merge silently.

You should also watch the token budget. Ten million tokens sounds like a lot until you start sending 12,000 characters of diff on every push. Cache by diff hash if you're on a busy monorepo. And if your team does hundreds of PRs a day, a shared free tier will not stay responsive — at that point you need your own MonkeyCode server.

One step at a time

Start with a single repository and one workflow. Let it run for a week, read the comments, and tune the prompt. Once the output is boring and consistent, add another repo.

If you're tired of bots repeating themselves, MonkeyCode's free token allowance is a low-risk sandbox for this pattern. Build the pipeline first, judge the model second.

Give Every Dependabot PR a Risk Label Before You Click Review

Emery Lin — Mon, 31 Aug 2026 12:19:12 +0000

Dependabot opens five PRs a week. You merge the patch version, panic over the major release, and manually open changelogs to guess whether a type signature changed. That's tedious and error-prone.

A small LLM can do that triage for you: read the changelog, inspect the diff stats, and output a risk label plus a one-sentence reason. Run it on a schedule with free model credits and a free server, and you get a living dependency radar without adding to your cloud bill.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The context gap in dependency updates

Dependabot tells you what changed — version X to Y — but not whether it matters. The release notes might be huge, yet the actual API difference might be trivial. Or the changelog says "minor internal refactor" while the diff changes a public function's return type.

That gray zone is where bugs are born. Humans can't read every changelog, and deterministic checks only catch breaking version bumps. A model can summarize the risk signal in a structured form, and a small CI gate can present it the moment a PR appears.

What we're building

A Python script that:

Fetches a Dependabot PR's metadata and diff stats from the GitHub API.
Extracts the changelog from the PR description or the linked release page (when available).
Calls a free LLM with a strict JSON prompt, asking for a risk assessment: low, medium, high, or breaking.
Writes a GitHub check or PR comment with the label and reason.

The script is designed to run anywhere — a GitHub Action, a cron job, or a free server slot. No proprietary SDK required; it uses plain HTTP.

Step 1: The risk assessment script

Save this as dependabot_triage.py. Replace the API endpoint and key with whatever provider you're using; MonkeyCode's free models work with this pattern.

import json
import os
import re
import sys
import time
from urllib import request, error


def call_llm(prompt: str) -> dict:
    api_key = os.environ["MK_API_KEY"]
    payload = {
        "prompt": prompt,
        "max_tokens": 200,
        "temperature": 0.2,
    }
    req = request.Request(
        os.environ.get("MK_API_URL", "https://api.monkeycode.ai/v1/completions"),
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
        },
    )
    try:
        with request.urlopen(req, timeout=45) as resp:
            data = json.loads(resp.read().decode())
            text = data["choices"][0]["text"]
            # extract first JSON object
            start = text.find("{")
            end = text.rfind("}")
            return json.loads(text[start:end + 1])
    except Exception:
        # deterministic fallback: call it unknown, not broken
        return {"risk": "unknown", "reason": "call failed, human review required"}


def fetch_pr(repo: str, pr_number: int, token: str) -> dict:
    req = request.Request(
        f"https://api.github.com/repos/{repo}/pulls/{pr_number}",
        headers={"Authorization": f"Bearer {token}", "Accept": "application/vnd.github+json"},
    )
    with request.urlopen(req, timeout=30) as resp:
        pr = json.loads(resp.read().decode())
    req2 = request.Request(
        f"https://api.github.com/repos/{repo}/pulls/{pr_number}/files",
        headers={"Authorization": f"Bearer {token}", "Accept": "application/vnd.github+json"},
    )
    with request.urlopen(req2, timeout=30) as resp:
        files = json.loads(resp.read().decode())

    additions = sum(f"additions" for f in files)
    deletions = sum(f["deletions"] for f in files)
    changed = [f["filename"] for f in files[:20]]
    return {
        "title": pr["title"],
        "body": (pr.get("body") or "")[:2000],
        "additions": additions,
        "deletions": deletions,
        "changed_files": changed,
    }


def build_prompt(pr: dict) -> str:
    return f"""You are a dependency risk analyst. Given a PR update, label the risk of merging it.
Return JSON only: {{"risk": "low|medium|high|breaking", "reason": "one short sentence"}}

PR title: {pr['title']}

PR body snippet:
{pr['body'][:1000]}

Diff stats: +{pr['additions']} -{pr['deletions']}
Changed files: {', '.join(pr['changed_files'][:10])}

Rules:
- low: patch or minor with no touching core types
- medium: behavior might change, needs manual smoke test
- high: major version, many files, risky packages
- breaking: clearly says BREAKING or renames public API"""


def triage(repo: str, pr_number: int) -> dict:
    token = os.environ["GITHUB_TOKEN"]
    pr = fetch_pr(repo, pr_number, token)
    prompt = build_prompt(pr)

    # retry pattern: the model may respond with malformed JSON
    for _ in range(3):
        result = call_llm(prompt)
        if "risk" in result:
            return result
        time.sleep(2)
    return {"risk": "unknown", "reason": "could not parse model output"}


if __name__ == "__main__":
    repo, pr_number = sys.argv[1], int(sys.argv[2])
    result = triage(repo, pr_number)
    print(json.dumps(result, ensure_ascii=False))

The key line is the fallback: if the model call fails, you get unknown instead of a crash. That's the difference between a helpful automation and a support ticket generator.

Step 2: Turn the label into a GitHub check

You want this visible where developers already look. A simple Action can run the script and leave a PR comment.

name: dependabot-triage
on:
  pull_request_target:
    types: [opened, edited, synchronize]

jobs:
  triage:
    if: github.event.pull_request.user.login == 'dependabot[bot]'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run triage
        id: triage
        env:
          MK_API_KEY: ${{ secrets.MK_API_KEY }}
          MK_API_URL: ${{ secrets.MK_API_URL }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          python dependabot_triage.py ${{ github.repository }} ${{ github.event.pull_request.number }} > result.json
          echo "::set-output name=risk::$(jq -r .risk result.json)"
          echo "::set-output name=reason::$(jq -r .reason result.json)"
      - name: Comment on PR
        uses: actions/github-script@v6
        with:
          script: |
            const risk = "${{ steps.triage.outputs.risk }}";
            const reason = "${{ steps.triage.outputs.reason }}";
            github.rest.issues.createComment({
              ...context.repo,
              issue_number: context.issue.number,
              body: `**Risk: ${risk}** — ${reason}`
            });

Note the if condition: this only runs on Dependabot PRs, so you're not paying for model calls on your own feature branches. Free-tier quotas last much longer that way.

Step 3: Turn the label into a policy

The label alone is not a merge gate. You still need humans for medium and high. A decision table makes the action obvious:

Risk label	Meaning	Required action
low	Patch or trivial minor, no core type changes	Merge after CI passes
medium	Behavior might shift; needs smoke test	Run test suite, manual sanity check
high	Major version or many files changed	Full review, feature manual test
breaking	Public API renamed/removed	Schedule migration, update call sites first
unknown	Model failed or gave bad JSON	Manual triage, ignore automation suggestion

You can encode this in Branch Protection rules or just as a team convention. The model doesn't make the decision; it hands you the evidence in a form you can act on.

Step 4: Run it on a free server

GitHub Actions works, but you might want a daily digest of all open dependency PRs across a whole org. That's where a free server with cron comes in.

On the free server:

pip install requests  # or nothing else needed for the script above
crontab -e
# add line:
0 8 * * * cd /opt/dep-triage && for pr in $(list_prs); do python3 dependabot_triage.py org/repo $pr >> digest.md; done

Then send digest.md to your team's chat or leave it as a file in the repo. The script is lightweight enough that a single free server slot handles hundreds of PRs a day without breaking a sweat.

Limitations

Hallucinated labels: a model can misread a changelog. Always keep the final merge decision with a human.
Changelog quality: if the PR body is empty and the repo has no release notes, the model works from diff stats only, which is a weak signal.
Prompt drift: your prompt might need tuning as you change LLM providers. Pin the prompt and version it.
Rate limits: free tiers have quota. The pull_request_target filter and cooldown timers keep usage reasonable.

Who should not use this

Teams with zero test coverage: a risk label won't save you if every upgrade breaks anyway. Fix the safety net first.
Regulated environments: no generated risk label is audit-grade. You still need a documented human review.
Repos with huge release notes every week: feeding 50KB of changelog into a free model gets expensive and slow. Truncate smarter or only process major bumps.

The human stays in the loop

Automation should compress the boring part of dependency upgrades — reading three changelogs and squinting at a diff — while leaving the judgment call for a person. A cheap model on a cron schedule is the right tool for that compression.

Next time Dependabot opens a PR, spend 30 seconds wiring up a triage label instead of clicking merge with your eyes half closed. The label won't make the decision for you, but it will make sure you never skip the update that deserves your attention.

Classify CI Failures Before You Read the Logs: A Free-Tier Webhook Setup

Emery Lin — Sun, 30 Aug 2026 11:24:15 +0000

CI failures are not all equal. Some are real bugs. Many are flakes that eat your afternoon. You don't need to read every log to tell them apart — a small model can do a first pass. The trick is making that pass reliable and observable, not another black box.

The community has been talking about AI turning every developer into a reviewer without testing the reviewer. This article is the opposite: we build a tiny classifier, deploy it for free, and then test it in CI.

The workflow in one sentence

A failed test step posts its log tail to a webhook. The webhook asks a free-tier model to return flake or bug with a one-line reason. The result lands in your PR as a comment.

Why MonkeyCode for this?

MonkeyCode is an open-source project that gives you 10 million free tokens and a free server for small HTTP services. That combination fits this use case perfectly: low traffic, short logs, and no persistent state.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server means you don't have to touch AWS or heroku. You write one file, deploy, and get a URL. The free tokens mean this classifier costs you nothing for realistic experiments. The exact quota may change, so check the repo before you rely on it.

Step 1: Define the contract

The webhook accepts a JSON POST:

{"log_tail": "ERROR: connection reset by peer..."}

And returns:

{"verdict": "flake", "reason": "Network error outside test scope."}

Keep it tiny. The model only needs the last few KB of a log.

Step 2: Deploy the classifier

Here's a minimal Python service. It's deliberately short. The model endpoint is a placeholder because MonkeyCode's API is subject to change — check the docs.

# classifier.py
import os, json, requests
from fastapi import FastAPI, Request

app = FastAPI()
MODEL = os.getenv('MONKEYCODE_MODEL', 'default-free')

SYSTEM = 'You classify CI failure log tails. Respond ONLY with JSON like {"verdict":"flake|bug|unknown","reason":"short reason"}.'

@app.post('/classify')
async def classify(req: Request):
    body = await req.json()
    tail = (body.get('log_tail') or '')[-4000:]
    resp = requests.post(
        'https://api.monkeycode.dev/v1/chat/completions',  # verify in docs
        headers={'Authorization': 'Bearer ' + os.environ['MONKEYCODE_API_KEY']},
        json={
            'model': MODEL,
            'messages': [
                {'role': 'system', 'content': SYSTEM},
                {'role': 'user', 'content': 'Log tail: ' + tail}
            ],
            'temperature': 0.2,
        },
        timeout=20,
    )
    result = resp.json()['choices'][0]['message']['content']
    return json.loads(result)

This is not production-grade error handling. It's enough for a thin experiment.

Step 3: Hook it into GitHub Actions

You don't need a special action. A regular workflow step can capture the log and curl the webhook.

- name: Run tests
  run: npm test 2>&1 | tee test.log
  continue-on-error: true

- name: Triage
  if: failure()
  env:
    WEBHOOK: ${{ secrets.CLASSIFIER_URL }}
  run: |
    tail -c 4000 test.log > tail.txt
    python - <<'PY'
import json, os, urllib.request
payload = {'log_tail': open('tail.txt').read()}
req = urllib.request.Request(
    os.environ['WEBHOOK'] + '/classify',
    data=json.dumps(payload).encode(),
    headers={'Content-Type': 'application/json'},
)
with urllib.request.urlopen(req) as resp:
    print(resp.read().decode())
PY

That prints the verdict in CI logs. You can then send it to a PR comment using actions/github-script if you want.

Step 4: Test the tester

This is the part most people skip. You don't ship a function without unit tests. Why ship a model call without fixtures?

Add a small test that runs inside CI:

# test_classifier.py
import os, requests

FIXTURES = [
    ('''error: Error: connect ECONNREFUSED 127.0.0.1:5432
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1141:16)''', 'flake'),
    ('''TypeError: Cannot read properties of undefined (reading 'map')
        at renderDashboard (Dashboard.jsx:42)''', 'bug'),
]

def test_verdicts():
    for log, expected in FIXTURES:
        r = requests.post(os.environ['CLASSIFIER_URL'] + '/classify',
                          json={'log_tail': log}, timeout=30)
        assert r.json()['verdict'] == expected, f'{log}: {r.json()}'

Run this on every PR. If the model's behavior drifts, your CI catches it.

Limitations

A free model will misclassify. The unknown verdict exists for a reason. Keep humans in the loop: the classifier only proposes, it never blocks a merge.

The free server may go cold. First request could take a few seconds. That's fine for a PR comment, not for a 99th-percentile SLA.

Your logs go to an external service. If your codebase has secrets in test output, you shouldn't use this as-is. Did you redact log lines? No? Then don't ship it.

Who should not use this

Teams with strict data-residency requirements.
Projects where misclassification could cause a security incident.
Developers who want a merge gate. Don't make a probabilistic model your only gate. Use it as a triage aid, not a judge.

Try it

The setup is small enough to test in a weekend. Grab a free MonkeyCode token allowance, deploy the tiny server, and point a CI workflow at it. You'll learn more about your failures than the model does.

Zero-Budget Prompt Labs: What Free Models Actually Afford You (and How to Size One)

Emery Lin — Sat, 29 Aug 2026 10:33:48 +0000

A free model quota reads like a discount, but the real bottleneck is sizing: you cannot know how many daily checks those tokens buy until you count them. A small regression lab, defined by one formula and one replay script, can run about two hundred evaluations per day when each check stays under two hundred tokens. This article shows how to size that workload honestly, build the runner, and schedule it on a free server. The reference implementation is the open-source MonkeyCode project, and every step transfers to any OpenAI-compatible endpoint.

MonkeyCode's free tier includes free models and a free server, which is enough for the lab described below. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Token names, quotas, and server limits change, so check the current values in the repository before you build a routine on them.

Why monthly model releases make sizing urgent

Every release cycle brings a new benchmark chart and a wave of migration posts aimed at developers who never measured their own usage. The danger is not the model itself; it is the assumption that a bigger model always pays for its cost. Free models remove the price risk but leave the waste risk: a nightly job that re-runs every transcript burns quota on signals you already have.

A regression lab solves the waste problem because it samples, compares, and stops. It answers one narrow question: does the current prompt match the behavior we recorded last month? Benchmarks cannot answer that, and neither can a new model release. Your own transcripts are the only evidence that matters, so the evaluation workload has to be sized against your traffic, not against a leaderboard.

Step 1: Count a real workload in tokens

The correct unit for planning is tokens per day, not prompts per day. Define the workload with this formula:

daily_tokens = Σ over tasks of (weekly_traffic × sample_rate × (input_tokens + output_tokens))

Assume summaries average 600 input tokens and 120 output tokens, reviews average 900 and 150, and explainers average 400 and 80. With 1,500 weekly summary calls, a 5% sample gives 75 evaluations a day; reviews at 4% sample give 28; explainers at 10% give 40. The daily total lands near 36,000 tokens, which fits a plan that starts with tens of millions per month.

That calculation teaches a discipline: raise the sample rate only for tasks whose failure is expensive. The sizing step is the artifact that survives model changes, because your traffic data stays your own.

Step 2: Build a small replay-and-score runner

The runner replays a JSONL file of transcripts, scores each output, and prints a daily aggregate. It is deliberately endpoint-agnostic, so it works with any OpenAI-compatible API, including the free models exposed through MonkeyCode. The code below is a minimal, unexecuted example, so adapt it to your endpoint before relying on it.

import json

def load_transcripts(path):
    with open(path, encoding="utf-8") as fh:
        return [json.loads(line) for line in fh if line.strip()]

def score(pair):
    # Replace with keyword overlap, rubric, or a second model call.
    expected, actual = pair["expected"], pair["actual"]
    overlap = len(set(expected.split()) & set(actual.split()))
    return overlap / max(1.0, len(set(expected.split())))

def run(path, complete, threshold=0.8):
    rows = load_transcripts(path)
    scores = [score({"expected": row["expected"], "actual": complete(row["input"])}) for row in rows]
    passed = sum(s >= threshold for s in scores)
    print(f"n={len(rows)} pass_rate={passed / len(rows):.2%}")
    return passed / len(rows)

The complete function is the only integration point; point it at the endpoint that hosts your current prompt. Keep the scorer simple at first, because a complicated scorer becomes a second project that also needs evaluation.

Step 3: Schedule the lab on a free server

A daily report has to run without a laptop. The free server from MonkeyCode can host the script as a cron job, and the same idea works on any small box you already rent or own.

0 2 * * * cd /opt/prompt-lab && python runner.py --transcripts data/sample.jsonl >> reports/daily.log 2>&1

Pin the prompt version in the command or in a config file, so the report always tells you which prompt and which model produced the numbers. Logs become the audit trail for future decisions; treat them as a primary output, not a byproduct.

Step 4: Turn the daily score into a decision matrix

A single number means little until you tie it to a decision. The matrix below maps the daily pass rate to concrete actions, and it avoids the trap of alarming on noise.

Daily pass rate	Movement	Action
≥ 95%	stable or rising	log only, refresh sample monthly
85–95%	falling more than 3 points in a week	double the sample rate, inspect failing transcripts
< 85%	any drop	pause prompt rollout, rerun against previous prompt tag

The goal is not to make the number green; it is to know exactly what you can and cannot conclude from it. A falling score next to a frozen sample means the fixture set is stale, not necessarily that the model regressed.

Step 5: Feed every failure back into the sample

When a transcript fails, it belongs in the labeled set with a short note about why the output was wrong. The sample expands, the threshold may move, and the whole lab becomes more sensitive exactly where your traffic is.

If the pass rate stays flat for months, resist the urge to celebrate; it probably means the fixtures stopped reflecting current behavior. Rebuild the sample from production logs every month and re-run the sizing formula each quarter.

When the free tier is the wrong tool

Free models do not offer the strongest reasoning on hard multi-step tasks, so the lab is a regression detector, not a quality oracle. For safety-critical outputs, human review and formal validation still win.

The free server is a shared resource, and cron delays are possible; do not attach it to a live checkout or a payment flow. Quotas rotate, so long-term scheduling requires a re-check of the terms before each quarter. And a model-side claim that cannot be reproduced on your recorded transcripts carries no weight in this lab.

The deliverable is the accounting, not the score

The lasting value of the lab is not any green number; it is the method that tells you what your traffic costs in tokens and what your prompts do on real inputs. When the next model release dominates the headlines, you can respond with that accounting instead of a hunch.

Start with the free models and the free server from MonkeyCode, run the sizing formula against your own logs, and keep the runner script in a git repository. Verify the current limits first, then let your traffic decide whether the free tier is enough.

Your AI Review Bot Double-Posts. A Webhook Replay Test in CI Fixes It.

Emery Lin — Fri, 28 Aug 2026 04:00:41 +0000

When an AI review bot misbehaves, the model is usually innocent. The webhook arrived twice. The free server cold-started for 40 seconds, GitHub retried the delivery, and your bot analyzed the same diff twice, commented twice, and burned tokens twice.

This fails more often than a wrong review. And a better prompt will not fix it. What fixes it is plumbing: idempotency keys, replayed webhook fixtures, and a CI job that treats those fixtures as the test suite for your bot. That pattern is what this article builds, step by step. It runs entirely on a free server with free model access, so you can reproduce it without spending anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The failure is the webhook, not the model

Webhook delivery is retry-based. GitHub documents that deliveries are sent again when the endpoint fails (docs). On a free server, cold starts make this worse: the first request wakes the process, the sleep adds latency, the timeout trips, and the retry lands on a fresh process that never marked the first event as handled.

The hot take on DEV this week is that review quality is the hard problem. In practice, delivery reliability fails first. Once you fix delivery, quality discussions actually mean something.

The three failure classes

Duplicate comments — the retry produces a second review and a second comment on the PR.
Duplicate spend — every retry burns tokens against the same diff. On a free allowance, this is how a quiet week becomes a quota surprise.
Missed events — if the bot crashes before doing any work, the event is silently lost.

All three are delivery bugs. None of them is fixed by a new system prompt.

Design rule: mark before the slow part

The core rule is mark before, not after. Deduplicate the event before you call the model, because the model call is the slow, expensive, and killable part. This is the smallest FastAPI webhook that follows the rule:

import hashlib
from fastapi import FastAPI, Request

app = FastAPI()

def event_key(delivery_id: str, repo: str, pr: int, action: str) -> str:
    return hashlib.sha256(f"{delivery_id}:{repo}:{pr}:{action}".encode()).hexdigest()

seen = {}

@app.post("/webhook")
async def webhook(request: Request):
    h = request.headers
    delivery_id = h.get("x-github-delivery") or h.get("x-gitlab-event-uuid")
    body = await request.json()

    if not delivery_id:
        return {"status": "ignored"}  # manual or fixture calls only

    key = event_key(delivery_id, body["repository"]["full_name"],
                    body["pull_request"]["number"], body["action"])

    if key in seen:
        return {"status": "duplicate", "key": key}

    seen[key] = True  # before the model call, not after

    review = await run_review(body)   # free model tier, via MonkeyCode
    await post_comment(body, review)
    return {"status": "reviewed", "key": key}

The ordering is the whole trick. If the process dies after the model call but before the comment, the retry finds key in seen and skips the expensive part. You still need a sweep to post the missing comment, but you no longer double-analyze. That one line halves the cold-start blast radius.

I ran this against recorded webhooks using the free server option from MonkeyCode, with model access on its free tier. Same code, same fixtures — only the deployment target was a no-cost server.

Build the fixture corpus

You cannot test delivery with payloads you invented by hand. You capture real ones. Over a normal week of PR activity, I recorded four classes of event:

fixtures/
  pr.opened.base.json          # the first delivery
  pr.opened.retry.json         # same payload, new delivery id
  pr.synchronize.second-commit.json
  push.no-pr.json              # should be ignored, cheap to assert

The setup is three steps:

Add a minimal logger to your webhook that stores every raw payload plus the delivery header.
Sanitize what you store: usernames, repo names, and secrets get replaced so the fixtures are safe to commit.
Save one retry pair intentionally. When the provider retries, you get two deliveries of the same body with different ids — that pair is your most valuable fixture.

The CI replay job

Now the fixtures become a test suite. Every change to the bot — a new dependency, a refactor, even a prompt change — must survive a replay of the week's events. A pytest test for the retry pair looks like this:

def test_retry_does_not_duplicate(client, fixtures):
    first = client.post("/webhook",
                        headers=fixtures.headers("pr.opened.base"),
                        json=fixtures.body("pr.opened.base"))
    second = client.post("/webhook",
                         headers=fixtures.headers("pr.opened.retry"),
                         json=fixtures.body("pr.opened.base"))
    assert first.status_code == 200
    assert first.json()["status"] == "reviewed"
    assert second.json()["status"] == "duplicate"

Add a second test that replays pr.opened.base against a fresh in-memory store and asserts a comment was posted. Add a third that replays push.no-pr.json and asserts no comment and no model call. Together they pin the behavior production actually depends on.

Then wire the job into your workflow:

name: bot-replay
on:
  pull_request:
    paths: ["webhook/**", "fixtures/**", "tests/**"]

jobs:
  replay:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest tests/ -q

That is the green-to-merge path. Branch protection requires bot-replay to pass, exactly like any other unit test. The bot is no longer invisible machinery; it is code with a regression suite.

Where flake still lives — and how to budget it

You cannot make the model deterministic, and you should not pretend otherwise. The point of the replay job is to separate your flakes from the model's flakes. The decision table I use:

Observed failure	Where it lives	Budget	Action
Duplicate comment	your dedupe	zero	block the merge
Missing comment after retry	crash before mark	zero	block the merge
Same diff, different wording	model nondeterminism	tolerated	inspect, don't block
Provider 503 on model call	model service	a few per day	retry with backoff

That table is the flake control. Zero-budget failures fail CI. Tolerated ones produce a warning artifact instead. This is the honest way to run LLM output in a test suite: assert the plumbing exactly, assert the prose loosely.

Who should not use this pattern

Be direct about the limits. A free server is a shared, reclaimable resource: cold starts happen, the process can be stopped, and your retry window is not guaranteed. A free model tier can change its quotas without notice. So this pattern is for a project bot, a personal workflow, or a low-stakes automation — not for a compliance checkpoint, a billing system, or anything where a missed review has regulatory teeth. If your org needs a strict SLA, rent the paid version of the same idea.

The takeaway

Review bots fail in delivery more often than in judgment. Fix the delivery with idempotency, capture real webhooks as fixtures, and replay them in CI on every change. That turns a flaky black box into a testable service with a green-to-merge path.

If you want to run this exact pattern end-to-end: MonkeyCode's free tier, as of late August 2026, includes a 10M token model allowance and a free server option — enough for a week of replays and several PR reviews. Free tiers change, so check the current terms before you depend on them. I do.

A Merge Gate for Prompt Changes: Budgeted CI Evals on a Free Server

Emery Lin — Thu, 27 Aug 2026 03:07:16 +0000

Liquid syntax error: Unknown tag 'endraw'

An AI Review Gate That Doesn't Flake: Hooks, Fixtures, and a Green-to-Merge Path

Emery Lin — Wed, 26 Aug 2026 11:24:05 +0000

An AI review agent in CI is only worth the trouble if it fails loudly, flakes rarely, and never blocks a merge on vibes. This article walks through a pipeline design that does all three: a hook that fires on every pull request, a fixture suite that tests the reviewer before it reviews, and a severity-based gate with a visible escape hatch. The cost model matters too. With free model access and a free server option, the per-PR price of this gate is zero — and that changes when you can afford to run it.

Why most AI review gates annoy everyone

Three failure modes kill AI review gates in practice.

First, cost. A paid model call on every PR adds up fast, so teams run the reviewer only on big PRs — exactly when it's least useful. Second, infrastructure. A dedicated runner that sits idle between PRs is a bill you can see. Third, flakiness. The same diff gets a pass on Monday and a block on Tuesday, and nobody trusts the gate anymore.

This design addresses all three. The agent runs in a short-lived CI job on the free server option. The model calls use free model access. And a fixture suite keeps the output honest.

The pieces

MonkeyCode is an open-source AI coding agent you can run from the command line. In this pipeline it plays one role: turn a diff into a structured review report. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The pipeline has four parts: a hook, a wrapper, a gate, and a fixture suite. You can steal all four and swap the agent for any CLI tool that emits JSON.

Step 1: The hook

The hook is a GitHub Actions workflow. It fires on opened and synchronize so the gate re-runs when you push. It also fires on labeled and unlabeled — that's the override mechanism.

name: ai-review-gate

on:
  pull_request:
    types: [opened, synchronize, labeled, unlabeled]

permissions:
  contents: read
  pull-requests: write

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Run the reviewer
        id: review
        run: |
          ./scripts/review_agent.sh \
            --base "${{ github.event.pull_request.base.sha }}" \
            --head "${{ github.event.pull_request.head.sha }}" \
            --out /tmp/review.json

      - name: Upload the report
        uses: actions/upload-artifact@v4
        with:
          name: review-report
          path: /tmp/review.json

      - name: Gate on severity
        env:
          HAS_OVERRIDE: ${{ contains(github.event.pull_request.labels.*.name, 'override:ai-review') }}
        run: |
          if [ "$HAS_OVERRIDE" = "true" ]; then
            echo "override:ai-review present — passing with override logged"
            exit 0
          fi
          python scripts/gate.py /tmp/review.json

Two details matter. The fetch-depth: 0 checkout gives the wrapper the full history it needs to diff two SHAs. And the gate step reads the override label from the event payload, not from the repo, so a maintainer can unblock a PR without touching the branch.

Step 2: The wrapper

The wrapper keeps the agent call in one place. It takes two SHAs, produces a diff, and hands it to the agent. The CLI shape below is a placeholder; check your installed agent version for the exact flags.

#!/usr/bin/env bash
set -euo pipefail

BASE=""; HEAD=""; OUT=""; DIFF=""

while [[ $# -gt 0 ]]; do
  case "$1" in
    --base) BASE="$2"; shift 2 ;;
    --head) HEAD="$2"; shift 2 ;;
    --diff) DIFF="$2"; shift 2 ;;
    --out) OUT="$2"; shift 2 ;;
    *) echo "unknown argument: $1" >&2; exit 2 ;;
  esac
done

if [[ -n "$DIFF" ]]; then
  DIFF_FILE="$DIFF"
else
  DIFF_FILE=$(mktemp)
  git diff --unified=3 "$BASE" "$HEAD" > "$DIFF_FILE"
fi

monkeycode review \
  --diff "$DIFF_FILE" \
  --repo "$PWD" \
  --format json \
  --out "$OUT"

Keep the wrapper thin. The fixture suite will call it hundreds of times.

Step 3: The fixtures

Here's the part most people skip. Before you let an AI gate block real PRs, you need to know what it does on diffs you already understand. Create a fixtures/ directory with four kinds of cases:

bug.diff — a real, injected bug (off-by-one, missing null check)
security.diff — an injection or a leaked secret
clean.diff — a clean refactor with no findings
noise.diff — style nits only

A runner executes the reviewer against each fixture and compares the result with an expected JSON file.

#!/usr/bin/env bash
set -euo pipefail

PASS=0; FAIL=0

for fixture in fixtures/*.diff; do
  name=$(basename "$fixture" .diff)
  expected="fixtures/${name}.expected.json"

  ./scripts/review_agent.sh --diff "$fixture" --out "/tmp/${name}.json"

  if python scripts/compare_findings.py \
      --actual "/tmp/${name}.json" \
      --expected "$expected"; then
    PASS=$((PASS + 1))
  else
    FAIL=$((FAIL + 1))
    echo "FAIL: $name"
  fi
done

echo "fixtures passed: $PASS, failed: $FAIL"
[ "$FAIL" -eq 0 ]

compare_findings.py compares the severity and location of each finding against the expected JSON — strict on critical, lenient on nits. Three numbers come out of the suite: recall (does it find the injected bug?), false positives (does it stay quiet on clean?), and stability (does it give the same answer three times in a row?). If any of the three is bad, the gate is not ready for real PRs.

Step 4: Flake control

Flakes are what kill trust in AI gates. Five rules keep them down:

Pin the model version in the agent config. Unpinned models drift, and the gate changes behavior without a code change.
Set temperature to 0 if the model supports it. Review output should be reproducible.
Retry only on transport or parse errors, never on content disagreement. A model that says "clean" twice and "bug" once is a flake you want to see, not retry away.
Give the agent an explicit abstain path. A low-confidence "maybe" should never block a merge.
Run every prompt or config change through the fixture suite before it reaches the gate.

Step 5: The gate

The gate is a small script that reads the report and exits 0 or 1. GitHub turns that into a required status check.

#!/usr/bin/env python3
import json
import sys

report = json.load(open(sys.argv[1]))
blocking = [f for f in report.get("findings", [])
            if f.get("severity") == "critical"]

for finding in blocking:
    print(f"BLOCK: {finding.get('file')} — {finding.get('message')}")

sys.exit(1 if blocking else 0)

The decision table is the contract between the agent and the team:

Severity	Example	Gate behavior
critical	SQL injection, auth bypass, secret in diff	Block merge; human review required
warning	null deref, resource leak, race	Comment on the PR; no block
nit	naming, formatting	Ignored
abstain	confidence below threshold	Pass; logged for audit

Step 6: The green-to-merge path

Here's the full loop:

You push a PR. The hook fires.
The reviewer runs in a short-lived job on the free server option. A few minutes per PR is the right shape for this; it's not a production runner.
The report is uploaded as an artifact. Every decision is auditable later.
The gate exits 0 or 1. No critical findings means the check is green.
A critical finding turns the check red. A maintainer can add the override:ai-review label, which re-triggers the workflow and lets the gate pass — but the override is visible in the log and the label history, not silent.

The override is the part that keeps the gate honest. An AI reviewer that cannot be overridden becomes a process problem. One that logs its overrides stays a tool.

What this costs

The reason this pattern is worth building now is the cost curve. The job runs on the free server option, and the model calls use free model access. As of this writing, the free tier includes 10 million tokens, which covers a lot of small diffs; check the project repo for current terms, because free tiers change.

Compare that with the alternative: a paid model on a dedicated runner, used only on PRs that feel risky. The free tier flips the default. You can afford to review every PR, and the fixture suite keeps the reviewer cheap to trust.

Limitations and who should skip this

The reviewer sees a diff, not the whole codebase. It will miss cross-file issues that a human with context would catch. Free model access usually means shared or rate-limited resources, so don't build a gate that needs a 99.9% SLA. And the fixture suite is a maintenance cost: if the fixtures rot, the gate lies.

Skip this pattern if you're under a compliance regime that requires human-only review, or if your PRs are so large that the diff doesn't fit the model's context. If your team won't maintain the fixtures, the gate will quietly become noise — which is worse than no gate at all.

Start with four diffs

The fixture directory is the real deliverable here. Start with four diffs, wire the hook, and let the gate earn trust before you make it required. If you want to try the pattern, the MonkeyCode repo has the agent and the free tier — the scripts above are a good starting point.

A Daily Commit Digest on Free LLM Credits: A Case Study in Low-Stakes Automation

Emery Lin — Fri, 21 Aug 2026 13:28:27 +0000

A Daily Commit Digest on Free LLM Credits: A Case Study in Low-Stakes Automation

Free-tier LLM access is enough for a surprising amount of real work — if you pick the right job. This case study covers one such job: a daily commit digest. It summarizes the last 24 hours of changes in a GitHub repository. The pipeline runs on a cron job, a free server, and a free model allowance. You can reproduce it in an afternoon. The failure modes it exposes matter more than the prompts.

Background

The problem is familiar. Your repository has more commits than you can read daily. Changelogs lag. Release notes get written from memory. A small script can fix that: fetch commits, summarize them, publish the digest. The real question is whether you can run that script without paying for infrastructure or burning expensive API credits. This project answers that question with a concrete design.

Goal

Define success before you write code. For this project: one repo, one digest per day, one human reader who can spot a wrong summary. A successful week means seven digests, zero paid API calls, and no silent failures. Everything else is negotiable.

Implementation

1. Fetch the commits

GitHub's REST API gives you everything you need. One request per day sits far inside the unauthenticated rate limit. A public repo needs no token at all.

# fetch_commits.py — works with any public GitHub repo
import os
import requests
from datetime import datetime, timedelta, timezone

REPO = os.getenv("DIGEST_REPO", "owner/name")
SINCE = (datetime.now(timezone.utc) - timedelta(hours=24)).isoformat()

resp = requests.get(
    f"https://api.github.com/repos/{REPO}/commits",
    params={"since": SINCE, "per_page": 100},
    timeout=30,
)
resp.raise_for_status()
commits = resp.json()
print(f"fetched {len(commits)} commits")

2. Compress the input

Do not send raw diffs to the model. Extract the short hash, the first line of the message, and the author. This keeps the token count low. It also forces the digest to stay readable.

def compress(commits):
    lines = []
    for c in commits:
        message = c["commit"]["message"].splitlines()[0]
        author = c["commit"]["author"]["name"]
        lines.append(f"- {c['sha'][:7]} {message} ({author})")
    return "\n".join(lines)

3. Constrain the prompt

The prompt is the contract. It fixes the output format and caps the summary length. Most importantly, it tells the model it is allowed to say "nothing notable". Quiet days are real. Without that instruction, the model will invent drama to fill them.

PROMPT = """You write a daily changelog for a software repository.

Below are the commits from the last 24 hours. Produce:
1. A one-paragraph summary (max 80 words).
2. A bullet list of notable changes (max 5 bullets).
3. A "risk flags" line if any commit touches tests, dependencies,
   migrations, or security-related files. Otherwise write "no flags".

If the list is empty or trivial, reply with exactly:
"Nothing notable in the last 24 hours."

Commits:
{commits}
"""

Example output (format only; your model will phrase it differently):

Summary: 14 commits landed, mostly around the new export endpoint.
Notable: export now supports CSV; retry logic added to the sync client.
Risk flags: dependency bump in requirements.txt — verify compatibility.

4. Call the free model

MonkeyCode's free model access is what makes this project viable without a paid API budget. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The integration is a single chat-completion call. The exact client name depends on the runtime you deploy to, so treat the snippet below as the shape of the call, not the import.

# summarize.py — pseudocode; the client API varies by runtime
def summarize(commits_text: str) -> str:
    client = get_client()  # configured from environment variables
    response = client.complete(
        system="You summarize git history. Be concise.",
        user=PROMPT.format(commits=commits_text),
    )
    return response.text

A run with 50 commits typically lands in the low thousands of tokens. At one run per day, the current free allowance of 10 million tokens is far more than this workload needs. Do the arithmetic for your own frequency before you scale anything.

5. Deploy on the free server

MonkeyCode's free server option gives the cron job a home. For a job this size you do not need a VPS, a container registry, or a cloud account.

0 8 * * * cd ~/commit-digest && python digest.py >> digest.log 2>&1

6. Plan the degradation

The most important design decision is not the prompt. It is what happens when the model call fails. Do not fail silently. Fall back to the raw commit list. A plain list of commits is still useful; a missing digest is not.

def main():
    commits = fetch_commits()
    text = compress(commits)
    try:
        digest = summarize(text)
    except Exception as exc:
        digest = f"Model call failed ({exc}). Raw commits:\n{text}"
    write_digest(digest)

7. Keep a ledger

Append every run to a markdown file: date, commit count, model status, digest, and a human verdict column. This is the audit trail that tells you whether the free tier is actually good enough for your repo.

| date       | commits | model | verdict |
|------------|---------|-------|---------|
| 2026-08-21 | 14      | ok    |         |

Results

The results section of this case study is a template. Run the project for seven days and fill in the verdict column. Three failure modes matter:

The digest missed a breaking change.
The digest invented a change that did not happen.
The digest buried the one commit that mattered.

Each is a different failure. Missing a change is a recall problem. Inventing a change is a hallucination problem. Burying the important commit is a prompt problem. The ledger tells you which one you have.

Who should not use this approach

If the digest feeds an automated system, free-tier availability is not enough. You need a paid SLA.
If your repo sees more than a hundred commits a day, the digest stops being a digest.
If the output is security- or compliance-critical, a human must review every line.
If you need sub-minute latency, a cron job is the wrong shape entirely.

Lessons learned

Three lessons carry over to any similar project. First, constraints are the design: the free tier forces you to compress inputs, and the digest is better for it. Second, degradation beats retries: a fallback to raw data is more honest than a silent failure. Third, the ledger is what makes the output trustworthy. Without the human verdict column, you are trusting the model on faith. With it, you have data.

Free-tier LLM infrastructure is not a toy. It is a tool with a specific shape: low frequency, low stakes, and a human in the loop. A daily commit digest fits that shape exactly. If you want to try it, MonkeyCode's free model access and free server option are a reasonable starting point. The rest of the design transfers to any provider you prefer.

Cheap Model First, Strong Model on Failure: Building an Auditable Two-Tier LLM Pipeline

Emery Lin — Thu, 13 Aug 2026 04:00:12 +0000

There's a pattern I see every release cycle: a new budget-friendly model ships, the discourse explodes with hot takes, and within 48 hours half my feed has declared it a drop-in replacement for everything. The claim might even be true. But here's what nobody posting those takes can tell you: whether it's true for your specific workload. And most of the time, that's the only question that matters.

A healthier mental model: stop treating model selection as a one-time shopping decision and start treating it as a runtime policy. Route work to the inexpensive option by default, check the output with something that isn't a model, and only pay for the heavyweight option when the check fails. Below is a working implementation of that policy, plus the measurement discipline that turns it from a hunch into an auditable system.

Two lanes, not two opinions

My configuration is deliberately boring:

Lane A (try first): the current low-cost or free option. I'm intentionally not naming a specific checkpoint in this article — model names decay faster than the code below does, and you should be re-benchmarking against whatever shipped this month anyway.
Lane B (fallback): the pricier, more capable model, invoked only when Lane A's output fails an objective check.

On the cost side, one note: I iterate on this pipeline using MonkeyCode, which at the time of writing provides free model access along with a free server option, so experimentation doesn't rack up a bill. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The pipeline itself is provider-agnostic — anything speaking the OpenAI-compatible chat API drops in, so treat endpoints as configuration, not commitment.

The core rule: escalation needs a non-model referee

The single most important design decision: never let a model decide whether its own output (or a peer's) was good enough. LLMs report confidence regardless of correctness. Instead, escalate only when an external, deterministic check fails — a test suite, a parser, a schema validator, a diff against expected output.

Here's a compact implementation. Standard library plus requests, fully rerunnable, and every routing decision gets written to a JSONL audit log:

# lane_router.py
import hashlib, json, subprocess, time
from dataclasses import dataclass, asdict

import requests

@dataclass
class RouteRecord:
    job_id: str
    lane: str            # "A" or "B"
    fell_back: bool
    check_ok: bool
    seconds: float
    prompt_fingerprint: str

def chat(endpoint: str, model: str, prompt: str) -> str:
    resp = requests.post(
        f"{endpoint}/v1/chat/completions",
        json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.2,
        },
        timeout=120,
    )
    resp.raise_for_status()
    return resp.json()["choices"][0]["message"]["content"]

def objective_check(job: dict, candidate: str) -> bool:
    """Deterministic validation only. No LLM-as-judge allowed here."""
    mode = job["check"]
    if mode == "pytest":
        path = job["write_to"]
        with open(path, "w") as fh:
            fh.write(strip_fences(candidate))
        run = subprocess.run(job["command"], shell=True,
                             capture_output=True, timeout=90)
        return run.returncode == 0
    if mode == "valid_json":
        try:
            parsed = json.loads(strip_fences(candidate))
        except json.JSONDecodeError:
            return False
        required = job.get("required_keys", [])
        return all(k in parsed for k in required)
    if mode == "regex":
        import re
        return re.fullmatch(job["pattern"], candidate.strip()) is not None
    raise ValueError(f"unsupported check: {mode}")

def strip_fences(text: str) -> str:
    """Pull code out of a markdown fence if present; else return as-is."""
    if "```

" not in text:
        return text
    block = text.split("

```")[1]
    lines = block.splitlines()
    if lines and lines[0].strip().isalpha():  # language tag line
        lines = lines[1:]
    return "\n".join(lines)

def route(job: dict, lanes: list[dict], log_path: str = "routes.jsonl") -> RouteRecord:
    fp = hashlib.sha256(job["prompt"].encode()).hexdigest()[:12]
    fell_back = False
    for idx, lane in enumerate(lanes):
        start = time.time()
        candidate = chat(lane["endpoint"], lane["model"], job["prompt"])
        elapsed = round(time.time() - start, 2)
        passed = objective_check(job, candidate)
        if passed or idx == len(lanes) - 1:
            record = RouteRecord(
                job_id=job["id"],
                lane=lane["label"],
                fell_back=fell_back,
                check_ok=passed,
                seconds=elapsed,
                prompt_fingerprint=fp,
            )
            with open(log_path, "a") as fh:
                fh.write(json.dumps(asdict(record)) + "\n")
            return record
        fell_back = True

Wiring it up:

lanes = [
    {"label": "A", "endpoint": "https://lane-a-endpoint", "model": "current-budget-model"},
    {"label": "B", "endpoint": "https://lane-b-endpoint", "model": "premium-model"},
]

job = {
    "id": "csv-to-json-migration-014",
    "prompt": (
        "Convert the transformation in migrate.py so it emits newline-delimited JSON. "
        "Return only the complete updated file inside a code fence."
    ),
    "check": "pytest",
    "write_to": "migrate.py",
    "command": "python -m pytest tests/test_migrate.py -q",
}

print(route(job, lanes))

The audit log is the actual deliverable

The router code is maybe a weekend of effort. The compounding value lives in routes.jsonl. After a few weeks of real traffic you can compute things that are otherwise pure speculation:

Fallback rate per job family. Bucket records by job type and look at fell_back. If data-formatting jobs pass on Lane A 92% of the time, Lane A is a rational default there. If multi-file refactors fail 60% of the time, Lane A is a false economy for that family — you're paying for a doomed first attempt plus added latency on most calls.
Effective cost per successful task. One Lane A call at $X that succeeds beats one Lane B call at $5X. A Lane A call that fails and triggers Lane B costs $6X. Do the arithmetic per category; the answer is frequently different per category, which is the entire point of routing instead of picking one model globally.
Regression testing new releases. When the next checkpoint drops, point Lane A at it and replay your logged job corpus (keep the prompts, or a sanitized corpus, alongside the log). Comparing fallback rates across model versions on your jobs beats any public leaderboard for your decision.

Operating rules I'd insist on:

The referee is deterministic, full stop. The moment you add "ask a second LLM to grade it," you've rebuilt the confidence problem with extra latency. Tests, parsers, schemas, regexes, exact-match — nothing that hallucinates.
Fingerprint prompts, don't necessarily store them. A SHA-256 prefix lets you dedupe and correlate without persisting potentially sensitive content in the log.
Two lanes maximum. A three-tier cascade reads like clever engineering and behaves like a latency generator with a bigger debug surface. If Lane B fails the check, that's a human's problem, not a third model's.

Where this breaks down

Jobs without an oracle. This approach is only as strong as its checks. Code with tests, structured extraction, format conversions — great. Open-ended prose, architectural judgment, "summarize this for an exec" — there is no deterministic referee, and routing those on vibes is precisely the guesswork this pipeline exists to eliminate.
Synchronous, latency-bound paths. A failed Lane A attempt plus a Lane B retry can roughly double wall-clock time. Keep the router in background workers, batch jobs, and async pipelines until you have p99 numbers proving otherwise.
Assuming free stays free. Free model access and free server options are statements about the present, not contracts. The endpoints-as-config design above is your insurance policy; don't hardcode a provider's generosity into your architecture.
Tiny samples. Fifteen routed jobs prove nothing. I don't trust a fallback-rate number until a job family has roughly 50+ records behind it, and even then I look at the distribution, not just the mean.

If most of your workload is unverifiable generation, or everything you run is on a hard latency budget, honestly — skip the router. Picking the strong model outright is the simpler and more correct engineering decision in that world.

A concrete starting point

Pull your last ~50 real prompts, sort them into "has a deterministic check" versus "doesn't," and push the checkable subset through the two-lane setup for a week. If you want the measurement phase to cost nothing, MonkeyCode's free model access and free server option work fine as Lane A and host while you gather data — and since the log format is provider-neutral, whatever you learn transfers when you point the lanes elsewhere.

The question "is the cheap model good enough?" has an answer, and it's already sitting in your prompt history. Measure it; don't outsource the decision to launch-week sentiment.

The Release-Day Reality Check: A Small Model Evaluation You Can Rerun

Emery Lin — Mon, 10 Aug 2026 11:00:22 +0000

A new model announcement can create useful evidence very quickly, but it rarely creates evidence about your work. Release posts usually answer broad questions; a developer evaluating a coding assistant needs narrower ones:

Can it repair the kind of defect I encounter?
Does it respect the libraries and style already in the project?
Can it follow a small interface contract?
Is it responsive enough for the way I work?

Instead of assembling a fresh experiment after every launch, I use a reusable release scorecard. It is small, deliberately personal, and designed to run against any OpenAI-compatible chat endpoint. It will not produce a leaderboard ranking, but it can tell you whether a new option deserves a deeper trial.

Start with a question, not a benchmark number

A useful personal scorecard compares the candidate against a task you already understand. I group mine into four lanes:

Lane	Number of cases	What it exposes
Fault isolation	4	Whether the model finds the likely cause instead of restating symptoms
Controlled edits	4	Whether it changes only the requested behavior
Contract work	3	Whether types, validation, errors, and compatibility survive
Project explanation	3	Whether it can reason from unfamiliar context without inventing files

Fourteen prompts are manageable to review carefully. More importantly, every prompt has an expected outcome because it comes from a repository, ticket, or refactoring exercise I already know.

A case should contain three things: the prompt, the constraints, and a review note describing what a good answer must preserve.

Create cases.json:

{
  "suite": "release-reality-check-v1",
  "tests": [
    {
      "name": "session-cookie-regression",
      "lane": "fault-isolation",
      "prompt": "Users remain authenticated for only one request after login. Here is the middleware and cookie configuration. Find the most probable defect and propose the smallest safe correction.\n\n<redacted source goes here>",
      "constraints": [
        "Do not replace the authentication framework",
        "Explain the failure before editing",
        "Call out any security side effect"
      ],
      "expected": "Identifies the cookie attribute or proxy mismatch, preserves the current session flow, and suggests a focused regression test."
    },
    {
      "name": "price-rounding-contract",
      "lane": "contract-work",
      "prompt": "Update this checkout helper so totals are rounded only at the final display boundary. Preserve the public function signature.\n\n<redacted source goes here>",
      "constraints": [
        "No floating-point money totals",
        "No breaking change to returned fields",
        "Include boundary examples"
      ],
      "expected": "Uses integer minor units or an equivalent decimal representation, keeps the existing API, and covers half-cent boundaries."
    }
  ]
}

The examples above are placeholders rather than executed results. Replace them with cases from your own codebase after removing credentials, customer information, and anything else you cannot send to an inference service.

A portable runner

This Node.js runner avoids a vendor-specific SDK. It records the exact suite, model identifier, response, status, and elapsed time for every attempt.

Save it as run-scorecard.mjs:

import { readFile, writeFile } from "node:fs/promises";
import { createHash } from "node:crypto";

const baseUrl = process.env.MODEL_BASE_URL;
const model = process.env.MODEL_ID;
const apiKey = process.env.MODEL_API_KEY;

if (!baseUrl || !model) {
  throw new Error("Set MODEL_BASE_URL and MODEL_ID before running.");
}

const source = await readFile("cases.json", "utf8");
const suite = JSON.parse(source);
const suiteHash = createHash("sha256").update(source).digest("hex");

const report = {
  suite: suite.suite,
  suiteHash,
  model,
  startedAt: new Date().toISOString(),
  attempts: []
};

for (const test of suite.tests) {
  const prompt = [
    test.prompt,
    "",
    "Constraints:",
    ...test.constraints.map((item) => `- ${item}`)
  ].join("\n");

  const started = performance.now();

  const response = await fetch(`${baseUrl.replace(/\/$/, "")}/chat/completions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      ...(apiKey ? { Authorization: `Bearer ${apiKey}` } : {})
    },
    body: JSON.stringify({
      model,
      temperature: 0,
      messages: [
        {
          role: "user",
          content: prompt
        }
      ]
    })
  });

  const elapsedMs = Math.round(performance.now() - started);
  const text = await response.text();

  let answer = "";
  let parseError = null;

  try {
    const payload = JSON.parse(text);
    answer = payload.choices?.[0]?.message?.content ?? "";
  } catch (error) {
    parseError = error.message;
  }

  report.attempts.push({
    name: test.name,
    lane: test.lane,
    httpStatus: response.status,
    elapsedMs,
    expected: test.expected,
    answer,
    rawResponse: answer ? undefined : text,
    parseError
  });
}

const outputName = `report-${Date.now()}.json`;
await writeFile(outputName, `${JSON.stringify(report, null, 2)}\n`);
console.log(`Wrote ${outputName}`);

Run it with environment variables rather than placing credentials in the repository:

export MODEL_BASE_URL="https://your-compatible-host/v1"
export MODEL_ID="model-under-test"
export MODEL_API_KEY="your-token"

node run-scorecard.mjs

A few details are intentional:

The case-file hash makes it obvious whether two reports used the same prompts.
temperature: 0 reduces avoidable variation, although it should not be treated as a proof of deterministic serving.
Raw error responses are preserved when an endpoint fails or returns an unexpected body.
Timing is stored per case, but it should be read as a rough usability signal—not a controlled benchmark.

Grade against consequences

After the run, score each response without looking at the model name:

Score	Meaning
2	The answer satisfies the expected note and could plausibly be applied after normal review
1	It has the right direction but misses an important constraint or needs substantial repair
0	It misunderstands the problem, invents project details, or creates a risky change

I also mark any response with one or more failure labels:

constraint-ignored
fabricated-context
unsafe-edit
scope-expansion
unverifiable-claim
timeout-or-endpoint-error

The labels are more useful than the total. Two candidates can receive similar scores while failing for completely different reasons. One may be too cautious; another may confidently invent missing APIs. Those behaviors lead to different adoption decisions.

For a cleaner comparison, grade the current assistant first, put the notes away, and then grade the release candidate. If possible, shuffle the reports so the newest model does not automatically receive the benefit of the doubt.

Using a free evaluation environment

Hosted inference is the part that often stops a personal evaluation before it starts. One practical option is MonkeyCode, which currently provides free model access along with a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The runner does not need product-specific code: set its endpoint and model identifier through the environment variables shown above. That separation matters because hosted offerings, available models, limits, and operating conditions can change. Keep the scorecard independent so you can move it to another compatible endpoint or a local server without editing the evaluation logic.

Free access is especially useful for the first pass: deciding whether a model is worth a paid, private, or longer evaluation. It should not be treated as evidence that the same environment is suitable for production traffic.

A compact report format

Once grading is complete, write a short summary that another developer could inspect later:

# Model release check

- Date:
- Candidate identifier:
- Scorecard version/hash:
- Endpoint type:
- Scores by lane:
  - Fault isolation: x/8
  - Controlled edits: x/8
  - Contract work: x/6
  - Project explanation: x/6
- Median observed response time:
- Repeated failure labels:
- Most concerning response:
- Best response:
- Decision:
  - [ ] Ignore for now
  - [ ] Watch for the next revision
  - [ ] Test in a noncritical workflow
  - [ ] Run a larger evaluation

The decision is the important output. A score without an action tends to become another benchmark screenshot.

What this cannot prove

This method is intentionally narrow. It has several limitations:

Small samples have weak statistical power. Fourteen cases can reveal obvious fit problems, but they cannot establish a small percentage improvement.
Personal cases introduce personal bias. Write the expected result and constraints before viewing the response.
Observed latency depends on the hosting environment. Queueing, geography, shared capacity, and request size can dominate the measurement.
A single run is not a regression suite. Rerun the same scorecard when the model, endpoint, prompt, or serving configuration changes.
A hosted endpoint may be inappropriate for sensitive material. If source code cannot leave your environment, connect the same runner to local inference instead.

Skip this approach when making a high-stakes procurement decision, validating production service levels, testing regulated data, or publishing a comparative benchmark. Those situations need representative datasets, controlled infrastructure, documented privacy terms, and proper statistical analysis.

For launch-week triage, however, a stable case pack and a portable runner are usually enough to separate "interesting announcement" from "worth trying in my workflow." If you need somewhere to run that first pass, MonkeyCode is one compatible option; the same scorecard can also be pointed at any other endpoint you trust.

Your AI Coding Agent Has Tools. Here's a Test Suite for the Sandbox You Hope It Respects

Emery Lin — Mon, 10 Aug 2026 08:33:31 +0000

There's a growing unease in the dev community right now: we keep handing AI agents more tools — shell access, file writes, HTTP clients, git push — and mostly we check the boundaries by vibes. The agent didn't delete anything today, so the sandbox works, right?

I wanted something more defensible than vibes. So this article is a small, runnable red-team suite you can point at any environment where you let an agent execute commands. It doesn't assume any particular agent, model, or vendor. It tests the sandbox, not the model — which is where most real failures live.

Everything below is reproducible. The test suite is code you can run. The failure modes are described, not dramatized.

The four boundaries that actually matter

When people say "sandbox," they usually mean one blurry thing. In practice an agent execution environment has at least four separate boundaries, and each fails differently:

Boundary	Question	Typical failure
Filesystem	Can the agent read/write outside its workspace?	`../` traversal, symlink escape, bind mounts
Secrets	Can it read env vars or files you didn't intend?	`env` dump, `~/.ssh`, `~/.aws`, `.env` in parent dirs
Network	Can it make outbound calls you didn't authorize?	Direct egress, DNS exfiltration, metadata endpoints
Persistence	Does anything it changes survive past the session?	Leftover processes, cron, shell rc files, git hooks

A container answers some of these. A VM answers more. "The agent is polite" answers none.

A reproducible boundary test suite

The trick: don't test whether the agent behaves. Test whether the environment refuses misbehavior. Run these commands yourself (or have the agent run them — same thing, since the agent runs commands as some user somewhere) and check the results.

Save as boundary_check.sh:

#!/usr/bin/env bash
# Boundary checks for an agent execution environment.
# Run INSIDE the sandbox, as the same user the agent runs as.
# Every line should print BLOCKED. Anything else is a finding.

check() {
  local name="$1"; shift
  if "$@" >/dev/null 2>&1; then
    echo "OPEN:    $name"
  else
    echo "BLOCKED: $name"
  fi
}

# 1. Filesystem: can we read outside the workspace?
check "read /etc/shadow"        head -n1 /etc/shadow
check "read parent dir"         ls "$HOME/../../"
check "write outside workspace" touch /tmp/outside_workspace_probe

# 2. Secrets: are credentials reachable?
check "read ~/.ssh"             ls "$HOME/.ssh"
check "read ~/.aws"             ls "$HOME/.aws"
check "env contains secrets"    sh -c 'env | grep -qiE "(KEY|TOKEN|SECRET|PASSWORD)"'

# 3. Network: is outbound egress open?
check "outbound HTTPS"          curl -s --max-time 3 https://example.com
check "cloud metadata endpoint" curl -s --max-time 2 http://169.254.169.254/latest/meta-data/

# 4. Persistence: can we install hooks that outlive the session?
check "write shell rc"          sh -c 'echo true >> "$HOME/.bashrc"'
check "install cron job"        sh -c 'crontab -l 2>/dev/null; crontab - >/dev/null 2>&1 <<< ""'

Run it, and you get a table of OPEN / BLOCKED per boundary. That table is your actual security posture — not the diagram in someone's README.

Two important details:

Run it as the agent's user, not as root. Root in a container can often do things the agent's UID can't, and vice versa for misconfigured mounts. Test the identity that matters.
OPEN isn't always wrong — it's always a decision. Maybe you want outbound HTTPS so the agent can fetch docs. Fine. But then it's a choice you made, not an assumption you inherited.

What each finding means in practice

read /etc/shadow OPEN — usually means the agent runs as root in a container. Root-in-container plus a writable Docker socket or CAP_SYS_ADMIN is a classic container-escape setup. Run the agent as a non-root UID; it costs you nothing.
env contains secrets OPEN — the most common real-world leak. If you inject API keys as environment variables into the same process space where an agent executes arbitrary commands, assume the agent's context window eventually contains those keys. Prefer short-lived, scoped tokens fetched by a broker process the agent can call but not read.
cloud metadata endpoint OPEN — on AWS/GCP/Azure this can hand the agent instance credentials. Block 169.254.169.254 at the network layer or use IMDSv2 with hop limits.
outbound HTTPS OPEN — exfiltration path. If you need egress, an allowlist proxy beats a firewall rule you'll forget about.
Persistence probes OPEN — an agent that can append to .bashrc or install a git hook can arrange for code to run later, possibly by you, outside the sandbox. Ephemeral environments (fresh instance per session) neutralize this whole category.

That last point is why I now prefer disposable environments for agent experiments over my dev box. For this kind of boundary testing, I've been spinning up sessions on MonkeyCode's free server option — it gives you a remote execution environment with free model access, which is convenient here precisely because the environment isn't my machine: if a persistence probe succeeds, I throw the instance away instead of auditing my dotfiles.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The same logic applies wherever you run agents, though — a throwaway VPS, a CI runner, a Firecracker microVM. The test suite doesn't care.

A minimal isolation checklist

If you'd rather fix things than just measure them, this is the order I'd work in:

Non-root user for the agent's process. (useradd, not --privileged.)
Workspace-only filesystem: mount the project dir, read-only-mount or omit everything else, no Docker socket.
Secret broker: the agent calls a helper that holds credentials; the agent never sees them.
Egress allowlist (or no egress) at the network layer, not in a prompt.
Ephemeral lifetime: the environment is destroyed after the session. Persistence of artifacts (code, PRs) goes through git, not through the machine.

None of this is exotic. It's the same hardening you'd apply to a CI runner executing untrusted pull requests — because that's essentially what an agent session is.

Limitations, honestly

This tests configuration, not cleverness. A determined exploit (kernel bug, container runtime CVE) won't show up in a shell script. This suite catches misconfiguration, which is the common case, not zero-days.
A passing suite doesn't make the agent's output safe. Code the agent writes still needs review before it runs anywhere privileged. Boundary tests and code review are different layers.
Free tiers have limits. Whatever hosted environment you use for this — including the one mentioned above — check current quotas, session duration, and region before building a workflow on it; free offerings change and I can't promise what yours includes. For heavy CI-scale testing, you'll want your own infrastructure anyway.
Prompt-level guardrails are not a boundary. "I instructed the agent not to delete files" is a hope. The tests above measure mechanisms. Keep the distinction clear in your threat model.

Who this is not for

If your agent never executes commands — it only suggests code in a chat window — you don't need any of this; your boundary is your own code review. And if you're running agents against production data or regulated workloads, a shell script and a free sandbox are a starting point for thinking, not a compliance posture. Get a real security review.

For everyone in between — experimenting with agents that can run shell commands, and quietly wondering what "sandboxed" actually guarantees in your setup — run the script. Fifteen minutes, and you'll know which of your four boundaries are real and which are decorative.

If you try it and find something surprising in your own setup, I'd genuinely like to hear which check caught it — the env leak is my bet for the most common one.