Case Study: A Checklist Bot Built to Keep Working When the Model Is Down

The cheapest model call is the one your system can survive without. That sentence became the design rule for a small internal tool I finished over two weekends: a bot that turns a pull-request diff into a short reviewer checklist. The interesting part is not the prompt; it is the rule that the tool must return something useful at every level of degradation, including the level where no model answers at all. Below is the whole project end to end, including the budgeter, the fallback ladder, the fixture test plan, and the parts I would not do again.

The constraint that shaped everything

The project ran against two operator-supplied facts about MonkeyCode: free model access, and a free server option for hosting small services. The operator describes the free allowance as 10 million tokens; treat that number as a starting point you re-check on the product's own pricing page, because allowances and terms move. The tool I built assumes only that the model is cheap, rate-limited at some unknown point, and occasionally slow or absent — three assumptions that survive almost any change to the free tier.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

There was also a mood constraint. The last week of DEV discussion has been full of people admitting they are tired of AI woven into every hot path, and the "design for churn" argument keeps making the same point from a different angle: a dependency you cannot remove is a liability. So the goal was an AI tool that a reviewer could ignore, disable, or outlive without losing the PR workflow.

Goal and scope

The target repository is a mid-sized Python service with roughly 40 endpoints. A reviewer opening a diff of 300 changed lines routinely misses the boring questions, so the bot posts a comment with a small checklist. I wrote down four requirements before any code:

1. One model call per diff, maximum, with a hard timeout and no retries.
2. A deterministic checklist from path and size rules whenever the model is unavailable or the diff is too large to fit a budget.
3. No partial context: if the diff does not fit whole, the bot does not call the model at all.
4. Every degradation level is observable in the comment footer, so trust in the tool can be audited.

That third rule is the one people argue about, and it is the one I would defend hardest. Sending the model half a diff produces confident nonsense about code it never saw, which is worse than a boring path-based checklist.

The degradation ladder

Numbered levels make the fallback path testable instead of aspirational:

• Level 0 — empty: no file changed, return a one-line note, no model call.
• Level 1 — heuristic: path and size rules only. Used when the diff overflows the character budget.
• Level 2 — heuristic after model fault: timeout, network error, or a response that fails the shape check.
• Level 3 — model: a single call, a single timeout, a response that parses as a checklist.

You can see the design bias immediately: three of the four levels never touch the network. The model is an accelerator bolted onto a boring, working baseline, which is exactly the posture that makes AI fatigue tolerable.

Implementation

Step 1: Rank hunks before you count anything

Token budgets are the wrong first question for a diff. Ranking is cheaper and more honest: a migration ranks above an auth change, which ranks above an endpoint change, while lockfiles and docs sink to the bottom.

# budget.py
from dataclasses import dataclass

@dataclass(frozen=True)
class Hunk:
    path: str
    size: int
    text: str

RANK = {"migration": 0, "auth": 1, "api": 2, "test": 8, "docs": 9, "lock": 10}

def kind_of(path: str) -> str:
    lowered = path.lower()
    return next((k for k in RANK if k in lowered), "src")

def rank(hunks: list[Hunk]) -> list[Hunk]:
    return sorted(hunks, key=lambda h: (RANK.get(kind_of(h.path), 5), -h.size))

def pack(hunks: list[Hunk], char_budget: int) -> tuple[str, int]:
    ordered = rank(hunks)
    total, kept, dropped = 0, [], 0
    for h in ordered:
        cost = len(h.text)
        if total + cost > char_budget:
            dropped += 1
            continue
        kept.append(h)
        total += cost
    return "\n".join(h.text for h in kept), dropped

Note that pack reports how much it dropped. That count is the whole trigger for rule three: dropped > 0 means no model call, ever.

Step 2: One call, one timeout, no retry loop

The handler is deliberately dumb. It has no queue, no backoff, and no second attempt, because a reviewer is waiting and a stale checklist is noise.

# handler.py
import os
from budget import Hunk, pack
from rules import heuristic_checklist, looks_like_checklist

CALL_TIMEOUT_S = float(os.getenv("CALL_TIMEOUT_S", "8"))
CHAR_BUDGET = int(os.getenv("CHAR_BUDGET", "24000"))
MODEL_ENABLED = os.getenv("MODEL_ENABLED", "0") == "1"

def checklist_for(hunks: list[Hunk], call_model) -> tuple[str, str]:
    if not hunks:
        return "L0-empty", "No observable change; nothing to review."
    packed, dropped = pack(hunks, CHAR_BUDGET)
    baseline = heuristic_checklist(packed)
    if not MODEL_ENABLED or dropped > 0:
        return "L1-heuristic", baseline
    try:
        out = call_model(packed, timeout=CALL_TIMEOUT_S)
    except TimeoutError:
        return "L2-timeout", baseline
    if not looks_like_checklist(out):
        return "L2-invalid", baseline
    return "L3-model", out

The return value carries the level string straight into the PR comment footer, so anyone reading the comment knows which path produced it without opening a dashboard.

Step 3: Deploy on the free server

Keep the model behind an environment variable so you can disable it with a config change rather than a redeploy. A minimal process launch looks like this:

python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
python -m pytest -q tests/            # run the fixtures first
MODEL_ENABLED=1 CHAR_BUDGET=24000 CALL_TIMEOUT_S=8 python -m bot.server --port 8080

The same command runs on the free server option and on my laptop, which matters more than it sounds: the ability to reproduce a production comment locally turned two debugging sessions into two minutes.

The test plan that actually caught bugs

Five fixtures, each a saved diff under tests/fixtures/, each asserting a level and a shape:

Fixture	Expected level	Assertion
empty.diff	L0	comment has no checklist block
lockfile_only.diff	L3 or L1	no endpoint paths mentioned
oversized.diff	L1	footer says L1-heuristic
model_timeout	L2	baseline checklist still present
model_garbage	L2	response rejected by shape check

The oversized and garbage fixtures found real problems. The first revealed that my packer silently truncated a single enormous hunk, and the second showed that my shape check accepted prose bullets with no action items. Both are now assertions, not vibes.

Results

Qualitative results were better than the quantitative ones, which is the honest summary. Reviewers used the L1 path-based checklist more than the model output, because it named the migration and the auth module reliably and never hallucinated an endpoint. The model path added value mainly on medium diffs that touched several modules at once.

I am not quoting an accept-rate or a latency percentile here. Any numbers from a two-weekend project on an unknown network would be decoration, and the fixture suite lets you produce your own numbers in under an hour on your repository. If you want one metric, count how often the footer says L2-invalid — that ratio tells you more about your prompt than any satisfaction survey.

Limitations and who should skip this

• If your diffs are routinely larger than the character budget, most comments will be L1 and the model integration earns nothing.
• If your reviewers expect prose explanations of intent, path-based rules will read as shallow, and that is a fair complaint.
• If your repository has no stable directory conventions, ranking by path is close to random.
• If you need guaranteed availability and a support contract, a free tier is the wrong foundation regardless of vendor.
• Skip this entirely if a human already writes a consistent checklist; you would be automating a solved problem.

Lessons learned

1. Write the fallback before the prompt. The fallback defines what "useful" means, and the prompt is judged against it.
2. Never send a partial diff. Silent truncation is a hallucination factory with a clean audit trail.
3. Put the degradation level in user-visible output. It converts quiet quality drift into something a reviewer can argue with.
4. Budget in characters, not tokens, at the boundary. You cannot count tokens for a model you have not called yet.

If you want to try the same shape, start with the fixture suite and the ranking function on a free server, keep the model call behind a flag, and only then spend any of the free token allowance on prompts. MonkeyCode's free model access and free server option are what made a two-weekend proof viable for me, and the same arrangement is a reasonable place to test whether the idea survives your own repository.