A Noise Budget for Automated PR Reviews: Free Models, a Free Server, and a Protocol That Keeps Humans Listening

An AI reviewer that comments on everything is worse than one that stays silent. The silent bot costs a token budget and nothing else; the chatty one trains a whole team to ignore every flag it raises, including the genuine blockers. That failure mode is why automated review gates tend to die within two weeks, and it is a people problem a team can fix with a budget instead of a better model.

This article describes a noise budget protocol for AI-assisted pull request review. The pipeline runs on free models served from a free server, it posts only high-severity findings, and it makes every bot comment accountable to a reviewable file. The reproducible artifacts are a severity-gated runner, a GitHub Action, and a calibration plan for finding the threshold your team can tolerate.

The failure mode nobody prepares for

Teams adopt an AI reviewer, watch it produce twenty comments on the first PR, and feel productive. Then the second PR produces the same twenty comments, three of them wrong, and the reviewer's credibility never recovers. The bot did not fail because its model was weak; it failed because it had no cost model for human attention.

Human reviewers have a natural noise budget: they spend their credibility on the two findings that matter and skip the formatting nit. An automated reviewer needs the same discipline encoded in software. Without it, the system optimizes for coverage and destroys trust.

The fix is a severity gate with an explicit comment cap, plus a log that tracks how often the bot is wrong. This is a maintenance workflow, not a one-time script, and the rest of this article shows how to run it close to zero cost.

Environment: a free server, free models, and one rule

The reference setup uses MonkeyCode, an open-source project that offers two things a small team usually cannot afford: access to free models and a hosted free server that runs the review loop without a dedicated machine. The project's docs, as of August 31, 2026, describe a starting grant of 10 million tokens on that free server, and any serious deployment should re-check that figure before relying on it. Free tiers move, so the number is a setup step, not a contract. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One rule governs the whole pipeline: the bot may only comment on findings it classifies as blockers, and it may never exceed the configured cap per PR. Everything below that severity goes into the PR description as a quiet summary, visible to humans who opt in and invisible to everyone else.

Artifact: a severity-gated runner

The runner consumes the model's raw review as JSON and reduces it to a short, accountable list. The example below is pseudocode-quality but shaped like a real implementation, and it assumes arguments rather than inventing them.

# runner.py - consume raw review JSON, emit only blocker comments
import json, os, sys

SEVERITY_HINTS = [
    ("blocker", ["race", "deadlock", "null", "injection", "credential", "data loss"]),
    ("warning", ["timeout", "error handling", "memory", "retry"]),
    ("nit", ["format", "naming", "style", "comment"]),
]

def classify(text: str) -> str:
    lowered = text.lower()
    for severity, hints in SEVERITY_HINTS:
        if any(h in lowered for h in hints):
            return severity
    return "nit"

def main() -> int:
    review = json.load(sys.stdin)
    cap = int(os.environ.get("NOISE_CAP", "3"))
    blockers = [c for c in review["comments"] if classify(c["text"]) == "blocker"]
    for comment in blockers[:cap]:
        print(f"- [ ] {comment['file']}:{comment['line']} - {comment['text']}")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

The classifier is intentionally crude so a team can read it in one sitting. A keyword miss means the bot stays quiet, which is the safe failure; a keyword hit on a false positive still lands in the log, which is how the team learns where the budget leaks.

The GitHub Action that wraps the runner is short on purpose:

name: noisy-bot-gate
on: pull_request

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: git fetch --unshallow origin ${{ github.event.pull_request.base.ref }}
      - name: run gated review
        env:
          REVIEW_ENDPOINT: ${{ secrets.REVIEW_ENDPOINT }}
          REVIEW_TOKEN: ${{ secrets.REVIEW_TOKEN }}
          NOISE_CAP: "3"
        run: |
          bash review.sh origin/${{ github.event.pull_request.base.ref }} | python runner.py

Secrets live in GitHub secret storage, never in the repo, and every output line is prefixed with the bot's identity so humans always know the source.

The decision table that keeps the bot honest

The gate expresses its policy as a table, not as vibes. Each severity class has a posting rule and a documentation rule, and the combination is what makes review comments auditable after the fact.

Severity	Example signal	Posted to PR?	Recorded in summary?	Human response
Blocker	race condition, null dereference, credential leak	Yes, up to cap	Yes	Must triage before merge
Warning	missing timeout, weak retry logic	No	Yes	Optional follow-up issue
Nit	naming, formatting, style	No	No	Ignored silently

A blocker that slips past the classifier becomes a calibration case. A nit that gets promoted to blocker tells the team the prompt drifted. The table gives every disagreement a deterministic verdict, which is the difference between tuning a process and arguing with a chatbot.

Calibration: find your cap in ten PRs

The cap of three comes from nowhere until a team measures it. The calibration protocol is a ten-PR experiment across typical, unremarkable diffs. For each PR, a human records three numbers: bot blockers posted, bot blockers that were real, and real bugs the bot missed.

PR	Posted	Real	Missed	Verdict
1	2	2	0	Keep cap
2	3	1	1	Cap too high
3	1	0	2	Classifier too strict
...	...	...	...	...

A healthy gate shows a rising real-to-posted ratio over ten PRs. A cap that produces fewer than one real finding per PR is a waste of tokens, and a cap that misses two real bugs in a row needs a broader keyword set. The budget log is the evidence file that makes the next tuning decision boring and fast.

Limits: who should not run this gate

This protocol assumes a tolerant team, a public or non-sensitive codebase, and a willingness to tune. None of those are universal, and the honest list matters.

• Teams reviewing proprietary code should not send whole diffs to a hosted free server; the self-hosted open-source project is the correct variant instead.
• Teams under a contractual SLA should not build a merge gate on free infrastructure, because free tiers can change quotas or endpoints without notice.
• Teams that want a security expert on every PR will not get one from a free model tier; the gate finds obvious blockers, not nuanced architecture judgment.

If the model has never been observed failing on the team's actual code, the cap and the classifier are guesses. That is why the calibration log is part of the protocol and not an optional extra.

Before you go

Install the gate on a low-stakes repository, set the cap to one, and let the bot earn a second comment only after its first flag turns out to be real. When the team stops flinching at every notification, raise the cap one notch. The article's artifacts are small enough to read in an afternoon, and the decision table is the part worth keeping even if the code gets rewritten. The budget number in MonkeyCode's docs is worth verifying before the first run, because the first lesson of free infrastructure is the same as the first lesson of noisy bots: check the source before you trust the signal.