The Review Bot Said "No Issues" for 72 Hours Straight. The Root Cause Wasn't the Model.

A review bot that silently reports "no issues" is more dangerous than one that crashes, because a crash demands attention while an empty verdict builds false confidence. This retrospective follows a 72-hour incident in which an AI-assisted review pipeline returned "No issues detected" for every pull request. The root cause turned out to be a timeout configuration rather than a weak model, and the fix was surprisingly small. Three reusable techniques emerged from the post-mortem: treat empty model output as an error, embed a canary bug in every run, and log raw responses instead of parsed summaries.

The Setup

The team hosted a review bot on a free server and used MonkeyCode's free model access to analyze every pull request diff. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that provides free model access and a free server option, which kept the bot's cloud bill at zero, and the current free tier includes 10 million tokens. The bot fetched the diff, sent it to the model with a structured review prompt, and posted the parsed response back to the pull request. For two weeks it caught real problems like unhandled promise rejections and unsafe string interpolation in SQL queries.

The Symptom

On the first day of the incident, the bot posted "No issues detected" on a pull request that contained a clear child_process.exec call with unsanitized user input. The comment appeared in 1.8 seconds instead of the usual 25 to 40 seconds, which was the first clue that something had changed. Over the next three days, every pull request received the same verdict, and the team initially suspected that the model had degraded or that the prompt had drifted.

The Investigation

Step 1: Reproduce outside production

The first move was to run the exact same diff through the model locally with the identical prompt and temperature settings. The model flagged the injection risk immediately, which ruled out model quality as the primary suspect. This step matters because production environments accumulate configuration drift that has nothing to do with the model itself.

Step 2: Read the raw logs

The server logs contained no errors, but every request duration was exactly 1500 milliseconds. Round numbers like that are almost always a timeout or a rate-limit signature rather than a natural completion time. The team had been reading parsed summaries in the database and never looked at the raw response bodies, which hid the pattern for three days.

Step 3: Trace the response path

The client code parsed the model response with a default fallback that silently swallowed failures:

def parse_review_response(raw_text: str) -> ReviewResult:
    if not raw_text:
        return ReviewResult(issues=[])  # empty response becomes "no issues"
    ...

When the server timed out, it returned an empty string, and the parser converted that empty string into an empty issue list. The "No issues detected" comment was therefore not a model verdict at all; it was the empty-result branch of the code executing as designed.

Step 4: Find the timeout

A deployment two days earlier had introduced REQUEST_TIMEOUT_MS=1500 into the server environment. The model typically needed 20 to 40 seconds to review a full diff, so every request hit the timeout, and the client silently translated the failure into success. The configuration change was unrelated to the review feature and slipped in with an infrastructure update.

The Fix

Three changes went into the pipeline, and all three were small enough to ship within an hour.

1. Fail loudly on empty responses. The parser now raises a ReviewTimeoutError when the raw response is empty or malformed, and the bot posts "Review failed, check the logs" instead of a clean bill of health.
2. Add a canary snippet. Every run appends a known-buggy code sample to the prompt, and the pipeline treats the whole run as invalid if the model does not flag it.

CANARY = '''
# This snippet has a real bug: user input is interpolated into SQL.
def find_user(db, name):
    return db.execute(f"SELECT * FROM users WHERE name = '{name}'")
'''

1. Make the timeout configurable and generous. The new default is 90 seconds, and the bot writes the raw response body to a log file for every request so future incidents are visible in the first minute.

The Reusable Debugging Checklist

The specific failure was a timeout, but the techniques generalize to any pipeline that consumes model output.

• Treat empty output as an anomaly, not as a successful verdict.
• Look for suspiciously round durations in request logs, because timeouts leave fingerprints.
• Reproduce with the exact payload outside production before blaming the model.
• Log raw responses, not just parsed summaries, so silent degradation becomes visible.
• Add an invariant check that must fail when the pipeline quietly stops doing its job.

Limitations and Who Should Skip This

This retrospective describes one team's configuration, and the exact timeout values and environment variables are illustrative rather than universal. Free tiers and server options change over time, so teams should verify current quotas from the project repository before relying on them. The approach also assumes that a best-effort review bot is acceptable; teams with strict latency or accuracy SLAs should not route critical reviews through a free-tier pipeline without a human fallback.

Conclusion

The model was never the problem, and the real lesson is that silent degradation is the most dangerous failure mode in AI tooling. A canary check and a loud error path turned a blind reviewer into a trustworthy one, and the same pattern applies to any automated gate that consumes model output. If you want to reproduce this setup, the MonkeyCode repository contains the server and client code used here, and the free tier is enough to run the experiment on a real project.