The pager went off at 2:07 AM on a Friday, which is never a good sign, and the alert said the checkout service was returning 500s for nearly every request. The deploy had been routine, a small dependency bump that the CI pipeline blessed with a green checkmark and a cheerful comment from our AI review bot. By the time I opened the dashboard, the error rate had climbed past twelve percent, and the rollback button was starting to look very attractive.
The timeline matters more than the blame, so here is the sequence that actually happened. At 14:02 a contributor opened a pull request that touched the payment retry logic, and at 14:07 the review bot commented “LGTM, no issues found” after analyzing the diff for about forty seconds. The merge landed at 14:11, the deploy pipeline ran at 14:40, and the first failed health check arrived at 15:03, which meant the incident was already forty minutes old before anyone noticed. The rollback completed at 15:31, and the total user-facing impact was roughly ninety minutes of intermittent checkout failures.
Three contributing factors came together to create this incident, and none of them was a single bad line of code. The first factor was that the review bot failed open: when the hosted model API returned a rate-limit error, the code caught the exception and returned a passing review instead of blocking the merge, because the original author assumed a quiet bot was better than a stuck pipeline. The second factor was that the quota was shared across the whole workspace, so a spike of activity from other consumers exhausted the budget right when our pull request needed it most. The third factor was that nothing in the review path had ever been tested against a local model, which made the failure mode invisible until production started complaining.
The durable fix was not to remove the bot, because that bot had caught real bugs for months, but to give it a fallback target that we actually controlled. We changed the review job so it first tries the hosted API, and when that call fails it retries against a local server running on the same machine as the CI runner, which is a scenario MonkeyCode's free server option covers. MonkeyCode is an open-source project, and its current offering includes free model access plus a free server you can run yourself, so the fallback costs nothing beyond the hardware you already own. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The configuration change is small when your client speaks the OpenAI-compatible protocol, because you only swap the base URL and the API key, and the same pattern works for any local server you already trust.
# review_bot.py
import os
from openai import OpenAI
hosted = OpenAI(api_key=os.getenv("HOSTED_API_KEY"))
local = OpenAI(
base_url=os.getenv("LOCAL_BASE_URL", "http://127.0.0.1:8080/v1"),
api_key=os.getenv("LOCAL_API_KEY", "local"),
)
def review(diff: str, clients=None) -> str:
for client in (clients or [hosted, local]):
try:
response = client.chat.completions.create(
model=os.getenv("REVIEW_MODEL", "local-model"),
messages=[
{'role': 'system', 'content': 'Review this diff. Reply APPROVE or BLOCK with one reason.'},
{'role': 'user', 'content': diff},
],
timeout=30,
)
return response.choices[0].message.content
except Exception:
continue
return "BLOCK"
The critical part is the order of operations, because the fallback only helps if it is fast and predictable. The hosted call gets a thirty-second timeout, the local call gets the same, and if both fail the function returns a hard BLOCK instead of a quiet pass, which is the behavior that would have prevented this incident entirely. You can verify the local endpoint is alive before the review job even starts with a single curl command, and you can test the failure path by passing an empty client list and asserting that the result is BLOCK.
# check the local server before the review job starts
curl -s http://127.0.0.1:8080/v1/models | head
# tests/test_review_bot.py
from review_bot import review
def test_review_blocks_when_every_endpoint_is_down():
assert review("some diff", clients=[]) == "BLOCK"
Why does this matter for your own pipeline, beyond the obvious lesson about fail-closed gates? The economics of a fallback are usually the real blocker, because paying for a second hosted API just to cover rare outages is a hard sell in most engineering budgets. A local server that runs on hardware you already own changes that calculation, and MonkeyCode's current free tier includes a ten million token allowance that was accurate as of this writing, which is enough to run a meaningful review workload without opening a purchase order. The general principle stands on its own though: any AI-assisted gate in your CI should have a local fallback and a defined behavior when every option is down.
There are honest limitations to this approach, and you should hear them before you copy the pattern. Teams that need the strongest available model for nuanced review will notice the quality gap on a local fallback, so treat it as a safety net rather than an upgrade path. Teams that review very large diffs may hit context window limits on local models, and teams with strict compliance requirements should still verify where the code goes before wiring anything to a third party. The fallback pattern also assumes your CI runner can actually host the server, which is not true for every managed pipeline, so check that constraint before you commit to the design.
The next time your pager goes off at 2 AM, the question is not whether your AI reviewer will fail, because every external dependency fails eventually, but whether your pipeline knows what to do when it does. Ours now blocks the merge and tells the author exactly why, and that single change has been worth more than any model upgrade we have tried since. If you want to test the same fallback pattern in your own repository, MonkeyCode's free server and free model access are a reasonable place to start, and the rest of this article remains useful even if you never touch the product.
Top comments (0)