Your AI Review Bot Double-Posts. A Webhook Replay Test in CI Fixes It.

When an AI review bot misbehaves, the model is usually innocent. The webhook arrived twice. The free server cold-started for 40 seconds, GitHub retried the delivery, and your bot analyzed the same diff twice, commented twice, and burned tokens twice.

This fails more often than a wrong review. And a better prompt will not fix it. What fixes it is plumbing: idempotency keys, replayed webhook fixtures, and a CI job that treats those fixtures as the test suite for your bot. That pattern is what this article builds, step by step. It runs entirely on a free server with free model access, so you can reproduce it without spending anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The failure is the webhook, not the model

Webhook delivery is retry-based. GitHub documents that deliveries are sent again when the endpoint fails (docs). On a free server, cold starts make this worse: the first request wakes the process, the sleep adds latency, the timeout trips, and the retry lands on a fresh process that never marked the first event as handled.

The hot take on DEV this week is that review quality is the hard problem. In practice, delivery reliability fails first. Once you fix delivery, quality discussions actually mean something.

The three failure classes

1. Duplicate comments — the retry produces a second review and a second comment on the PR.
2. Duplicate spend — every retry burns tokens against the same diff. On a free allowance, this is how a quiet week becomes a quota surprise.
3. Missed events — if the bot crashes before doing any work, the event is silently lost.

All three are delivery bugs. None of them is fixed by a new system prompt.

Design rule: mark before the slow part

The core rule is mark before, not after. Deduplicate the event before you call the model, because the model call is the slow, expensive, and killable part. This is the smallest FastAPI webhook that follows the rule:

import hashlib
from fastapi import FastAPI, Request

app = FastAPI()

def event_key(delivery_id: str, repo: str, pr: int, action: str) -> str:
    return hashlib.sha256(f"{delivery_id}:{repo}:{pr}:{action}".encode()).hexdigest()

seen = {}

@app.post("/webhook")
async def webhook(request: Request):
    h = request.headers
    delivery_id = h.get("x-github-delivery") or h.get("x-gitlab-event-uuid")
    body = await request.json()

    if not delivery_id:
        return {"status": "ignored"}  # manual or fixture calls only

    key = event_key(delivery_id, body["repository"]["full_name"],
                    body["pull_request"]["number"], body["action"])

    if key in seen:
        return {"status": "duplicate", "key": key}

    seen[key] = True  # before the model call, not after

    review = await run_review(body)   # free model tier, via MonkeyCode
    await post_comment(body, review)
    return {"status": "reviewed", "key": key}

The ordering is the whole trick. If the process dies after the model call but before the comment, the retry finds key in seen and skips the expensive part. You still need a sweep to post the missing comment, but you no longer double-analyze. That one line halves the cold-start blast radius.

I ran this against recorded webhooks using the free server option from MonkeyCode, with model access on its free tier. Same code, same fixtures — only the deployment target was a no-cost server.

Build the fixture corpus

You cannot test delivery with payloads you invented by hand. You capture real ones. Over a normal week of PR activity, I recorded four classes of event:

fixtures/
  pr.opened.base.json          # the first delivery
  pr.opened.retry.json         # same payload, new delivery id
  pr.synchronize.second-commit.json
  push.no-pr.json              # should be ignored, cheap to assert

The setup is three steps:

1. Add a minimal logger to your webhook that stores every raw payload plus the delivery header.
2. Sanitize what you store: usernames, repo names, and secrets get replaced so the fixtures are safe to commit.
3. Save one retry pair intentionally. When the provider retries, you get two deliveries of the same body with different ids — that pair is your most valuable fixture.

The CI replay job

Now the fixtures become a test suite. Every change to the bot — a new dependency, a refactor, even a prompt change — must survive a replay of the week's events. A pytest test for the retry pair looks like this:

def test_retry_does_not_duplicate(client, fixtures):
    first = client.post("/webhook",
                        headers=fixtures.headers("pr.opened.base"),
                        json=fixtures.body("pr.opened.base"))
    second = client.post("/webhook",
                         headers=fixtures.headers("pr.opened.retry"),
                         json=fixtures.body("pr.opened.base"))
    assert first.status_code == 200
    assert first.json()["status"] == "reviewed"
    assert second.json()["status"] == "duplicate"

Add a second test that replays pr.opened.base against a fresh in-memory store and asserts a comment was posted. Add a third that replays push.no-pr.json and asserts no comment and no model call. Together they pin the behavior production actually depends on.

Then wire the job into your workflow:

name: bot-replay
on:
  pull_request:
    paths: ["webhook/**", "fixtures/**", "tests/**"]

jobs:
  replay:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest tests/ -q

That is the green-to-merge path. Branch protection requires bot-replay to pass, exactly like any other unit test. The bot is no longer invisible machinery; it is code with a regression suite.

Where flake still lives — and how to budget it

You cannot make the model deterministic, and you should not pretend otherwise. The point of the replay job is to separate your flakes from the model's flakes. The decision table I use:

Observed failure	Where it lives	Budget	Action
Duplicate comment	your dedupe	zero	block the merge
Missing comment after retry	crash before mark	zero	block the merge
Same diff, different wording	model nondeterminism	tolerated	inspect, don't block
Provider 503 on model call	model service	a few per day	retry with backoff

That table is the flake control. Zero-budget failures fail CI. Tolerated ones produce a warning artifact instead. This is the honest way to run LLM output in a test suite: assert the plumbing exactly, assert the prose loosely.

Who should not use this pattern

Be direct about the limits. A free server is a shared, reclaimable resource: cold starts happen, the process can be stopped, and your retry window is not guaranteed. A free model tier can change its quotas without notice. So this pattern is for a project bot, a personal workflow, or a low-stakes automation — not for a compliance checkpoint, a billing system, or anything where a missed review has regulatory teeth. If your org needs a strict SLA, rent the paid version of the same idea.

The takeaway

Review bots fail in delivery more often than in judgment. Fix the delivery with idempotency, capture real webhooks as fixtures, and replay them in CI on every change. That turns a flaky black box into a testable service with a green-to-merge path.

If you want to run this exact pattern end-to-end: MonkeyCode's free tier, as of late August 2026, includes a 10M token model allowance and a free server option — enough for a week of replays and several PR reviews. Free tiers change, so check the current terms before you depend on them. I do.