Opinion: An AI Review You Can't Replay Is a Review You Can't Trust

#ai #testing #cicd #productivity

Most teams treat AI review output like a chat message: read it, act on it, and forget it. The correct mental model is test output, which means it must be re-runnable, comparable, and archived. A review you cannot replay is a review you cannot audit, and an unauditable review is worse than none because it manufactures confidence. My position is that replayability is the missing quality gate in the AI review loop, and that free model access finally makes it affordable.

The three failures that replay exposes

Review pipelines fail in three recurring ways, and all three stay invisible until you try to re-run the review.

The unexplainable false positive. A model flags a refactor as risky, a human spends an hour investigating, and the finding turns out to be noise. When the review is ephemeral, you cannot reconstruct what the model saw, so you cannot learn whether the prompt or the diff caused the noise.
Silent config drift. Someone edits the system prompt or bumps the model, and the review behavior shifts underneath the team. Nothing fails, no test breaks, and the merge discipline changes without a single deliberate decision.
The provider swap. A hosted model is replaced upstream, verdicts flip on identical diffs, and your gate suddenly blocks more or less than it did last week. Without a replay, the change looks like a change in code quality when it is actually a change in the reviewer.

The recent DEV conversation about AI reviewers never being tested points at the same gap from a different direction. We test the code the reviewer produces, but we never test the reviewer itself, and replay is the cheapest test that exists.

Why tests get re-run and reviews do not

A unit test is valuable because you can run it again, and a failing re-run tells you the code changed. AI review output is stochastic, so the contract is weaker: identical inputs can produce different outputs, which means you must snapshot more, not less. The discipline is to archive every input that shaped the review — the diff, the prompt, the model identifier, and a hash of the review configuration — alongside the output. That archive turns a one-off opinion into a reproducible record, and it turns drift into a detectable event.

A replay harness you can run today

The artifact below is a minimal replay harness that assumes your review server exposes an HTTP endpoint and that you archive each review as a JSON record. It re-runs an archived review, compares the verdict, and then diffs the full output. Adapt the payload to your own API, since the exact field names will vary.

#!/usr/bin/env bash
set -euo pipefail

RECORD="${1:-reviews/latest.json}"
BASE_URL="${REVIEW_SERVER_URL:-http://127.0.0.1:8787}"

DIFF_FILE="$(jq -r .diff_file "$RECORD")"
PROMPT_FILE="$(jq -r .prompt_file "$RECORD")"
MODEL="$(jq -r .model "$RECORD")"
CONFIG_HASH="$(jq -r .config_hash "$RECORD")"
ARCHIVED_OUTPUT="$(jq -r .output_file "$RECORD")"

curl -sS -X POST "$BASE_URL/review" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --rawfile diff "$DIFF_FILE" --rawfile prompt "$PROMPT_FILE" \
        --arg model "$MODEL" --arg config_hash "$CONFIG_HASH" \
        '{diff: $diff, prompt: $prompt, model: $model, config_hash: $config_hash}')" \
  > /tmp/replay_output.json

NEW_VERDICT="$(jq -r .verdict /tmp/replay_output.json)"
OLD_VERDICT="$(jq -r .verdict "$ARCHIVED_OUTPUT")"

if [[ "$NEW_VERDICT" != "$OLD_VERDICT" ]]; then
  echo "REPLAY MISMATCH: verdict changed from $OLD_VERDICT to $NEW_VERDICT"
  exit 1
fi

if ! diff <(jq -S . "$ARCHIVED_OUTPUT") <(jq -S . /tmp/replay_output.json) > /tmp/replay.diff; then
  echo "REPLAY DRIFT: output changed for identical inputs"
  cat /tmp/replay.diff
  exit 1
fi

echo "REPLAY OK: output matches the archived review"

The workflow has four steps, and each one maps to a normal engineering habit.

Archive the record. After every review, write a JSON record that references the diff file, the prompt file, the model identifier, and a hash of the review configuration. The record is the unit of audit.
Replay before merge. Run the harness on the review for the current pull request, and treat a verdict change as a blocker. A reviewer that contradicts itself on identical input needs a human decision, not a silent coin flip.
Replay on config change. When you edit the prompt or switch models, replay the last ten archived reviews and measure how many verdicts flip. That number is the blast radius of your change, and it belongs in the pull request description.
Replay on a schedule. A weekly cron job that replays a sample of archived reviews turns drift into a trend. The job does not need to block anything; it exists to make invisible changes visible.

Why free execution changes the incentive

Replay only becomes a habit when the marginal cost of a re-run is near zero. Metered review APIs punish exactly the behavior this workflow requires, because running the same input twice means paying for the same work twice. Free model access and a free server option invert that incentive, since the marginal cost of a replay stops being the dominant term in the decision. MonkeyCode offers both, which makes the replay harness a default rather than a luxury. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The economics matter because discipline follows cost. When a re-run is free, the archive stops being a compliance artifact and starts being a debugging tool, and teams actually use debugging tools.

Limitations and who should skip this

This workflow is not a determinism guarantee, and I want to be explicit about that. Temperature and sampling mean identical inputs can still produce different outputs, so the harness detects drift rather than eliminating it, and a passing replay is evidence of stability, not proof of correctness. The approach also assumes your review server can be pointed at archived inputs, which is a property of the server implementation rather than the model. Teams with a handful of weekly reviews will find the archive overhead larger than the benefit, and teams whose compliance rules require a hosted vendor in the loop cannot use a local replay at all.

The honest conclusion is that replayability is a trust mechanism, not a quality mechanism. It will not make your AI reviewer smarter, but it will tell you when the reviewer changes, and that knowledge is the precondition for every other improvement you might attempt. If your pipeline already archives diffs, replay the last ten reviews this week; the mismatches will teach you more about your reviewer than the reviews ever did.

DEV Community