When Your AI Reviewer Remembers Too Much: A Two-Phase Memory Probe

Most AI code-reviewer evaluations treat the candidate as an amnesiac: feed it one pull request, read one verdict, and move on. Persistent-memory reviewers break that model because they keep history across PRs, and that history becomes a second source of bugs. The dominant failure is no longer amnesia but overconfidence in stale context. A two-phase probe exposes whether a candidate trusts its own memory more than the repository's current decisions.

This article supplies the complete take-home package: a fixture repository, a reusable candidate prompt, an HTTP-flavored scoring rubric, a reference solution, and a zero-cost runner script. The probe uses two synthetic PRs and measures one skill: which convention source wins inside the reviewer's context window. That focus separates it from single-shot snapshot tests, which cannot observe memory effects at all.

Why Memory Changed the Review Game

Review agents increasingly index merged PRs, cache decision logs, and carry state between sessions; memory is now a product feature rather than an accident. A bot that recalled yesterday's debate can produce faster and better reviews than a cold-start model. The same memory can poison verdicts when it retrieves an obsolete decision or anchors on the first PR it ever saw.

Hiring decisions usually rest on a one-off trial that optimizes for prompt compliance, not for long-run behavior. A bot can ace a snapshot test and then fail its third week by citing a convention that the repository replaced. The probe below converts that risk into a scored, reproducible exercise.

The Fixture Repository

fixture/
├── docs/decisions/0001-metrics-pipeline.md    # accepted 2026-07-02
├── docs/decisions/0012-rename-to-telemetry.md # accepted 2026-08-14
├── src/metrics_service.py                     # legacy module, 120 lines
├── src/telemetry_service.py                   # replacement module, 140 lines
└── pyproject.toml                             # lint: E501 disabled for telemetry only

The fixture encodes a deliberate conflict: the team renamed the metrics pipeline to telemetry in decision 0012, while the legacy module still exists on the main branch. A memoryless reviewer sees only current code and never learns about the rename. A memory-bound reviewer should retrieve decision 0012 and apply it to both phases.

Phase 1 — Familiarization PR

The first PR adds retry logic to telemetry_service.py and touches pyproject.toml; it is intentionally boring. Its real job is to let the candidate observe the repository history, read both decision files, and form a picture of its conventions. Nothing in this phase is graded.

Phase 2 — Probe PR

The second PR deletes metrics_service.py, promotes telemetry_service.py to the canonical module, and adds a seeded bug: transmit sends an empty payload without a guard and raises ValueError at runtime. A correct review must block on the missing guard while accepting the rename and citing decision 0012.

Candidate Prompt

You are reviewing PR #14 against the fixture repository.

Convention source priority, highest first:
1. docs/decisions/*.md and DEPRECATED.md
2. recently merged PR descriptions and issue threads
3. the historical code being replaced

Return three sections:
- blocking: correctness issues with file:line references
- consistency: conflicts with current decisions
- uncertain: claims you could not verify

Do not flag a difference from deleted code as a regression unless the
deleted behavior is still enforced by a live decision file.
Cite the exact path for every convention claim.

Reference Solution

• Blocking: transmit(payload) must guard against None or empty payload before calling client.send; an early raise ValueError is the expected fix.
• Consistency: the rename is correct per ADR-0012, and deleting the legacy module is expected rather than churn.
• Uncertain: none required; a strong review may ask whether downstream callers were migrated before the old module disappears.

Scoring Rubric

Verdict	Score band	Review signature
200 OK	90-100	Seeded bug found; rename accepted; at least one decision citation; no stale-convention complaints
301 Moved Permanently	60-89	Change recognized, but the rename is flagged as unnecessary churn; the bug may be found or missed
409 Conflict	30-59	Review asserts the old namespace is canonical and contradicts ADR-0012
404 Not Found	0-29	Seeded bug missed; summary contains no file-level claims

The HTTP mapping makes each verdict easy to communicate to a hiring panel and hints at its operational meaning. A 404 bot will miss regressions in production; a 409 bot will block valid migrations until its cache is reset. Scores of 90 or above indicate the candidate can reconcile memory with current ground truth.

Zero-Cost Runner

#!/usr/bin/env bash
# run_memory_probe.sh — two-phase AI reviewer probe (abridged reference harness)
set -euo pipefail
REVIEWER_CMD=${1:?pass the reviewer CLI, e.g. "monkeycode review --json"}
FIXTURE=${2:?pass the fixture repo path}
WORKDIR=${WORKDIR:-/tmp/memory-probe-$(date +%s)}

git clone --quiet "$FIXTURE" "$WORKDIR/repo"
cd "$WORKDIR/repo"

# Phase 1: boring retry refactor; lets the bot read repository history
git checkout -q -b phase1-retry
# ... apply the retry commit from the task kit ...
"$REVIEWER_CMD" --base main --head phase1-retry > "$WORKDIR/phase1.json"

# Phase 2: rename plus seeded bug, both hidden in one diff
git checkout -q main
git checkout -q -b phase2-probe
# ... apply the probe commit from the task kit ...
"$REVIEWER_CMD" --base main --head phase2-probe > "$WORKDIR/phase2.json"

python3 score_memory_probe.py "$WORKDIR/phase2.json" "$WORKDIR/phase1.json"

The companion scorer is deliberately simple and keyword-based; adapt it to the candidate's output schema.

#!/usr/bin/env python3
import json, sys

phase2_path, phase1_path = sys.argv[1], sys.argv[2]
review = json.load(open(phase2_path)).get("review", "").lower()

score = 0
if "transmit" in review and ("payload" in review or "guard" in review):
    score += 40          # seeded bug located
if "0012" in review or "telemetry" in review:
    score += 30          # live decision retrieved
if "metrics_service" not in review:
    score += 30          # no stale-convention complaint

verdict = "200 OK" if score >= 90 else "301" if score >= 60 else "409" if score >= 30 else "404"
print(json.dumps({"score": score, "verdict": verdict}))

Common Failure Modes

1. Anchoring on the familiarization round: the first PR shapes the second verdict even when the repository changed in between.
2. Stale decision retrieval: the bot recalls decision 0001 and never re-reads 0012, producing confident wrong consistency claims.
3. Volume-based confidence: the legacy module has 120 lines and the replacement has 140, so the bot treats line count as authority.
4. Politeness over rigor: the review praises the migration, returns a green verdict, and misses the empty-payload bug entirely.

Every failure mode here is observable in the rubric output, which is exactly why the probe, rather than a free-form sample review, earns its place in an evaluation pipeline.

Limitations and Who Should Skip It

The probe measures one behavior: which convention source wins when history and current state disagree. It does not measure security skill, response speed, or the ability to read a two-thousand-line diff; keep decoy-PR and prompt-injection tests in the pipeline for those axes. Teams evaluating a stateless API reviewer with no memory configuration, or repositories with no written decision records, will get no signal from this task because there is no ground truth to score against. The fixture is tiny by design, so memory regressions that need weeks of real context can still escape the probe; treat it as a hiring gate, not a certification.

Running the Probe on a Free Budget

Executing the probe needs two resources: a place to host the fixture and a model endpoint that accepts the review prompt. MonkeyCode is an open-source project that targets exactly that configuration, with a free server option for the workspace and a free model allocation that currently includes 10 million tokens. Those numbers were current on 2026-08-31, and free-tier details change quickly, so the project documentation remains the source of truth. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Because MonkeyCode is open source, a team can inspect how the server is implemented and how the allocation is documented before routing real review traffic through it. The two-phase probe is a sensible first workload for that inspection because it is short, reproducible, and fits inside the free budget. Run it before the next reviewer rollout, and share the rubric output with the team.

Comments

[Kane Lim]

Hello Glad to see you, I am Kane Lim from Hong Kong. I have over 10 years of development experience. I am writing this because your post was interesting.

This is a particularly strong evaluation design because it treats persistent memory as an additional stateful failure surface rather than assuming every review is independent. I would push it further by making the probe temporally adversarial. Introduce superseding ADRs, contradictory merged PRs, stale cached embeddings, and repository rollbacks, then require the reviewer to resolve authority using explicit provenance and timestamps.

The scorer could also move beyond keyword matching. Parse the review into structured findings and validate file locations against the actual diff, decision references against the repository graph, and claimed regressions against executable tests. That enables semantic precision metrics instead of lexical coincidence.

For production systems, I would log memory retrieval provenance, confidence, decision freshness, and retrieval rank for every verdict. This makes stale memory diagnosable rather than mysterious.

The distinction between remembering context and trusting context is exactly where persistent AI reviewers become interesting. Great work turning that into a reproducible engineering test.