DEV Community

Niv L.
Niv L.

Posted on

My AI PR Reviewer Kept Repeating Itself — Here's the Bug and the Fix

#showdev #github #typescript #ai

I built ai-pr-reviewer — a GitHub Action that runs an LLM-powered code review on pull requests, posting inline comments the way a human reviewer would. It worked. Then I started actually using it on my own PRs, and noticed something annoying: every time I pushed a new commit to an open PR, it would re-post comments I'd already seen — including ones I'd already fixed, and one I'd already argued it was wrong about in a reply.

Not duplicate text. The same substance, reviewed fresh, every single push.
Example inline review comment posted by the bot

Why
The orchestration logic was about as simple as you'd expect:

async reviewPullRequest(req: ReviewRequest): Promise<void> {
  const { owner, repo, prNumber, headSha } = req;

  const rawDiff = await this.github.fetchPullRequestDiff(owner, repo, prNumber);
  const diff = this.filterDiff(rawDiff);
  // ... size guardrail, then hand off to the LLM
  const result = await this.llm.reviewDiff(diff);
  await this.github.postReview(owner, repo, prNumber, headSha, result.summary, result.comments);
}
Enter fullscreen mode Exit fullscreen mode

fetchPullRequestDiff always pulls the full base...head diff for the entire PR — not just what changed since the last time it ran. Every push re-triggers this from scratch, with zero memory of anything it said last time. Ten commits into a PR, it's re-reading and re-judging the same nine commits' worth of code it already reviewed, nine times over.

The fix: a marker, not a database
The obvious fix is "remember what you already reviewed." The less obvious part is where to remember it. I didn't want to add a database just for this — the whole service is intentionally stateless. But GitHub already keeps a full history of every review posted on a PR. So: stamp a hidden marker into every review body, and use GitHub's own API as the source of truth.

// Hidden in every review body we post, so we can recognize our own past
// reviews on a PR (and find the commit they were posted against) without
// needing a database — GitHub's own review list is the source of truth.
const REVIEW_MARKER = '<!-- ai-pr-reviewer:review -->';
Enter fullscreen mode Exit fullscreen mode

Then, before reviewing, look back through the PR's review history for our own last review, and pull the commit it was posted against:

async findLastReviewedCommit(owner: string, repo: string, prNumber: number): Promise<string | null> {
  const reviews = await this.octokit.paginate(this.octokit.pulls.listReviews, {
    owner, repo, pull_number: prNumber, per_page: 100,
  });

  const ours = reviews.filter((review) => review.body?.includes(REVIEW_MARKER));
  if (ours.length === 0) return null;

  // Sort by id (monotonically increasing, assigned at creation) rather
  // than trusting listReviews' response order to stay oldest-first.
  ours.sort((a, b) => a.id - b.id);
  return ours[ours.length - 1].commit_id ?? null;
}
Enter fullscreen mode Exit fullscreen mode

With that in hand, the orchestrator diffs from there to the new head, instead of from the PR's base:

private async fetchDiff(owner: string, repo: string, prNumber: number, headSha: string): Promise<string> {
  const lastReviewedSha = await this.github.findLastReviewedCommit(owner, repo, prNumber);

  if (!lastReviewedSha) {
    return this.github.fetchPullRequestDiff(owner, repo, prNumber); // first review on this PR
  }
  if (lastReviewedSha === headSha) {
    return ''; // nothing's changed since we last looked
  }

  try {
    return await this.github.fetchDiffSince(owner, repo, lastReviewedSha, headSha);
  } catch {
    return this.github.fetchPullRequestDiff(owner, repo, prNumber); // e.g. old commit unreachable after a force-push
  }
}
Enter fullscreen mode Exit fullscreen mode

Three states, three behaviors: never reviewed this PR → full diff. Already reviewed this exact commit → do nothing, don't even call the LLM. Reviewed an earlier commit → diff just the new part.

The fix caught a bug in itself
Here's the part I didn't expect: this repo dogfoods itself — it reviews its own pull requests. When I opened the PR for this exact change, it reviewed itself and pointed out that ours[ours.length - 1] was trusting listReviews' response order to stay oldest-first, which isn't actually a documented guarantee — sorting by id (as shown above) was the fix, added after the bot flagged it on its own PR. A review-deduplication feature getting correctness-reviewed by the thing it was built to fix felt like a good sign it was working.

Result
Verified end to end: a PR with no changes since the last review now gets skipped entirely — no LLM call, no repeated comments. A PR with new commits gets reviewed on just the delta.

Full diff and tests: PR #9. If you want to try it: it's on the GitHub Marketplace now, one line in a workflow file.

I'm Niv, a backend Tech Lead working on NestJS/AI infra projects. GitHub · LinkedIn

Top comments (0)