I taught my code reviewer with Hindsight—and it remembered too well

jyothisri laveti — Mon, 28 Sep 2026 18:03:32 +0000

Our team switched from Redux to Zustand four months ago. Last week, my AI code reviewer told a new teammate to "use Redux, as decided in PR #12."

The memory worked perfectly. That was the problem.

What I built

I built a code review agent that connects to GitHub pull requests. When a human reviewer leaves feedback such as "use Tailwind, not inline CSS" or "validate API parameters for null," the agent stores it. On the next PR, it recalls the relevant team decisions and uses them in its review.

The system has three parts:

A GitHub webhook that receives new and updated pull requests
An LLM that reads the diff and drafts the review
Hindsight, the open-source agent memory system, which stores team decisions (retain) and brings them back when they're relevant (recall)

Without memory, the reviewer behaves like a generic linter with opinions. With memory, it sounds like a teammate who's been on the project for a year. Most of this article is about that second half — and about what went wrong once it actually worked.

The problem: teams repeat the same review comments

Every team I've worked with leaves the same pull request comments again and again. "Don't use inline CSS, we use Tailwind." "Add a null check on this API parameter." "Follow the folder structure." The rule was decided once, but people forget it, new teammates never hear it, and a senior reviewer ends up typing it out for the fourth time.

Generic AI reviewers don't fix this. They catch syntax and style issues, but they know nothing about the decisions your team made in earlier reviews. Every review starts from zero.

What I wanted was a reviewer that gets better with every pull request: generic on day one, aware of the team's conventions after a few reviews, and eventually able to flag the mistakes that keep recurring.

Why I stopped using SQLite for memory

My first prototype was a SQLite table of past review comments with keyword search. It broke within a day, because reviewers rarely repeat themselves in the same words. One writes "don't style inline," another writes "use utility classes." Both describe the same rule, but keyword search sees two unrelated sentences.

I needed retrieval by meaning, and I didn't want to spend the project building a retrieval layer from scratch. That's what agent memory is for. Moving to Hindsight gave me three things I would otherwise have had to build by hand:

Recall by meaning. A query about "styling conventions" finds the Tailwind decision even when the original comment never used those exact words.
Separate memory per team. Each team gets its own memory bank, so one team's conventions never leak into another team's reviews.
A small, predictable API. The Hindsight documentation got me to a working retain-and-recall loop in an afternoon. The whole integration is really two function calls. ## The core loop

When a human reviewer leaves a comment on a PR, the agent retains it along with its source:

hindsight.retain(
    bank_id="team-rules",
    content="PR #42 (reviewer: priya): Team decided to use Tailwind "
            "classes instead of inline CSS.",
)

When a new PR comes in, the agent recalls before it writes anything:

memories = hindsight.recall(
    bank_id="team-rules",
    query="styling conventions relevant to this diff",
)

prompt = f"""Review this diff.
Team decisions you must respect:
{format_memories(memories)}

Diff:
{diff}"""

The recalled memories go straight into the prompt, next to the diff. Every memory carries its source PR number, so the review can point to exactly where a rule came from. That detail mattered more than I expected — developers accept "decided in PR #42" far more readily than "the AI suggests."

When the same mistake shows up a fifth time

Recall on its own answers "what did we decide." It doesn't answer "is this happening again and again." For that, MemReview counts how often each recalled rule actually matches a new diff, and once a rule crosses a threshold, the review cites every violated convention at once instead of just one:

This violates the convention from PR #51: route handlers must not call the database directly; use the service layer.
This violates the convention from PR #37: all API handler functions must null-check parameters before use.

By the time a pattern has come up this many times, the agent isn't just linting — it's pointing at a real, recurring gap in the codebase, sourced back to the exact PR where the team decided the rule.

Letting developers grade the agent's memory

Not every recalled memory is still useful, and I didn't want MemReview to just assume its own suggestions were right. Every review comment ships with Accept and Reject buttons.

A Reject doesn't delete the underlying memory — it's a signal. A rule that gets rejected repeatedly is probably outdated, worded badly, or no longer matches how the team actually works. An Accept reinforces that the memory is still pulling its weight.

This turned out to matter more than the recall logic itself. An agent that repeats a stale rule with total confidence is worse than one with no memory, because people stop reading its comments at all — including the correct ones.

Right now, a rejected suggestion lowers that memory's trust weight, so a rule that gets rejected repeatedly shows up less often in future recalls instead of continuing to interrupt reviews.

What didn't work

Recall was too broad early on. Before I narrowed queries to the diff's actual topic, unrelated conventions crept into reviews and made them noisy instead of helpful.
Not every rejection means the rule is wrong. Sometimes a developer rejects a correct suggestion just because it's inconvenient that week. Trust weight helps, but it isn't a perfect signal on its own.
The agent only knows what got written down. A decision made in a call or a Slack thread and never mentioned in a PR simply doesn't exist for it.
Citing multiple PRs at once can overwhelm a small diff. When three conventions fire on one file, the review reads more like a checklist than feedback. I'm still tuning how many memories to surface per comment.

Lessons learned

Memory needs feedback, not just storage. Recall alone doesn't tell you if a rule is still good — Accept/Reject does.
Store the source with every memory. A PR number turns a suggestion into something a developer can go verify, and it's the single detail that built the most trust.
Recurring violations are more convincing than single ones. Citing PR #51 and PR #37 together on one diff makes the pattern undeniable in a way one citation doesn't.
Let humans push back cheaply. A one-click Reject is enough friction to catch bad memories without turning code review into a chore.
Retrieve by meaning, and don't build it yourself. For messy human feedback, semantic recall beats keyword matching every time, and using Hindsight meant my time went into the review logic instead of a retrieval layer.

If you're building agents that need to remember, start with the Hindsight docs and design a way for humans to correct it from day one — not as an afterthought once the memory has already gone stale.

The Hindsight GitHub repository: https://github.com/vectorize-io/hindsight
The documentation for Hindsight: https://hindsight.vectorize.io/
The agent memory page on Vectorize: https://vectorize.io/what-is-agent-memory

DEV Community: jyothisri laveti