DEV Community

Ramesh S
Ramesh S

Posted on

Building an AI Code Review Agent: Architecture, LangGraph, and Practical Lessons

#ai #codereview #langchain #agentaichallenge

How I built a multi-step AI agent that reviews pull requests automatically - and what I learned making it actually useful, not just clever.

The problem: code review doesn't scale with team size

Every team I've worked with hits the same wall. Pull requests pile up. Senior engineers are the only ones who catch the subtle bugs - the N+1 query, the hardcoded secret, the missing edge case - but senior engineers are also the busiest people on the team. So reviews get rushed, or they get delayed, and either way something slips through.

Most "AI code review" tools I looked at were closed-source, expensive, or both. I wanted to understand how one of these actually works under the hood, and build something a small team could run themselves, on their own repos, without sending code to a black box.

So I built agentic-code-reviewer - an open-source bot that listens for new pull requests on GitHub and reviews them using a multi-step AI agent built with LangGraph, then posts specific, line-level feedback as a PR comment.

The solution: webhook in, structured review out

When a pull request opens, the bot does this:

  1. Receives the event through a GitHub webhook
  2. Fetches the diff using the GitHub API (Octokit)
  3. Runs it through a LangGraph agent that checks for security issues, performance problems, and architecture concerns, step by step
  4. Posts a comment on the PR with specific, line-referenced feedback
  5. Stores the review in a database, so review history builds up over time

The important part here is "step by step." A single prompt asking an LLM to "review this code" tends to give vague, generic feedback. Splitting the review into stages - security pass, performance pass, architecture pass - gives much more focused results, because each stage only has one job.

Tech stack

Layer Technology
Agent framework LangGraph (multi-step reasoning)
LLM OpenAI GPT-4 or Claude (you choose)
Backend Node.js + Express
Database PostgreSQL (review history)
GitHub integration Octokit
Webhooks Express middleware + signature check

I kept the stack close to what most Node.js teams already run. The idea was never to show off a new framework - it was to build something a team could actually host and trust.

Architecture 

 GitHub PR opened
        │
        ▼
 Webhook (signed, verified)
        │
        ▼
 Express API ──► Fetch diff (Octokit)
        │
        ▼
 LangGraph Agent
   ├── Security check
   ├── Performance check
   └── Architecture check
        │
        ▼
 Post PR comment + save to PostgreSQL
Enter fullscreen mode Exit fullscreen mode

Each check in the LangGraph flow is its own node. This means I can test, tune, or even disable one check without touching the others - useful when a team only cares about security issues, for example, and doesn't want architecture opinions on every PR.

Code walkthrough (the useful bits)

Webhook signature checking, so random requests can't trigger a fake review:

function verifySignature(payload: string, signature: string) {
  const hmac = crypto.createHmac("sha256", process.env.GITHUB_WEBHOOK_SECRET!);
  const digest = "sha256=" + hmac.update(payload).digest("hex");
  return crypto.timingSafeEqual(Buffer.from(digest), Buffer.from(signature));
}
Enter fullscreen mode Exit fullscreen mode

This is a small piece of code, but skipping it is a common mistake in webhook-based tools. Without it, anyone who finds your webhook URL could send fake events.

The LangGraph flow is built as separate nodes, not one big prompt:

const graph = new StateGraph(ReviewState)
  .addNode("security", securityCheckNode)
  .addNode("performance", performanceCheckNode)
  .addNode("architecture", architectureCheckNode)
  .addEdge("security", "performance")
  .addEdge("performance", "architecture");
Enter fullscreen mode Exit fullscreen mode

Each node gets the diff and returns only its own findings, which then get combined into one comment. This made debugging much easier - if the performance check gave a strange answer, I only had to look at one node, not one giant prompt.

Lessons learned

What went well:

  • Splitting the review into separate checks (security, performance, architecture) gave noticeably better, more specific feedback than one combined prompt. This was the single biggest quality improvement in the whole project.
  • Verifying webhook signatures from day one meant I never had to go back and bolt on security later - it was part of the design from the start.
  • Storing review history in PostgreSQL, even in a simple form, made it easy to see patterns later - which kinds of issues came up most often across PRs.

What was genuinely hard:

  • False positives are part of the deal. LLM-based review sometimes flags things that aren't really problems. I had to be upfront about this in the docs rather than pretend the tool is always right - trust matters more than a perfect-sounding feature list.
  • Cost per review adds up. Running three separate LLM calls per PR (security, performance, architecture) is more accurate than one call, but it costs more and takes longer. I had to find a balance between review quality and cost that a small team could actually afford.
  • Getting comments to look like a human wrote them - short, specific, pointing at a line number - took more prompt iteration than the actual LangGraph logic did. Nobody reads a five-paragraph AI essay in a PR comment.

Where this fits into a bigger picture

This sits alongside a small set of open-source AI projects I'm building - a document Q&A RAG system, a policy Q&A tool for citizen-facing services, a voice-to-Agile-user-story generator. The thread running through all of them is the same: taking LLM orchestration (LangChain/LangGraph) and applying it to a real, specific workflow problem, instead of building a generic demo. This project is not a finished, ready-to-deploy product yet - it's a working, open-source starting point that a team could build on and harden for their own setup.

Try it / contribute

The repo is open source and quick to try locally:

git clone https://github.com/Srameshgitnow/agentic-code-reviewer.git
cd agentic-code-reviewer
npm install
npm run dev
Enter fullscreen mode Exit fullscreen mode

Full setup steps, environment variables, and how to wire up the GitHub webhook are in the README.

If you try it, hit a false positive, or have ideas on the LangGraph flow, I'd like to hear about it - issues and pull requests are open. And if the project is useful to you, a ⭐ on the repo helps other people find it:

👉 github.com/Srameshgitnow/agentic-code-reviewer

I'm a full-stack / AI engineer (React, Node.js, LangChain/LangGraph) with a background in large-scale digital delivery. I write about applied AI engineering and open-source tools - follow along for the next post in this series.

Top comments (0)