When I started my MS, I realized pretty quickly that I missed having real code reviews. The kind where someone actually catches the stuff you didn't think to check.
At my previous company, there was always someone to catch the dumb stuff, a poorly named variable, a query that'd blow up at scale, an access control you forgot to check. In grad school, building coursework projects and personal side projects, that safety net was just... gone. It was just me and my code.
With all the AI and RAG buzz going on, I figured — why not build it myself? That's how PR Sentinel started: a selfish tool, built because I missed having someone look over my shoulder.
Here's what it actually took to build it.
The Architecture: Three Agents, One Review
The first real decision was: one AI agent or multiple?
I went with multiple, and not just for the architecture points. LLMs hallucinate. Ask one model to simultaneously check for SQL injection, N+1 queries, and whether your function names make sense — and it'll do all three badly. So I split the job into three focused agents:
- Security Agent — hardcoded secrets, injection vectors, broken access controls
- Performance Agent — N+1 queries, un-optimized loops, memory leaks
- Quality Agent — DRY violations, naming, modularity
Each agent runs independently using LangGraph, with asyncio.gather() firing them in parallel. The results come back to an aggregator node that assembles the final review. Compared to running them sequentially, this cuts total latency by over 60%.
But the architecture decision was the easy part. Getting it to actually work that took a while.
What Actually Broke (A Lot)
Before I even got to multi-agent stuff, I wasted serious time just deciding which LLM API to use. Claude? Gemini? Every API has different configurations, different rate limits, different quirks. I kept switching and re-learning each time.
Then came LangGraph itself. Understanding the cycle behind it, adding nodes to the graph, wiring up tools, setting conditional edges, etc. I genuinely messed this up multiple times. The docs make it look clean. It is not clean when you're building it.
The first thing that broke in a way I didn't expect: tool calling wasn't working. The agents were reviewing the changed files just fine, but they weren't exploring the rest of the codebase using the embedded vector store. Meaning: no real context. Just surface-level feedback on a diff.
I didn't catch it for a while. It was only after I sat down and actually read through the generated reviews carefully that I realized they were missing the bigger picture. When I dug into the code, it turned out to be a typo — a misspelled key that caused the tool to silently return nothing. No error. Just nothing.
That's when I learned the importance of logging at every single step. Obvious in hindsight. Painful to learn.
The Thundering Herd: Fixing the API Stampede
Once I got agents running in parallel, a new problem showed up: all three would fire their API requests at the exact same millisecond, immediately hitting Gemini's rate limits with 429 and 503 errors.
The fix was an exponential backoff with randomized jitter:
import random
import time
def call_with_gemini_retry(fn, *args, **kwargs):
MAX_RETRIES = 5
for attempt in range(1, MAX_RETRIES + 1):
try:
return fn(*args, **kwargs)
except Exception as e:
if "503" in str(e) or "429" in str(e):
delay = (2 ** attempt) + random.uniform(0, 1)
time.sleep(delay)
else:
raise e
The jitter part is what actually matters. Without it, all three agents back off by exactly the same amount and collide again on the retry. The random offset staggers them just enough.
And yes — free tier API limits are brutal. If you're building something like this, pick your model carefully before you start hammering it with test PRs.
Incremental RAG: The Accidental Optimization
For the agents to understand the full codebase — not just the diff — I integrated Pinecone as a vector database. Every file gets chunked and embedded, so agents can semantically search for relevant code before writing their review.
The naive approach was re-vectorizing the entire repo on every PR. That lasts about two test runs before you look at your usage bill and reconsider your life choices.
So I built incremental ingestion instead. When a webhook fires, the backend looks at the files_changed payload, deletes only the old embeddings for those files, and re-embeds the new versions:
for file_path in changed_file_names:
delete_file_chunks(file_path, namespace)
updated_files = fetch_specific_files(repo_url, changed_file_names, ref)
chunks = chunk_files(updated_files)
store_chunks(chunks, namespace)
This keeps Pinecone in sync with the repo while cutting token usage by over 99% compared to full re-ingestion.
There was also a subtle bug here I didn't catch for weeks: I was reading from the main branch during ingestion instead of the PR's head branch. Most of my test PRs only had modified files, not new ones — so it never surfaced clearly in the logs. New files were just silently skipped. Fixed it eventually, but it's the kind of bug that only shows up when the conditions are just right to hide it.
GitHub App Auth: Making It Actually Installable
The original version used a hardcoded Personal Access Token. Fine for local testing, completely unusable if anyone else wants to use it — they'd have to hand over their own tokens.
I transitioned to a proper GitHub App architecture. Now the server holds an RSA private key. When someone installs the app, GitHub sends an installation_id, and the backend dynamically signs a JWT and exchanges it for a scoped, 60-minute access token.
The user clicks Install. That's it. No token configuration, no API keys shared.
The GitHub docs for this flow are... fine. But there are enough gaps that I spent more time on this than I expected. If you're building a GitHub App for the first time, set aside extra time for the auth layer.
What I'd Do Differently
Honestly? I'd explore different RAG architectures more carefully before building. The current chunking is naive — splitting by character count means a function can get cut in half, which degrades retrieval quality. I want to replace it with AST-based chunking using tree-sitter, so chunks always align with complete functions and classes.
I'd also spend more time upfront thinking about which model actually fits the use case — balancing capability, rate limits, and cost — instead of swapping APIs mid-build when the limits hit me.
The Moment It Clicked
After all of that, the wrong API choices, the silent typos, the accidental re-vectorizations, the weeks-long PR branch bug, there was one moment that made it worth it.
I opened a test PR. The Pinecone namespace updated. One agent fired. The review came back. It worked.
And then I wired up the async multi-agent flow and watched all three fire simultaneously.
That was the moment. Seeing the thing actually work! Not as a demo, but as a real system doing what it was supposed to do, made all the broken stuff feel like it had a point.
If you want to try it on your own repos, the PR Sentinel GitHub App is publicly installable. The source code is on my GitHub if you want to dig into the implementation.
Top comments (2)
This was indeed an informative read,
Nice