DEV Community: Harsh Soni

Building PR Sentinel- A Multi-Agent AI Code Reviewer

Harsh Soni — Wed, 17 Jun 2026 04:42:03 +0000

When I started my MS, I realized pretty quickly that I missed having real code reviews. The kind where someone actually catches the stuff you didn't think to check.

At my previous company, there was always someone to catch the dumb stuff, a poorly named variable, a query that'd blow up at scale, an access control you forgot to check. In grad school, building coursework projects and personal side projects, that safety net was just... gone. It was just me and my code.

With all the AI and RAG buzz going on, I figured — why not build it myself? That's how PR Sentinel started: a selfish tool, built because I missed having someone look over my shoulder.

Here's what it actually took to build it.

The Architecture: Three Agents, One Review

The first real decision was: one AI agent or multiple?

I went with multiple, and not just for the architecture points. LLMs hallucinate. Ask one model to simultaneously check for SQL injection, N+1 queries, and whether your function names make sense — and it'll do all three badly. So I split the job into three focused agents:

Security Agent — hardcoded secrets, injection vectors, broken access controls
Performance Agent — N+1 queries, un-optimized loops, memory leaks
Quality Agent — DRY violations, naming, modularity

Each agent runs independently using LangGraph, with asyncio.gather() firing them in parallel. The results come back to an aggregator node that assembles the final review. Compared to running them sequentially, this cuts total latency by over 60%.

But the architecture decision was the easy part. Getting it to actually work that took a while.

What Actually Broke (A Lot)

Before I even got to multi-agent stuff, I wasted serious time just deciding which LLM API to use. Claude? Gemini? Every API has different configurations, different rate limits, different quirks. I kept switching and re-learning each time.

Then came LangGraph itself. Understanding the cycle behind it, adding nodes to the graph, wiring up tools, setting conditional edges, etc. I genuinely messed this up multiple times. The docs make it look clean. It is not clean when you're building it.

The first thing that broke in a way I didn't expect: tool calling wasn't working. The agents were reviewing the changed files just fine, but they weren't exploring the rest of the codebase using the embedded vector store. Meaning: no real context. Just surface-level feedback on a diff.

I didn't catch it for a while. It was only after I sat down and actually read through the generated reviews carefully that I realized they were missing the bigger picture. When I dug into the code, it turned out to be a typo — a misspelled key that caused the tool to silently return nothing. No error. Just nothing.

That's when I learned the importance of logging at every single step. Obvious in hindsight. Painful to learn.

The Thundering Herd: Fixing the API Stampede

Once I got agents running in parallel, a new problem showed up: all three would fire their API requests at the exact same millisecond, immediately hitting Gemini's rate limits with 429 and 503 errors.

The fix was an exponential backoff with randomized jitter:

import random
import time

def call_with_gemini_retry(fn, *args, **kwargs):
    MAX_RETRIES = 5
    for attempt in range(1, MAX_RETRIES + 1):
        try:
            return fn(*args, **kwargs)
        except Exception as e:
            if "503" in str(e) or "429" in str(e):
                delay = (2 ** attempt) + random.uniform(0, 1)
                time.sleep(delay)
            else:
                raise e

The jitter part is what actually matters. Without it, all three agents back off by exactly the same amount and collide again on the retry. The random offset staggers them just enough.

And yes — free tier API limits are brutal. If you're building something like this, pick your model carefully before you start hammering it with test PRs.

Incremental RAG: The Accidental Optimization

For the agents to understand the full codebase — not just the diff — I integrated Pinecone as a vector database. Every file gets chunked and embedded, so agents can semantically search for relevant code before writing their review.

The naive approach was re-vectorizing the entire repo on every PR. That lasts about two test runs before you look at your usage bill and reconsider your life choices.

So I built incremental ingestion instead. When a webhook fires, the backend looks at the files_changed payload, deletes only the old embeddings for those files, and re-embeds the new versions:

for file_path in changed_file_names:
    delete_file_chunks(file_path, namespace)

updated_files = fetch_specific_files(repo_url, changed_file_names, ref)
chunks = chunk_files(updated_files)
store_chunks(chunks, namespace)

This keeps Pinecone in sync with the repo while cutting token usage by over 99% compared to full re-ingestion.

There was also a subtle bug here I didn't catch for weeks: I was reading from the main branch during ingestion instead of the PR's head branch. Most of my test PRs only had modified files, not new ones — so it never surfaced clearly in the logs. New files were just silently skipped. Fixed it eventually, but it's the kind of bug that only shows up when the conditions are just right to hide it.

GitHub App Auth: Making It Actually Installable

The original version used a hardcoded Personal Access Token. Fine for local testing, completely unusable if anyone else wants to use it — they'd have to hand over their own tokens.

I transitioned to a proper GitHub App architecture. Now the server holds an RSA private key. When someone installs the app, GitHub sends an installation_id, and the backend dynamically signs a JWT and exchanges it for a scoped, 60-minute access token.

The user clicks Install. That's it. No token configuration, no API keys shared.

The GitHub docs for this flow are... fine. But there are enough gaps that I spent more time on this than I expected. If you're building a GitHub App for the first time, set aside extra time for the auth layer.

What I'd Do Differently

Honestly? I'd explore different RAG architectures more carefully before building. The current chunking is naive — splitting by character count means a function can get cut in half, which degrades retrieval quality. I want to replace it with AST-based chunking using tree-sitter, so chunks always align with complete functions and classes.

I'd also spend more time upfront thinking about which model actually fits the use case — balancing capability, rate limits, and cost — instead of swapping APIs mid-build when the limits hit me.

The Moment It Clicked

After all of that, the wrong API choices, the silent typos, the accidental re-vectorizations, the weeks-long PR branch bug, there was one moment that made it worth it.

I opened a test PR. The Pinecone namespace updated. One agent fired. The review came back. It worked.

And then I wired up the async multi-agent flow and watched all three fire simultaneously.

That was the moment. Seeing the thing actually work! Not as a demo, but as a real system doing what it was supposed to do, made all the broken stuff feel like it had a point.

If you want to try it on your own repos, the PR Sentinel GitHub App is publicly installable. The source code is on my GitHub if you want to dig into the implementation.

I built an AI agent that texts me LeetCode and system design summaries every morning, here's exactly how

Harsh Soni — Sun, 24 May 2026 09:06:59 +0000

Here's my problem: I study something once and completely forget it exists until the moment I actually need it, usually mid-interview, when it's too late.

I wanted to fix this without adding another thing to my already full plate. So I built two things:

An automated script that runs every morning at 8am, picks random topics from everything I've studied, and sends me a plain-text summary to iMessage. No app to open, no habit to build, it just shows up.
An interactive study agent using Antigravity CLI that I can talk to anytime. I ask it to quiz me, it quizzes me. I ask it to fill gaps in my notes, it tells me exactly what's missing.

The cool part: I take notes in Obsidian, and instead of syncing or exporting anything, I used symlinks to point the AI directly at my actual Obsidian files. Edit in Obsidian, agent sees it instantly. One source of truth.

Here's exactly how I built it.

The morning automation

The script itself is straightforward.

It sets the path to my Obsidian vault, walks through every folder, and collects all the .md files into a list. Then it picks one randomly using random.choice(), reads its content, and builds a prompt telling Gemini what to do with it: summarise this note, plain text, no markdown, keep it short.

The API call goes directly to Gemini, response comes back as JSON, and I extract the actual text from data["candidates"][0]["content"]["parts"][0]["text"]. Then an AppleScript — macOS's built-in automation language — tells the Messages app to send it to my number.

Run the script, summary lands in iMessage. Simple.

Here's the full script:

import os
import random
import subprocess
import urllib.request
import urllib.error
import json
import time

# auto scan entire Obsidian vault for all markdown notes
vault_path = "/Users/your-username/Documents/Obsidian Vault"
notes = []
for root, dirs, files in os.walk(vault_path):
    for file in files:
        if file.endswith(".md"):
            notes.append(os.path.join(root, file))

if not notes:
    print("No notes found in Obsidian vault")
    exit(1)

# pick one random note
picked = random.choice(notes)
note_name = os.path.basename(picked)

with open(picked, "r") as f:
    content = f.read()

# build prompt
prompt = f"""You are a system design study assistant for an MSCS student.

Below are the student's personal notes on "{note_name}".
Read them and give a concise morning summary in plain text — no markdown, no asterisks, no bold, no bullet symbols.

Format exactly like this:

TOPIC: <topic name>
WHAT IT COVERS: one line overview
KEY CONCEPTS:
1. first concept
2. second concept
3. third concept
KEY TAKEAWAY: one line the student must remember
WATCH OUT FOR: one common mistake or tricky part

Keep it short and sharp — this is a morning refresh, not a lecture.

Notes:
{content}
"""

# call Gemini API with retry
api_key = os.environ.get("GEMINI_API_KEY")
url = f"https://generativelanguage.googleapis.com/v1beta/models/gemini-3.5-flash:generateContent?key={api_key}"

payload = json.dumps({
    "contents": [{"parts": [{"text": prompt}]}]
}).encode("utf-8")

summary = None
for attempt in range(3):
    try:
        req = urllib.request.Request(
            url,
            data=payload,
            headers={"Content-Type": "application/json"},
            method="POST"
        )
        with urllib.request.urlopen(req) as response:
            data = json.loads(response.read())
            summary = data["candidates"][0]["content"]["parts"][0]["text"]
            break
    except urllib.error.HTTPError as e:
        if e.code == 429:
            print(f"Rate limited, waiting 30 seconds... (attempt {attempt+1}/3)")
            time.sleep(30)
        else:
            raise

if not summary:
    print("Failed after 3 attempts, try again later")
    exit(1)

# send via iMessage
imessage_target = "your-imessage-target"  # your phone number or Apple ID
applescript = f'''
tell application "Messages"
    set targetService to 1st service whose service type = iMessage
    set targetBuddy to buddy "{imessage_target}" of targetService
    send "{summary}" to targetBuddy
end tell
'''

subprocess.run(["osascript", "-e", applescript])
print("Summary sent to iMessage!")
print(summary)

For scheduling, I used launchd: macOS's built-in task scheduler. You write a .plist file describing three things: what script to run, when to run it, and where to save the logs. Load that file into launchd and it handles everything from there.

Here's the plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.harsh.sysdesign</string>

    <key>ProgramArguments</key>
    <array>
        <string>/opt/homebrew/bin/python3</string>
        <string>/Users/your-username/Documents/system-design-study/sysdesign-morning.py</string>
    </array>

    <key>EnvironmentVariables</key>
    <dict>
        <key>GEMINI_API_KEY</key>
        <string>your-gemini-api-key</string>
    </dict>

    <key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>8</integer>
        <key>Minute</key>
        <integer>15</integer>
    </dict>

    <key>StandardOutPath</key>
    <string>/Users/your-username/Documents/system-design-study/sysdesign-morning.log</string>

    <key>StandardErrorPath</key>
    <string>/Users/your-username/Documents/system-design-study/sysdesign-morning.log</string>
</dict>
</plist>

One problem: launchd runs at 8am but my Mac might still be asleep. So I used pmset to schedule an automatic wake at 7:55am every day:

sudo pmset repeat wakeorpoweron MTWRFSU 07:55:00

One requirement — the Mac needs to be plugged in and in sleep mode, not fully shut down. Just close the lid at night and you're good.

The interactive study agent

For the interactive part, I used Antigravity CLI — Google's new agentic coding tool. The idea is simple: instead of asking AI generic questions about system design, I wanted it to know MY notes and quiz me on what I actually studied.

Two files make this work.

AGENTS.md sits at the root of my project folder. It tells the agent who I am and how to behave — think of it as a letter you write to the AI before the conversation starts. It loads automatically every time I open that folder.

SKILL.md lives inside .agents/skills/system-design/.

---
name: system-design-notes
description: Use this skill when the user asks about system design, HLD, LLD, DDIA, distributed systems, architecture patterns, or wants to be quizzed on any system design topic they have studied.
---

You are a system design study partner for an MSCS student preparing for software engineering interviews.

The user's personal Obsidian notes are in references/:
- references/HLD/ → High Level Design notes
- references/LLD/ → Low Level Design notes  
- references/DDIA/ → Designing Data Intensive Applications notes

Treat these notes as the source of truth over your general knowledge.

When quizzing:
- Ask one question at a time
- Wait for the user's answer before responding
- Compare their answer to the notes and point out specific gaps
- Tell them which note/topic the answer came from

When filling gaps:
- Point out what's missing or shallow in their notes on a topic
- Suggest what they should add

It has two parts — a description that tells the agent when to activate this skill, and a body that tells it how to behave once it does. When I type "quiz me on consistent hashing", Antigravity reads the description, finds a match, and loads my notes into context automatically. I never call it by name.

Folder structure:

system-design-study/
├── AGENTS.md
└── .agents/
    └── skills/
        └── system-design/
            ├── SKILL.md
            └── references/
                ├── HLD  →  Obsidian/HLD Concepts
                ├── LLD  →  Obsidian/LLD
                └── DDIA →  Obsidian/DDIA

The notes themselves are linked via symlinks:

ln -s ~/Documents/Obsidian\ Vault/HLD .agents/skills/system-design/references/HLD

This creates a pointer inside the project that points to my real Obsidian folder. The agent reads through the pointer. I edit in Obsidian. Same file, always fresh, zero maintenance.

Now when I open the folder and type "quiz me on HLD", it quizzes me using my own notes — not generic internet knowledge.

What broke

Three things gave me the most trouble.

First, the Gemini API kept returning 429 — Too Many Requests — even on a fresh key with zero usage. Turned out my key was tied to a VS Code project that the Gemini extension was quietly burning through in the background. Creating a new key under a separate project fixed it immediately.

Second, figuring out the Mac wake issue. launchd can schedule a script but if your Mac is asleep, it won't run. I needed pmset to wake the Mac before the script fires. And it only works when plugged in — if the laptop is on battery and closed, nothing wakes it. The fix is simple: just don't shut down, close the lid instead. But it took me a while to figure out why the automation wasn't running at all.

What my mornings look like now

I wake up, check iMessage, and two summaries are sitting there. One with 5 random LeetCode problems and their approaches. One with a random system design topic from my Obsidian notes.

It takes two minutes to read both. I'm not learning anything new — I'm just keeping what I already studied from fading. That's the whole point.

For the interactive part, I open the terminal and type:

cd ~/Documents/system-design-study
agy

Then just talk to it. "Quiz me on HLD." "What gaps do my DDIA notes have?" It responds using my actual notes. It's like having a study partner who has read everything I've written.

Was it worth it?

Yes. It took a day. But, the system runs every morning without me doing anything. The only maintenance is adding one line to a text file every time I solve a LeetCode problem.

If you're an MSCS student or prepping for interviews, the setup is worth it. You don't need any prior knowledge of AI agents or macOS automation. You just need a free Gemini API key from AI Studio and a few hours.

The code is straightforward enough that you can adapt it to whatever you study. Swap the Obsidian vault for any folder of markdown files. Change the iMessage target to email or Telegram. Run it on whatever schedule works for you.

Built with: Antigravity 2.0 CLI, Gemini API (free tier), Python, launchd, pmset, AppleScript, Obsidian