What I Learned Letting an AI Agent Security-Review 300 Pull Requests

TL;DR

I wired a dedicated security-reviewer agent into my pull request flow and let it run on ~300 PRs over four months. It caught 11 real vulnerabilities my linters missed — and cried wolf a lot until I added a second agent whose only job was to disprove the first one. Here's the setup, the checklist, and the honest scorecard.

The Problem

I run a fully autonomous implementation system: agents pick up work, write code, and open pull requests largely without me. That works great for throughput and terribly for my peace of mind.

The specific fear wasn't "the agent writes bad code." Bad code shows up in tests. The fear was the agent writes code that works perfectly and is also a security hole. A happy-path integration test doesn't care that you interpolated a user-controlled string into a shell command. It passes. It ships.

My existing defenses had a shape-shaped hole in them:

• Linters (ESLint, Ruff) catch style, not intent.
• SAST tools (Semgrep, CodeQL) catch known patterns really well — and produce nothing at all for logic flaws like "this endpoint checks that you're authenticated but never checks that the record belongs to you."
• Dependency scanners catch CVEs in package.json, not in the code I just wrote.
• Me — reviewing my own agent's PRs at 11pm, which is exactly the review quality you'd expect.

That last category is where broken access control lives. It's the #1 item on the OWASP Top 10 and it is essentially invisible to pattern matchers, because the vulnerable code and the safe code look identical — the difference is one missing WHERE owner_id = ?.

So the question became: can a language model, which is genuinely good at "read this diff and tell me what the author assumed," cover the gap that pattern matching can't?

Short version: yes, but only with an adversarial second pass. Here's how I got there.

How I Solved It

The architecture

The security reviewer is a separate agent with its own system prompt, not an extra bullet in my main agent's instructions. This distinction turned out to matter more than anything else in the build, and I'll come back to why in the lessons.

flowchart TD
    A[PR opened] --> B[Collect diff + changed file context]
    B --> C[Security reviewer agent<br/>runs threat checklist]
    C -->|0 findings| G[Pass ✅]
    C -->|N findings| D{Verifier agent<br/>tries to REFUTE each}
    D -->|refuted| E[Dropped, logged only]
    D -->|survives| F[Blocking PR comment 🚩]
    F --> H[I review, then fix or dismiss]

The trigger is a plain Git hook plus a CI step, nothing exotic:

#!/usr/bin/env bash
# .git/hooks/pre-push — cheap local gate before CI even sees it
set -euo pipefail

DIFF=$(git diff --stat "origin/main...HEAD" -- \
  ':!*.lock' ':!*.snap' ':!dist/*' | tail -1)

# Skip the review entirely for docs-only pushes — no point burning tokens
if git diff --name-only "origin/main...HEAD" | grep -qvE '\.(md|txt|svg)$'; then
  claude -p "$(cat .github/prompts/security-review.md)" \
    --allowedTools "Read,Grep,Glob,Bash(git diff:*)" \
    --output-format json > /tmp/secreview.json
else
  echo "docs-only push, skipping security review"
fi

Note the --allowedTools list: read-only. The security reviewer can read files, grep, and inspect the diff. It cannot write, cannot run arbitrary shell, cannot push. A reviewer that can edit code is not a reviewer, it's a second author.

The checklist is the whole product

My first version of the prompt said, roughly, "review this diff for security issues." The output was useless — a wall of "consider validating user input" boilerplate attached to code that already validated user input.

What actually worked was giving it a specific, enumerated threat model and forcing structured output. Vague prompt, vague findings. The prompt is now about 60 lines and the load-bearing part looks like this:

For EVERY changed file, check these classes explicitly. For each one,
state either a finding or "N/A — <one line reason>". Do not skip a class
silently.

1. AUTHZ: Does every new data access path check *ownership*, not just
   authentication? Look for queries missing a tenant/user scope.
2. INJECTION: String interpolation into SQL, shell, HTML, or template
   contexts. Trace the value back to its source — is it user-reachable?
3. SECRETS: Hardcoded keys, tokens, or credentials. Also: secrets newly
   added to log statements or error messages.
4. SSRF/PATH: User-controlled URLs passed to fetch/request, or user
   strings joined into filesystem paths.
5. CRYPTO: Custom crypto, weak hashes for passwords, non-constant-time
   comparison of secrets.
6. DESERIALIZATION: pickle/yaml.load/eval on anything user-reachable.
7. RATE/DOS: New unauthenticated endpoints with no limit; unbounded
   regex or recursion on user input.

For each finding you MUST provide:
- file:line
- the exact attacker-controlled input
- a concrete exploitation path, step by step
- severity, and why it is that severity and not one lower

That last requirement — "why is it that severity and not one lower" — deleted roughly a third of my noise on its own. It's very hard to write a coherent justification for "high severity" when the input is a hardcoded enum from your own config file. The model would start writing the justification, fail to make it work, and downgrade or drop the finding.

The verifier is what made it usable

Even with a tight checklist, the first-pass agent flagged plenty of things that were fine. LLMs are agreeable. Ask one "is this a vulnerability?" and it feels a pull toward yes, because yes is the helpful-sounding answer.

So I stopped asking that question. The second stage spawns one verifier per finding, and the verifier's job is to refute:

You are trying to DISPROVE the following security finding.

<finding>{{finding_json}}</finding>

Read the actual code. Look specifically for:
- an upstream guard, middleware, or type constraint the reporter missed
- whether the "attacker-controlled" input is actually attacker-controlled
- whether the exploitation path survives contact with the real call chain

Default to refuted=true. Only return refuted=false if you can write a
working exploitation path yourself, in concrete steps, using real
identifiers from this codebase.

The "default to refuted" framing is doing real work. It flips the model's agreeableness to the side where the failure mode is cheap: a missed finding gets caught by the next review or by me, but a false positive that blocks a PR trains me to ignore the tool entirely — and an ignored security tool is worth exactly zero.

Findings that survive the verifier get posted as a blocking comment. Findings that get refuted are logged to a file, not shown. I read that log occasionally to see what kind of thing gets killed, which is how I tune the first-stage prompt.

The scorecard

Four months, ~300 PRs, mostly TypeScript and Python services. Numbers rounded and counted by hand from my log:

	Count
PRs reviewed	~300
Raw first-pass findings	412
Survived the verifier	47
Confirmed real by me	11
Real issues I'd call serious	3

The three serious ones, because specifics beat vibes:

1. A missing ownership check on a GET /api/documents/:id route. Authenticated, correctly. Any logged-in user could read any document by guessing an ID. This is the one that justified the entire project — Semgrep and CodeQL both said nothing, because there is nothing to pattern-match on.
2. A secret in an error path. The happy path redacted an API token from logs. The catch block, added later in a different PR, logged the whole config object. Two PRs, each fine alone.
3. A path traversal in an export feature that joined a user-supplied filename into an output directory. ../../ did what you'd expect.

Note the shape of #1 and #2: both are multi-file, multi-PR, semantic bugs. That's the niche. The agent has never once beaten Semgrep at finding a hardcoded AWS key — Semgrep does that instantly and for free. It wins where you need to understand what the code means.

Also worth stating plainly: 412 → 47 → 11 means the raw output was 97% noise, and even post-verifier it was under 25% precision. That's a usable tool only because the verifier stage exists and because the findings arrive at a moment when I'm already reviewing the PR.

Lessons Learned

1. Give the reviewer its own context, not an extra instruction.
My first attempt appended "also check for security issues" to the main coding agent's prompt. It never found anything meaningful, and the reason is obvious in hindsight: the agent that wrote the code already believes the code is correct. Its context is full of the reasoning that produced the bug. A fresh agent that sees only the diff has no such attachment. Separate context is the feature. This is the same reason you don't review your own PRs.

2. Enumerate the threat classes, or get boilerplate.
"Review for security issues" produces essays. "Check these seven classes, state N/A explicitly for each one you clear" produces findings. Forcing an explicit N/A also surfaces when the model is skipping a category, which is information you don't get from a silent pass.

3. An adversarial second pass beats a better first prompt.
I spent two weeks tuning the finder prompt to reduce false positives and got maybe a 20% improvement. I spent one afternoon adding a refute-by-default verifier and cut them by ~88%. Generation and evaluation are different jobs and models are visibly better at the second when you frame it as disproving something.

4. Read-only tools, always.
It's tempting to let the reviewer just fix what it finds. Don't. A reviewer with write access starts "fixing" its own false positives, and now your false positives are commits. Keep the human in the loop precisely at the point where a judgment call happens.

5. This complements SAST, it does not replace it.
Run both. Semgrep is faster, deterministic, free, and unbeatable at known patterns — use it as the first gate. The agent is slow, non-deterministic, costs tokens, and is the only thing in my stack that has ever found a broken access control bug. Wrong tool for each other's jobs.

6. Skip the review when the diff can't contain a vulnerability.
Docs-only, lockfile-only, and snapshot-only pushes get skipped. This sounds like a cost optimization, and it is, but the bigger win is that it keeps the signal-to-noise ratio of the tool's output high enough that I keep reading it.

What's Next

Two things I'm building on top of this:

• Feeding it the threat model, not just the diff. Right now the reviewer infers what matters from the code. I want it reading a short THREAT_MODEL.md per service, so it knows which endpoints are public, which data is sensitive, and which boundaries are actually trust boundaries. My hypothesis is this is where the next real jump in precision comes from.
• Tracking findings across PRs. Bug #2 above only existed because of an interaction between two separate PRs. A reviewer that sees one diff at a time is structurally blind to that class. I'm experimenting with giving it a small persistent index of prior findings and redaction points to check against.

I'll write both up once I have enough runtime to say something honest about whether they worked.

Versions, since this stuff rots fast: Claude Code (Aug 2026 release line), Node.js 22.x, Python 3.13, Semgrep OSS.

Wrap-up

If you're running AI agents that open PRs, you already have this problem — you just might not have the numbers yet. The setup is genuinely a weekend project: one prompt file with an explicit checklist, one verifier prompt that defaults to refuting, and a Git hook. Start with read-only tools and the refute stage from day one, or you'll build a tool you learn to ignore in a fortnight.

Your turn: if you've run an LLM as a security reviewer, I want to hear your precision numbers — mine were far worse than I expected before the verifier, and I suspect a lot of people are quietly seeing the same thing and not saying so. Drop them in the comments. 👇

And if you found this useful, follow me here on Dev.to — I'm writing up the rest of this autonomous system build as I go, war stories and dead ends included. 🚀