DEV Community

Istiak Mahmud
Istiak Mahmud

Posted on

I built an open-source AI that security-reviews every pull request — and maps each bug to PCI-DSS, SOC 2 & GDPR

#python #security #opensource #devops

Code review is where most security bugs are supposed to get caught. In practice it's slow, inconsistent, and depends entirely on who happens to be looking that day. So I built GuardianCI — a CI check that reviews every pull request for security issues automatically, comments inline at the exact line, and blocks the merge on anything CRITICAL.

It's open source (MIT), works on GitHub and GitLab, and runs for free on Google Gemini's tier — or your own OpenAI, Anthropic, Groq, or local Ollama endpoint.

Does it actually work?

I planted vulnerabilities in a real project and opened a PR. Every one got caught, with an inline comment at the exact line:

Planted Caught Severity
Hardcoded DB URL + OpenAI key ✅ CRITICAL
Hardcoded JWT secret ✅ CRITICAL
SQL injection (f-string) ✅ CRITICAL
Command injection (shell=True) ✅ CRITICAL
JWT alg=none bypass ✅ CRITICAL
Disabled TLS (verify=False) ✅ WARN
Wildcard CORS ["*"] ✅ WARN
Debug logging of a password ✅ → escalated to CRITICAL CRITICAL

The most interesting one: the password-logging issue was labelled WARN in my own rules, but the model escalated it to CRITICAL — correctly arguing that logging a plaintext password is a data-exposure issue (PCI-DSS 3.4, SOC 2 CC7.2, GDPR Art. 32), not just a config smell. That's reasoning, not pattern-matching.

How it works — defense in depth

GuardianCI doesn't trust the LLM alone. It runs three layers on each diff:

  1. Local regex pre-scan (free, no API call) — catches hardcoded secrets, alg=none, SQL f-strings, verify=False, sensitive logging.
  2. gitleaks — dedicated secret detection.
  3. LLM review — the reasoning layer that catches what patterns can't (command injection, broken CORS, missing auth checks) and maps each finding to a PCI-DSS 4.0 / SOC 2 / GDPR control.

Findings from all layers are deduped, validated against the changed lines (so the model can't invent issues outside the diff), and posted as inline review comments. CRITICAL findings block the merge; it can even open a draft fix PR.

It's built to be cheap and safe in CI: large diffs only send high-risk files to the LLM, already-reviewed commits are skipped, the diff is wrapped in delimiters to resist prompt injection, and any quota/parse error fails open with an explanatory comment instead of breaking your pipeline.

Try it (about 5 minutes)

  1. Copy the workflow + scripts/ + requirements/ into your repo.
  2. Add a GEMINI_API_KEY secret (free key from Google AI Studio).
  3. Open a PR.

Full setup, including OpenAI/Anthropic/GitLab config, is in SETUP.md.

Honest limitations

I'd rather you trust it because I'm upfront about where it stops:

  • The regex layer is heuristic — false positives happen (there's a one-click dismissal mechanism that remembers them).
  • The LLM is assistive, not a guarantee — every finding is still reviewed by a human before merge.
  • It's v0.1.0. Proven working, but young.

I'd love your help

Especially after PRs for:

  • New detectors (e.g. pickle.loads, SSRF patterns)
  • New CI platforms (Bitbucket, Azure DevOps)
  • An end-to-end test for the main review pipeline

There are good first issue labels to start from. Repo and contributing guide here:

👉 https://github.com/noobbatman/GuardianCI

If you give it a spin, I'd genuinely love to hear your false-positive rate — that's the number that decides whether a tool like this is worth keeping in CI. Drop it in the comments.

Top comments (0)