I Built a SAST/DAST Triage Pipeline Because Scanner Noise Was Killing My Signal
The Problem
I was running Bandit and Semgrep on a Python project and got 847 findings in a single scan.
I opened the report.
The top result was:
B310: Audit url open for permitted schemes. Allowing use of often unexpected schemes.
The URL was:
http://localhost:11434
Hardcoded. In a config file. Not exploitable in this context.
But the scanner doesn't know that.
This is the reality of SAST tooling: high recall, low precision. The philosophy is essentially flag everything and let someone sort it out.
At scale, that means hundreds of false positives per sprint, while the real SQL injection or path traversal can end up buried on page 6.
I wanted to fix that.
What I Built
P1 — SAST/DAST Triage Tool.
It's a Python pipeline that:
- Ingests findings from multiple security scanners
- Normalizes them into a common format
- Deduplicates overlapping findings
- Scores them using CWE-based heuristics
- Uses a local LLM to classify likely false positives
- Supports persistent suppressions
- Generates Markdown, HTML, and SARIF reports
- Integrates with GitHub Code Scanning
The goal isn't to replace SAST/DAST scanners.
It's to make their output useful to humans.
Stage 1: Ingest
P1 currently supports:
- Bandit JSON
- Semgrep JSON
- Trivy JSON
- OWASP ZAP
Each finding gets normalized into a common Finding object:
@dataclass
class Finding:
tool: str
rule_id: str
cwe_id: str | None
severity: str
file: str
line: int
title: str
description: str
status: str = "active" # active | likely_fp | suppressed
confidence: float | None = None # 0.0–1.0 from LLM
fp_reason: str | None = None
This makes the rest of the pipeline scanner-agnostic.
Adding another scanner becomes a parser problem instead of a pipeline redesign.
Stage 2: Deduplicate
This is where I think a lot of security pipelines fall down.
If Bandit and Semgrep both flag the same subprocess.call() on line 47, I don't want two findings.
I want one finding with both tools attached to it.
The deduplication key is:
key = (
finding.cwe_id or finding.rule_id,
norm_path,
finding.line,
)
So the identity is based on:
CWE/rule + normalized file path + line number
—not the scanner-specific rule ID.
Rule IDs are tool-specific, so they aren't useful for cross-scanner deduplication.
The result might look conceptually like:
CWE-78
src/utils.py:47
Tools: Bandit, Semgrep
Severity: HIGH
One issue. Two scanners.
Stage 3: Score
Next comes prioritization.
I'm using a simple CWE-based heuristic rather than CVSS.
For example:
| Category | Priority |
|---|---|
| Command injection / CWE-78 | HIGH |
| SQL injection / CWE-89 | HIGH |
| Path traversal / CWE-22 | HIGH |
| Weak cryptography | MEDIUM |
| Debug artifacts | LOW |
Why not CVSS?
Because CVSS is useful when you have enough context to meaningfully assess things like exploitability, attack complexity, privileges required, and impact.
At automated triage time, I often don't have that context.
A lightweight heuristic is faster and, for this particular purpose, good enough to establish an initial priority.
The important thing is that severity and triage priority are separate concepts.
Stage 4: LLM Filter
This is the interesting part.
I send each finding to a local Ollama instance.
No cloud API. No source code leaving the machine.
The prompt is roughly:
You are a security engineer doing SAST triage.
Given this finding, determine if it is a true positive
or likely false positive.
Return JSON:
{
"verdict": "true_positive|likely_fp",
"confidence": 0.0-1.0,
"reason": "string"
}
The model receives the finding metadata — title, description, file, and line number — rather than the entire source tree.
That keeps prompts small and latency low.
More importantly, this isn't intended to be an autonomous security decision-maker.
The LLM is acting as a triage assistant.
Findings classified as likely_fp with confidence above 0.8 are filtered from the primary report, but they're still retained in the audit trail.
That distinction matters.
Stage 5: Suppressions
Some findings are known-good and aren't going to become interesting on the next scan.
For example, that B310 urlopen warning isn't going to disappear just because I run the scanner again.
I don't want to manually dismiss it every sprint.
So P1 supports persistent suppressions:
suppressions:
- rule_id: B310
reason: >
All urlopen calls use scheme-validated HTTP/HTTPS
URLs from internal configuration.
- rule_id: python.lang.security.audit.subprocess-without-shell
file_glob: "fuzz_harness.py"
reason: >
Subprocess use is intentional — this file is the
fuzzing harness.
Matching uses AND logic when multiple fields are provided.
So if both rule_id and file_glob are specified, both must match.
The finding isn't deleted.
Its status becomes:
suppressed
That means the audit trail stays intact.
Stage 6: Output
P1 currently generates three report formats.
Markdown
Designed for humans:
- Severity summary
- Active findings
- Likely false positives
- Suppressed findings
- LLM reasoning
SARIF 2.1.0
This is the integration I care about most.
The findings can be imported into GitHub Code Scanning, which means security issues can appear directly alongside code review.
Developers don't have to leave GitHub, open another dashboard, and figure out which finding actually matters.
HTML
A self-contained dark-theme report with no external dependencies.
Useful for local reviews and sharing scan results as a single file.
Results
I ran P1 against itself.
The initial scan produced:
17 raw findings
After deduplication:
14 findings
After LLM triage:
4 findings requiring attention
The remaining findings were mostly things like:
- A path traversal finding
- Hardcoded credential patterns in test fixtures
- Findings that were suppressed by policy
- Likely false positives
Total pipeline runtime:
~18 seconds
That's fast enough for local development and CI without making security scanning feel like a separate workflow.
Why Not Just Use Snyk?
Snyk is excellent. So is Veracode.
I'm not trying to argue that a small Python project is going to replace commercial application security platforms.
The reason I built P1 is different.
1. Cost
Commercial security platforms can be expensive for a solo developer or small organization.
If you're already running open-source scanners, adding a lightweight triage layer can be much cheaper.
2. Data
P1 uses a local LLM through Ollama.
For environments where sending source code or security findings to a third-party cloud isn't acceptable, keeping the classification step local can be valuable.
3. Transparency
I don't want a black-box label that says:
False positive: 94%
and nothing else.
P1 stores the model's reasoning alongside the finding.
You can inspect why something was classified as a likely false positive.
4. Learning
This is probably the biggest reason.
I understand every stage of the pipeline.
I can change the deduplication strategy.
I can change the scoring model.
I can inspect the prompts.
I can add a new scanner.
I can decide exactly what gets suppressed.
It's not just a product I use.
It's a security system I understand.
What I Learned
The biggest lesson wasn't about LLMs.
It was about signal management.
Security scanners are optimized for finding things.
Humans are optimized for deciding what matters.
Those are different jobs.
Trying to make the scanner itself perfectly precise isn't necessarily the right answer.
A better architecture can be:
┌─────────────┐
│ Scanners │
│ Bandit │
│ Semgrep │
│ Trivy │
│ ZAP │
└──────┬──────┘
│
▼
┌─────────────┐
│ Ingest & │
│ Normalize │
└──────┬──────┘
│
▼
┌─────────────┐
│ Deduplicate │
└──────┬──────┘
│
▼
┌─────────────┐
│ Scoring │
└──────┬──────┘
│
▼
┌─────────────┐
│ Local LLM │
│ Triage │
└──────┬──────┘
│
▼
┌─────────────────────┐
│ Suppress / Review │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Markdown / HTML / │
│ SARIF │
└─────────────────────┘
The scanner produces the signal.
The triage layer makes the signal usable.
What's Next?
There are a few things I want to add next:
- Semgrep autofix integration — let the LLM suggest a fix, not just classify the finding
- GitHub Actions integration — optionally block PRs on CRITICAL findings
- Web UI — review findings, approve suppressions, and track triage decisions
- Better source-aware analysis — give the model targeted source context when the finding can't be classified confidently from metadata alone
- Feedback loops — use previous human triage decisions to improve future classification
The last one is probably the most interesting.
If a developer repeatedly marks a particular pattern as a false positive, the system should eventually learn that policy rather than asking the same question on every scan.
Final Thought
847 findings sounds like a security success.
It isn't, if nobody has the time to investigate them.
The goal of P1 isn't to produce fewer findings.
It's to produce fewer findings that humans actually need to think about.
That's the difference between having a security scanner and having a security workflow.
Code: https://github.com/PyHackSecGP/p1-sast-dast-triage
If you're building something similar, I'd love to hear how you're handling scanner noise, deduplication, and false-positive triage.
Top comments (0)