Let an AI clear out your false positives without letting it hide a real bug

Last time I wrote about wiring up a security gate that blocks merges in CI with Synapse. Someone left a comment that hit the exact sore spot:

"The gate is the easy part. Two weeks in, the team turns it off because it's red over false positives every single time."

Yep. A gate that cries wolf too often dies on its own. People start hitting "merge anyway," and by the time a real vulnerability shows up, nobody's looking anymore.

The obvious 2020s fix is to hand the whole pile to an LLM and say "delete the junk." Except: the moment a security tool lets an AI delete findings, you've just built a brand new vulnerability. The model guesses wrong once, a real SQL injection quietly drops off the report, and nobody's the wiser.

So Synapse does it differently. The AI is allowed to propose that a finding is a false positive. It is never allowed to confirm that on its own, and it is never allowed to delete anything. This whole post is a real run. Terminal output at the bottom, nothing faked.

Where the actual "gate" is

Three rules, and all three exist so you can trust the output:

1. The model only proposes. It doesn't return prose, and it doesn't return a delete command. It returns exactly one typed verdict: verdict is one of refuted | sound | uncertain, driver is a closed snake_case token (not a sentence), and confidence is 0–100. That grammar is validated in Go. The model can hallucinate a whole paragraph of reasoning and none of it reaches the report.

2. A second, different model has to agree. When the first model proposes "refuted" with confidence ≥ 75, a separate verifier model gets called to assess the same finding independently. The refutation only stands if the verifier also says "refuted" at ≥ 75. And the verifier is prompted adversarially. Its job is to stop a real bug from being waved away. This is the CLI version of the rule Synapse enforces on the server: a claim is only confirmed by a different reviewer's sealed verdict. No confirming your own work.

3. Nothing gets deleted. A "refuted" finding is retain-and-mark: it stays in the report, it stays evidence-sealed, it still shows up in the compliance table. It's just exempt from the gate's exit code. The worst a wrong verdict can do is let one finding skip --fail-on. It can never make a finding disappear.

One more thing, because it matters for the cost: this is the second layer. Before any model runs, a deterministic scope classifier already strips the obvious noise: test files, fixtures, that sort of thing. The model only gets the harder calls: is this sink actually attacker-controlled? Is that interpolated value a constant or user input? Model calls cost tokens, so they only run on production-scope, first-party source findings.

Turning it on

It's opt-in. Four environment variables:

export SYNAPSE_FP_TRIAGE_ENABLED=true                  # turn on AI false-positive triage
export SYNAPSE_LLM_BASE_URL=http://localhost:20128/v1  # any OpenAI-compatible endpoint
export SYNAPSE_FP_TRIAGE_MODEL=<proposer-model>        # a cheap/fast model to propose
export SYNAPSE_VERIFIER_MODEL=<different-model>         # a DIFFERENT model to verify (two-model consensus)

A few notes:

• SYNAPSE_LLM_BASE_URL is a plain OpenAI-style endpoint (/v1/chat/completions). Run it locally with Ollama or vLLM, or point it at OpenAI. Doesn't matter. Add SYNAPSE_LLM_API_KEY if yours needs a key.
• If you don't set SYNAPSE_VERIFIER_MODEL (or you set it to the same model), you get single-model mode. It still works, but the real gate is the second model being different. Set it to the proposer and Synapse ignores it. That's not a verifier.
• The bar is 75, the same threshold every other AI claim in Synapse has to clear.

Then you run the exact same scan command as always:

synapse-cli scan . --fail-on high

What actually happened

I built a small service with both real bugs and deliberate false positives:

• payments/store.py: a SQL query built with db.session.execute(f"... {account_id}") where account_id comes straight from request.args. That's a real SQL injection. Same file has a hashlib.md5(...) call used as a cache key for a static asset. The weak-hash rule flags it, but it has nothing to do with security.
• payments/settings.py: a hardcoded AWS access key. Real.
• reports/ledger.py: another f-string SQL query, but the interpolated value is PERIOD, a server-side constant, not user input. The pattern matcher flags every f-string SQL, so this is a textbook high-severity false positive.

Before: no AI

$ synapse-cli scan . --fail-on high

Synapse SCA dogfood – payments-service
  vulnerabilities: 0
  findings (promoted): 7
    critical  risk  0.00  Hardcoded AWS access key id (payments/settings.py:4)
    high      risk  0.00  SQL execution uses dynamic string construction (payments/store.py:14)
    high      risk  0.00  Python SQLAlchemy/raw SQL uses dynamic string construction (payments/store.py:14)
    medium    risk  0.00  Weak hash: MD5 (payments/store.py:21)
    high      risk  0.00  SQL execution uses dynamic string construction (reports/ledger.py:13)
    high      risk  0.00  Python SQLAlchemy/raw SQL uses dynamic string construction (reports/ledger.py:13)
    high      risk  0.00  AWS access key ID (payments/settings.py:4)

synapse-cli: 6 finding(s) at or above high
$ echo $?
1

Gate is red over 6 findings at or above high. Two of them (the SQL in reports/ledger.py) are false alarms you'd have to triage by hand.

After: AI on, two models

The proposer and the verifier are two different models on my endpoint:

$ export SYNAPSE_FP_TRIAGE_ENABLED=true
$ export SYNAPSE_FP_TRIAGE_MODEL=gpt-5.4-mini      # proposer
$ export SYNAPSE_VERIFIER_MODEL=gpt-5.4            # verifier (a DIFFERENT model)
$ synapse-cli scan . --fail-on high

synapse-cli: AI false-positive triage (gpt-5.4-mini, verified by gpt-5.4): critiqued 7 finding(s), 3 suspected false positive(s) held back from the gate

Synapse SCA dogfood – payments-service
  findings (promoted): 7            # <- still 7, nothing deleted
  ...
  compliance: Synapse AppSec Baseline v1.0 – 3/7 controls passing
    [FAIL] SAB-INJ-1  No injection weaknesses ...
           - SQL ... (payments/store.py:14)
           - SQL ... (reports/ledger.py:13)     # <- the FP still shows in compliance
    ...

synapse-cli: 2 suspected false positive(s) at or above high held back from the gate by AI triage (reported with a verdict; not deleted)
synapse-cli: 4 finding(s) at or above high
$ echo $?
1

Read those lines closely:

• critiqued 7 finding(s), 3 suspected false positive(s) held back: the AI looked at 7 findings and proposed 3 as false.
• 2 ... at or above high held back ... (reported with a verdict; not deleted): two of those were high-severity, held back from the gate but not deleted.
• 4 finding(s) at or above high: the 4 real bugs still count. Gate is still red (exit 1). The AI couldn't bury a real bug.
• findings (promoted): 7 is unchanged, and the false positives are still sitting in the compliance table. Nothing left the report.

Verdict by verdict

This is my favorite part. With --json, every finding carries the AI's verdict next to it:

[critical] Hardcoded AWS access key id (settings.py:4)
    verdict=sound    confidence=98   suspected_fp=false   driver=hardcoded_secret
[high]     SQL execution dynamic string (store.py:14)
    verdict=sound    confidence=99   suspected_fp=false   driver=confirmed_by_review
[high]     SQLAlchemy raw SQL dynamic (store.py:14)
    verdict=sound    confidence=99   suspected_fp=false   driver=dynamic_string_sql
[high]     AWS access key ID (settings.py:4)
    verdict=sound    confidence=99   suspected_fp=false   driver=literal
[medium]   Weak hash: MD5 (store.py:21)
    verdict=refuted  confidence=96   suspected_fp=true    driver=unspecified_refutation
[high]     SQL execution dynamic string (ledger.py:13)
    verdict=refuted  confidence=99   suspected_fp=true    driver=constant_literal
[high]     SQLAlchemy raw SQL dynamic (ledger.py:13)
    verdict=refuted  confidence=98   suspected_fp=true    driver=constant_literal

Look at the driver column. For reports/ledger.py the model worked out on its own that the interpolated value is a constant_literal, a constant, not input. For store.py it's sound: real bug, left alone. And suspected_fp=true is the only thing that exempts a finding from the gate. The key detail: because I configured a different verifier (the log literally says verified by gpt-5.4), a finding only gets suspected_fp=true when both models refute it at ≥ 75. Drop the verifier and these exact three refutations still have to clear a second model to stand.

End result: you're left with 4 real problems to fix instead of 6-with-2-fake mixed in. And more importantly, not one real bug got dimmed.

"But what if the AI is wrong?"

Right question. The whole design is built around it:

• AI wrongly refutes a real bug? The verifier is a different model, prompted to push back, and it has to agree before the refutation counts. If the verifier disagrees, is unsure, or errors out, the finding still counts toward the gate (fail-safe). Even if both models are wrong, the finding is still sitting in the report with its verdict attached. You can see it, it didn't vanish.
• Model times out / endpoint dies? That critique becomes "uncertain" and the finding gates normally. Triage is best-effort and never fails the scan.
• Model tries to smuggle prose into the report? driver is forced into a closed token grammar, verdict has to be from a closed vocabulary, anything else is rejected. No LLM gets prose into the report path.

Put another way: the worst a bad verdict can do is exempt one finding from the exit code while it stays visible in the report. Not hide a bug.

Wiring it into CI

Nothing about the workflow from the last post changes. Add four environment variables to the scan step and you're done:

      - name: Synapse scan (SARIF + gate on high)
        env:
          SYNAPSE_FP_TRIAGE_ENABLED: "true"
          SYNAPSE_LLM_BASE_URL: ${{ secrets.LLM_BASE_URL }}
          SYNAPSE_LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
          SYNAPSE_FP_TRIAGE_MODEL: "your-proposer-model"
          SYNAPSE_VERIFIER_MODEL: "your-verifier-model"   # different from the proposer
        run: synapse-cli scan . --sarif --fail-on high > synapse.sarif

The gate stays as deterministic as before. The pile of false alarms just stops eroding the team's trust in it. And since nothing gets deleted, the SARIF you push to Code scanning is still complete; the false positives are only marked, not dropped.

One honest caveat: this is a noise filter, not a replacement for a human. It gets you looking at the 4 real problems instead of 6 muddled ones. You still have to fix those 4.

Come contribute

Synapse is open source (Apache-2.0): github.com/KKloudTarus/synapse-ce.

• Star it if the "AI proposes, Go and a second model confirm" direction is one you want to follow.
• Run it against your own LLM endpoint and tell me what happened: which model refutes well, which one likes to make things up. Feedback from a real model is the most useful kind.
• The verifier's adversarial prompt, the driver vocabulary, the consensus threshold: it all lives in internal/usecase/fptriage, and it's short and easy to read.

If you turn this on in your own repo, drop a comment with how many false positives the AI held back on the first run, and whether it ever came close to refuting something that turned out to be real. I'm curious.