A security merge gate is only as good as its false positive rate

A hard gate on merge is the right instinct. Security findings are cheapest when they interrupt a change before it becomes shared state, before other branches build on it, before it ships. Blocking the merge is the correct place for that class of control.

But there is a failure mode that is worse than having no gate at all: a gate that blocks too eagerly. The first time it flags something that is obviously fine, people start looking for the bypass flag. The second time, they add it to their muscle memory. After that the gate is theater, and the one real finding it catches next quarter gets waved through with everything else.

So the interesting engineering problem is not "can we block a merge on a security finding." That part is easy. The problem is staying accurate enough that nobody wants to route around you.

Here is how we think about it on Synapse.

The gate blocked a real pull request until an embedded-credential git URL was rejected, then let it through.

The gate has to be explainable

A finding that blocks a merge carries four things by construction:

• what failed (a rule id and a CWE),
• why it matters (a short rationale),
• the evidence it used,
• and the exact change that makes it pass (a compliant example).

For a dependency finding the evidence is the scanner and the advisory database version. For a static analysis finding it is the matched pattern or the data flow, at a specific file and line. The evidence is hash chained, so a finding cannot be quietly edited after the fact and still pass the report gate.

The goal is that a developer never has to guess why the gate fired or argue with a black box. If the gate cannot tell you what to change, it has not earned the right to block you.

Precision is a release gate for the rules, not just for your code

This is the part most tools get wrong, and it is the part we care about most.

Our deterministic detectors (dependency scanning, pattern static analysis, taint, secrets, misconfiguration) publish findings with no model in the path. Same input, same finding, every time. There is no AI triage step softening the output. That is a feature, because it makes the gate reproducible and auditable. It is also a constraint, because it means a false positive prone rule is not a nuisance. It is a bug in the tool.

A concrete example. We were reviewing a community pull request that added a rule pack for Jupyter and IPython notebooks. Fifteen new rules, clean build, all tests green. Two of the rules looked completely reasonable on paper:

1. One flagged <script> markup saved in a notebook's output cell. Reasonable sounding, since script markup in output is an XSS smell.
2. One flagged local filesystem paths like /home/alice/... saved in a traceback, since those can disclose environment structure.

Both would have been a disaster on real notebooks.

Plotly, Bokeh, ipywidgets, and Altair all embed <script> tags in their normal chart output. Every interactive plot in every data science notebook would have tripped the first rule. And every saved Python exception contains site-packages frames under a home directory, so the second rule would fire on essentially any notebook that ever caught an error.

Neither rule is wrong in theory. Both are wrong in practice, because they cry wolf on ordinary code. So we blocked the merge until they were tightened:

• The script rule now requires a real injection signal in the markup, like an event handler attribute, document.write(, or eval(, not just the presence of a <script> tag.
• The traceback rule now excludes site-packages, dist-packages, and virtualenv frames, and fires only on a genuine project path.

The rule count stayed the same. The false positive rate went to near zero, with regression tests to keep it there.

The bar we hold is simple: if a rule fires on ordinary, safe code, it does not ship. Precision is a gate the rules have to pass before they are allowed to gate anyone else.

AI generated code and the shared blind spot

A growing share of pull requests are agent generated now, and they read clean at a glance. The dangerous case is not the code that looks sloppy. It is the code where the same agent also wrote the tests, so the suite inherits the exact blind spot the implementation has. Everything is green, and the vulnerable pattern sails through.

This is why the gate is pattern and data flow based, and deliberately independent of the test suite rather than derived from it. If the tests missed it because they shared the author's assumptions, the gate still does not. A gate that trusts the tests is only as good as whoever wrote the tests, which increasingly is the same model that wrote the code.

Let the model propose, do not let it decide

We do use models. We just do not let them block a merge on their own opinion.

Any analytical claim, whether it is reachability, a risk narrative, or a suspected exploit, is a proposal. It cannot gate anything until a separate, deterministic verifier seals a verdict over a threshold. A confident but wrong model output never becomes a merge blocker, because it never gets to be the one deciding.

For dependency findings, reachability and taint analysis decide whether a vulnerable symbol is actually reachable in your code before the finding escalates. "It exists somewhere in the tree" is not automatically a hard stop. That alone removes a large slice of the noise that makes teams resent dependency scanners.

The gate stays green because it is boring

None of this works if the checks are flaky. The detectors are plain, deterministic Go: build with and without CGO, vet, and a fast test suite that runs on every change. Reproducible checks are what let you trust a red result enough to block on it.

Deterministic checks: same input, same result, fast enough to run on every pull request.

The quiet part

Catching real issues is table stakes. Any scanner can produce findings. The actual engineering work is in the negative space: not firing on the Plotly chart, not firing on the site-packages traceback, not blocking a merge over a vulnerable function nobody calls.

A gate earns trust by being right, being specific about the fix, and being quiet when there is nothing wrong. Get that part right and people stop trying to route around it, because there is nothing to route around. Get it wrong and it does not matter how good your detection is, because the gate everyone silences is the same as no gate at all.

---

Come contribute

Synapse is open source (Apache-2.0): github.com/KKloudTarus/synapse-ce.

• Star it if the "AI proposes, Go and a second model confirm" direction is one you want to follow.
• Run it against your own LLM endpoint and tell me what happened: which model refutes well, which one likes to make things up. Feedback from a real model is the most useful kind.
• The verifier's adversarial prompt, the driver vocabulary, the consensus threshold: it all lives in internal/usecase/fptriage, and it's short and easy to read.

If you turn this on in your own repo, drop a comment with how many false positives the AI held back on the first run, and whether it ever came close to refuting something that turned out to be real. I'm curious.