✓ Human-authored analysis; AI used for formatting and proofreading.
Harry Wetherald (Maze) makes a case against auto-fixing vulnerabilities: most findings are false positives, every code or cloud change carries risk. So auto-fixing everything means introducing risk for zero security benefit. His proposed path: first improve understanding (AI investigates findings in context), then generate fix suggestions for humans, then eventually automate the fixes once they're reliable.
The problem identification is correct. The solution addresses the wrong layer.
What he gets right
The false positive rate is real. The answers range from 80% to 99%. Most findings don't matter when investigated in the context of the specific environment.
Auto-fixing false positives is worse than ignoring them. Every change to cloud or code carries risk such as downtime, new attack surface, regression. If 90% of findings are false positives, auto-fixing all of them means 90% of changes are risk for zero benefit.
AI-generated fix suggestions without context are not helpful. If the finding lacks organizational context, the fix suggestion will also lack the context. You get a wall of auto-fixes to review instead of a wall of findings to review. The shape of the problem changes; the size doesn't.
These observations are accurate.
Where the solution misses
The proposed progression: better AI investigation → fix suggestions → human review → eventually full automation.
This path accepts the scanner's output a finding, a signal as the starting point and tries to improve what happens after. It never questions the output itself.
The false positive problem is two problems
Wetherald's "80-99% false positives" combines two different things:
The first is false positives. The scanner is wrong about the configuration. The bucket does have versioning. The role doesn't have that permission. The scanner missed something. The fix for this is straightforward: fix the scanner. Better rules, parsing and coverage. This is an engineering problem with engineering solutions.
The second is true findings that aren't exploitable. The configuration is as the scanner says. The bucket really does lack versioning. But the context makes exploitation impossible: the bucket is empty, internal-only, behind a VPC endpoint policy, and protected by an SCP. The finding is true. The risk is zero.
These require different solutions. The first needs better detection. The second needs something else entirely.
Context is not an AI problem
The context that makes a true finding non-exploitable is the configuration graph. The relationships between resources, the organizational policies, the permission boundaries, the network topology. This context is static. It's deterministic. It's already in the cloud provider's API responses.
Wetherald proposes AI that "investigates findings in the context of your environment." But the context he describes is not ambiguous. The SCP either blocks the action or it doesn't. The VPC endpoint policy either restricts the path or it doesn't. The permission boundary either limits the role or it doesn't.
These are not judgment calls requiring AI investigation. They are predicates evaluable against a configuration snapshot. The answer is true or false. Same snapshot, predicate and answer, every time.
The reason scanners produce findings without this context is that scanners evaluate individual resources, not the relationships between them. A scanner checks: "does this bucket have versioning?" That's a property of one resource. Whether that bucket's lack of versioning matters depends on what writes to it, who can access it, what policies protect it, and whether it participates in an attack path. Those are relationships — edges in a graph, not properties of nodes.
The false positive is an architecture failure. The tool evaluates nodes. The risk lives in edges.
The output type is the root cause
A scanner produces a finding — "this bucket lacks versioning." That's a signal. It requires human interpretation before a machine can act on it. Is this finding real? Does it matter? Should we fix it? Those questions exist because the output is a signal.
Consider a different output: a verdict. "This configuration is NON_COMPLIANT with respect to control X, evaluated against predicate Y, from snapshot Z. Exit code 3." That's a decision. A pipeline reads the exit code. No interpretation, triage or "is this real?" question.
The false positive problem Wetherald describes is a property of signals, not a property of security evaluation. Signals require investigation because they're incomplete. They describe one resource without the graph. Verdicts don't require investigation because they evaluate the graph and produce a deterministic result.
Better AI investigating signals is still AI interpreting signals. The output is still probabilistic — "I'm 95% confident this matters." That's still a signal. A faster, smarter signal is still a signal.
The missing step: verification
Wetherald's path is: find → investigate → suggest fix → human executes fix. The fix happens. Then what?
How does the defender know the fix worked? How does the auto-fixer know the fix resolved the vulnerability rather than introducing a different one?
The fix-verification loop is absent from his progression. Without verification, "auto-fix" means "auto-change and hope." With verification:
- Finding identified (the verdict says NON_COMPLIANT)
- Fix applied (by human or automation)
- Re-evaluate (capture new snapshot, re-run evaluation)
- Verdict changes (NON_COMPLIANT → COMPLIANT, or it doesn't)
- If it doesn't change: the fix was wrong. Try again.
Verification makes auto-fix safe. Not better AI investigation or fix suggestions. Verification. A deterministic verdict engine that can confirm the forbidden state is gone after the fix is applied.
Self-healing is the wrong endpoint
Wetherald's vision: AI agents find, investigate, and fix without human involvement. Self-healing environments.
This assumes the bottleneck is human labor. Too many findings for humans to review, so automate the humans away.
The actual bottleneck is signal quality. If 90% of findings are false positives, the problem isn't that humans are slow. It's that the tool is producing 9 units of noise for every 1 unit of signal. Faster humans (or AI replacing humans) processing the same noise doesn't solve the problem. It processes noise faster.
The alternative is changing the output so the question "is this real?" doesn't arise:
A deterministic verdict doesn't need investigation. The predicate either matches the configuration or it doesn't.
An exploitability classification (exploitable / one change away / reachable only) doesn't need triage. The graph evaluation already identified which findings have every precondition present for an attack.
A fix-verification loop doesn't need hope. The verdict engine confirms the forbidden state is gone.
None of these require AI. They require a different tool architecture: one that evaluates the configuration graph deterministically, produces verdicts instead of signals, and can verify that a fix resolved the finding.
The layer error
Wetherald is building a better consumer of signals. That's useful if your tools produce signals, you need something to interpret them, and AI interpretation is better than human interpretation at scale.
But building a better signal consumer doesn't fix the fact that the signal is the wrong output type. It's optimizing the layer above the problem rather than fixing the layer where the problem lives.
The layer where the problem lives is the tool's primary function. A scanner that evaluates nodes will always produce findings that lack graph context. No amount of post-hoc AI investigation changes what the scanner evaluated. The investigation is reconstructing the graph that the scanner should have evaluated in the first place — expensively, probabilistically, differently each time.
A tool that evaluates the graph from the start doesn't produce the false positive to investigate. The "80-99% false positive rate" is a consequence of evaluating nodes. Evaluate edges and the rate changes.
The right path
Wetherald's path: find (signal) → investigate (AI) → suggest fix → human review → auto-fix → self-healing.
The alternative: evaluate (verdict) → classify (exploitable / one-away / reachable) → fix → verify (re-evaluate, verdict changes) → automate verified fixes.
The difference:
No investigation step. The verdict is the investigation. The predicate evaluated the graph and produced a deterministic result.
No triage step. The exploitability classification is the triage. Exploitable findings first. One-away findings next (protect the precondition before it flips). Reachable findings last.
Verification built in. Every fix is re-evaluated. The verdict either changes (fix worked) or it doesn't (fix was wrong). No "auto-fix and hope."
Auto-fix becomes safe when the fix is verifiable. Not when AI gets better at investigation. The deterministic verdict tells the auto-fixer what needs to change, and the re-evaluation confirms the change worked.
This path doesn't require AI to improve. It requires the tool to produce a different output. The AI era's contribution to security isn't better investigation of signals. It's recognizing that investigation is a workaround for the wrong output type.
Stave produces verdicts, not signals. Deterministic evaluation of configuration snapshots against a control catalog. Exploitability classification on every finding. Fix verification via snapshot diff. The false positives Wetherald describes don't arise, because they evaluate the graph. stave apply --observations ./your-snapshot/
Top comments (0)