mech.app

Posted on Sep 8 Originally published at mech.app

CVE Patch History as a Security Detector: How Executing Historical Fixes Reveals Agent Vulnerability Patterns

Public vulnerability databases document thousands of security flaws every year. Each CVE record includes the fixing commit: the exact diff that removed the vulnerability. That commit captures why the original code was unsafe, but it sits in a database designed for human inspection, not automated reuse. The same unsafe pattern can exist elsewhere in your codebase without triggering any alerts.

BUGSTONE-E2E treats CVE patch history as executable security tests. It mines fixing commits from 19,325 high-severity CVEs (2022 to 2026), extracts reusable detection rules, and runs them against target codebases. The system produces runtime evidence, not just static warnings. When it flags a potential vulnerability, it generates a patch and validates the fix with differential tests that prove the behavior changed.

This matters now because agent-generated code volume makes manual security review infeasible. If an agent writes a SQL query builder or a file upload handler, you need automated ways to catch known vulnerability patterns before deployment. Historical patches encode exactly what went wrong and how it was fixed. BUGSTONE-E2E turns that history into a detection pipeline.

Architecture: Funnel-Shaped Detection Pipeline

The system processes candidates in stages, applying progressively expensive analysis to a shrinking set of targets. Early stages handle thousands of call sites with lightweight tools. Later stages use LLM-based agents and runtime verification on the few candidates that survive.

Stage 1: Rule Mining

Parse fixing commits from CVE databases
Extract scan anchors (function names, API patterns)
Capture fix semantics (what changed and why)
Organize rules by CWE family and language
Result: 1,033 detection rules across 56 CWE families, packaged into 172 skills

Stage 2: Candidate Enumeration

Use Tree-sitter to find call sites matching rule anchors
No LLM calls at this stage
Output: large pool of potential matches

Stage 3: Lightweight Filtering

Apply heuristics to remove obviously benign sites
Check for sanitization patterns, type constraints, context clues
Still no LLM involvement
Output: reduced candidate set

Stage 4: LLM-Guided Inspection

Agent reads surviving candidates with rule context
Evaluates whether the unsafe condition from the CVE applies
Flags candidates that look vulnerable
Output: high-confidence findings

Stage 5: Runtime Verification

Generate scope-checked patches for flagged sites
Build two-sided differential tests: run original code and patched code
Confirm behavior actually changes
Output: runtime evidence, not just static suspicion

Across 14 programs, the pipeline produced runtime evidence for 644 findings. The funnel shape keeps costs manageable: you pay for LLM calls and test execution only on the small fraction of candidates that survive early filters.

Plumbing Details: From Commit Diff to Executable Rule

Extracting Rules from Patches

A fixing commit contains more than just a diff. It shows the vulnerable pattern (before) and the safe pattern (after). BUGSTONE-E2E parses both sides:

Anchor: the function or API call that appears in the vulnerable code
Context: surrounding code that helps identify when the pattern applies
Fix semantics: what changed (added input validation, switched to parameterized query, removed unsafe deserialization)

The system stores these as structured rules, not free-text descriptions. Each rule links back to its CVE and CWE classification.

Handling Version Mismatches

A patch might target library X version 1.2, but your codebase uses version 2.0. BUGSTONE-E2E does not attempt to transpile or rewrite rules across major version boundaries. Instead:

Rules include version constraints when extracting from CVE metadata
If the target codebase uses a different major version, the rule skips that candidate
For minor version differences, the system attempts matching but flags uncertainty in the output
Manual review queue captures skipped candidates for human triage

This avoids false positives from API changes between versions.

Tree-Sitter for Fast Enumeration

Tree-sitter parses source code into a syntax tree without invoking a full compiler. BUGSTONE-E2E uses it to find every call site matching a rule's anchor. For a rule targeting eval() calls in Python, Tree-sitter locates all eval nodes in seconds, even across millions of lines of code.

The output is a list of file paths, line numbers, and surrounding context. This feeds the next stage.

LLM Agent Inspection

The agent receives:

The candidate code snippet
The rule (including CVE description and fix semantics)
Surrounding context (up to 50 lines)

The agent's job is to answer: does this candidate exhibit the same unsafe condition as the original CVE? It checks for sanitization, type constraints, or other mitigations that might make the code safe despite matching the anchor.

The agent does not generate patches at this stage. It only flags candidates.

Two-Sided Differential Testing

For each flagged candidate, BUGSTONE-E2E generates a patch that applies the fix semantics from the rule. Then it builds two test harnesses:

Original harness: runs the vulnerable code path
Patched harness: runs the fixed code path

The system feeds both harnesses the same inputs (including malicious payloads from the CVE) and compares outputs. If behavior differs (e.g., the original crashes or leaks data, the patched version does not), that is runtime evidence of a real vulnerability.

This step catches false positives. If the differential test shows no behavior change, the candidate was likely safe to begin with.

Deployment Shape

CI/CD Integration

BUGSTONE-E2E runs as a security gate in pull request pipelines. When an agent (or human) submits code:

The pipeline extracts changed files
Tree-sitter enumerates call sites in those files
Lightweight filters run without blocking the PR
If candidates survive, the system queues LLM inspection and runtime tests
Results appear as PR comments with CVE links and evidence

For large codebases, you can run full scans nightly and incremental scans per-PR.

Rule Update Cadence

New CVEs publish daily. BUGSTONE-E2E includes a rule mining service that:

Polls NVD and GitHub Security Advisories
Extracts fixing commits
Parses diffs and generates new rules
Pushes updated rule packs to detection pipelines

You control the update frequency. Weekly updates balance freshness with stability.

Observability Hooks

The system emits structured logs at each pipeline stage:

Candidate counts after enumeration, filtering, inspection, and verification
LLM token usage and latency per inspection
Differential test pass/fail rates
False positive rates (when human review overrides a finding)

These metrics help tune filters and adjust LLM prompts.

Trade-Offs and Failure Modes

Dimension	BUGSTONE-E2E Approach	Trade-Off
Coverage	Only detects patterns with historical CVEs	Misses novel vulnerabilities with no prior record
Precision	Runtime verification reduces false positives	Adds latency and infrastructure cost (test harnesses)
Version Handling	Skips major version mismatches	May miss vulnerabilities in newer library versions
LLM Dependency	Agent inspection improves accuracy	Introduces nondeterminism and API rate limits
Rule Freshness	Automated mining from CVE feeds	Lag between CVE publication and rule availability

Failure Mode: Anchor Collision

If a rule anchors on a common function like open() or read(), enumeration produces thousands of candidates. Lightweight filters must be aggressive, or LLM costs explode. The system includes a heuristic budget: if a rule generates more than 500 candidates in a single codebase, it flags the rule for refinement (narrower anchor or additional context).

Failure Mode: Test Harness Generation

Differential testing requires building executable harnesses. If the candidate code depends on complex state (database connections, network services, environment variables), harness generation may fail. BUGSTONE-E2E falls back to static analysis for these cases, but loses runtime evidence.

Failure Mode: LLM Hallucination

The agent might flag a candidate as vulnerable when it is actually safe, or vice versa. Two mitigations:

Differential testing catches false positives (no behavior change means no vulnerability)
Human review queue surfaces low-confidence findings for manual triage

Code Snippet: Rule Matching with Tree-Sitter

import tree_sitter
from tree_sitter_languages import get_language, get_parser

def enumerate_candidates(source_code: str, rule_anchor: str, language: str):
    """
    Find all call sites matching the rule anchor using Tree-sitter.
    Returns list of (line_number, code_snippet) tuples.
    """
    parser = get_parser(language)
    tree = parser.parse(bytes(source_code, "utf8"))

    candidates = []

    def visit_node(node):
        # Match function calls by name
        if node.type == "call_expression":
            func_node = node.child_by_field_name("function")
            if func_node and func_node.text.decode("utf8") == rule_anchor:
                start_line = node.start_point[0]
                end_line = node.end_point[0]
                snippet = source_code.split("\n")[start_line:end_line + 1]
                candidates.append((start_line, "\n".join(snippet)))

        for child in node.children:
            visit_node(child)

    visit_node(tree.root_node)
    return candidates

# Example: find all eval() calls in Python code
source = """
def process_input(user_data):
    result = eval(user_data)  # Candidate 1
    return result

def safe_process(user_data):
    result = ast.literal_eval(user_data)  # Not a match
    return result
"""

matches = enumerate_candidates(source, "eval", "python")
# Output: [(3, '    result = eval(user_data)')]
# Note: line numbers are 0-indexed, so line 3 is the fourth line

This snippet shows the enumeration stage. Tree-sitter finds the eval call without executing the code or invoking an LLM. The next stage would apply heuristics (is user_data sanitized?) before passing to the agent.

When Historical Patches Miss the Mark

BUGSTONE-E2E excels at catching known patterns. It will not catch:

Zero-day vulnerabilities: no CVE means no rule
Logic bugs: if the vulnerability is a business logic flaw (e.g., missing authorization check), historical patches may not generalize
Configuration issues: misconfigurations in deployment manifests or environment variables fall outside the code-level detection scope

The system complements, rather than replaces, static analyzers and manual review. Use it as a first-pass filter for agent-generated code, then layer in other tools for broader coverage.

Technical Verdict

Use BUGSTONE-E2E when:

You generate code at scale (50+ agent PRs per week, templated services, auto-refactoring pipelines)
Your stack uses languages and libraries with rich CVE history (Python, JavaScript, Java, C/C++)
You can afford the infrastructure for differential testing (ephemeral test environments, CI/CD runner capacity with Docker or Kubernetes)
You want runtime evidence, not just static warnings
You need to audit agent-generated code without expanding your security team headcount

Avoid or defer when:

Your codebase is small enough for manual security review (under 10 PRs per week)
You work in a niche language with sparse CVE records (Elm, OCaml, newer languages without extensive vulnerability databases)
You lack CI/CD infrastructure to run test harnesses (no container orchestration, limited runner minutes)
You need coverage for novel vulnerabilities (use fuzzing or formal methods instead)
Your deployment pipeline cannot tolerate the latency of LLM calls and differential testing (sub-second merge requirements)

The core insight is simple: vulnerability history is executable. Every fixing commit is a test case waiting to run. BUGSTONE-E2E turns that history into a detection pipeline that scales with agent-generated code volume.

DEV Community