DEV Community: Anuththara Wickramasekara

ProofSec: Benchmarking Epistemic Robustness and Evidence-Grounded Vulnerability Reasoning in Frontier LLM

Anuththara Wickramasekara — Tue, 29 Sep 2026 11:03:28 +0000

This is a submission for the Kaggle Benchmarking Challenge

What happens when an LLM recognizes every lexical and semantic signature associated with a vulnerability - IDOR, BOLA, authorization bypass, predictable identifiers - but the available evidence does not actually establish that the vulnerability exists?

That question is the foundation of ProofSec, an evidence-centric security reasoning benchmark designed to evaluate whether LLMs can distinguish security indicators from substantiating evidence, reason under incomplete and contradictory observations, resist terminology and authority bias, incorporate falsifying evidence, and explicitly recognize when a security conclusion is not epistemically justified.

The Problem: Security Reasoning Is Not Pattern Matching

Large language models have become exceptionally capable at semantic retrieval.

Give a model:

GET /api/users/2841/invoices/9281

and introduce:

predictable numeric identifiers

and the model can immediately activate a large cybersecurity concept space:

IDOR
BOLA
Broken Access Control
Authorization Bypass

But there is a fundamental distinction between recognizing the semantic signature of a vulnerability and demonstrating that the underlying security property has actually been violated.

That distinction is where many security reasoning systems become unreliable.

Consider:

Authenticated user: 2841

GET /api/invoices/9281

HTTP/1.1 200 OK

{
    "invoice_id": 9281,
    "amount": 45000,
    "status": "paid"
}

Is this an IDOR?

The evidence is insufficient to establish that conclusion.

The observation does not independently establish:

who owns invoice 9281
whether the authenticated principal is authorized to access it
whether the object belongs to a different principal
whether an authorization invariant has been violated
whether access control is enforced upstream or downstream
whether the endpoint represents the protected resource
whether the response originated from production data
whether the response is synthetic, cached, fixture-generated, or otherwise non-authoritative

A predictable identifier is an indicator.

An HTTP 200 response is an observation.

Neither fact, in isolation, constitutes proof of unauthorized cross-principal access.

The central principle behind ProofSec therefore became:

A security indicator is not equivalent to security evidence.

This sounds straightforward.

In practice, it is a surprisingly difficult property for language models to maintain under adversarial framing, incomplete evidence, contradictory observations, and highly salient security terminology.

What I Benchmarked

ProofSec evaluates evidence-sensitive vulnerability classification.

Every scenario requires the model to classify the security state into exactly one canonical category:

Vulnerable
Not Vulnerable
Insufficient Evidence

The third state is deliberately first-class.

Traditional binary vulnerability classification implicitly encourages a forced decision:

YES
or
NO

Real security investigations rarely provide that luxury.

Evidence can be incomplete.

Observations can be ambiguous.

Telemetry can be contradictory.

Authorization boundaries can be unknown.

A response can be suspicious without being conclusive.

ProofSec therefore evaluates whether an LLM can recognize an epistemic boundary:

The available evidence is insufficient to establish the claim.

This changes the benchmark from a conventional vulnerability-recognition exercise into an evaluation of:

evidentiary sufficiency
causal relevance
authorization reasoning
uncertainty management
contradiction resolution
negative evidence integration
adversarial robustness
terminology robustness
controlled hypothesis revision
evidence-grounded classification

The benchmark is therefore not primarily asking:

"Does the model know what IDOR means?"

It is asking:

"Can the model determine whether the evidence presented actually establishes the security property in question?"

Benchmark Architecture

ProofSec v0.2 contains 110 security reasoning cases.

The corpus is deliberately constructed as an experimental evaluation set rather than an undifferentiated collection of cybersecurity questions.

The benchmark incorporates controlled evaluation families covering:

One-Fact Flip reasoning
Evidence ladders
Contradictory evidence
Negative evidence
Authority and terminology robustness
Baseline security reasoning

The architecture is designed around a core principle:

If the evidentiary state changes, the model should be sensitive to that change.

Conceptually:

                         ProofSec
                            │
             ┌──────────────┼──────────────┐
             │              │              │
        One-Fact Flip   Evidence State   Contradiction
             │              │              │
             └──────────────┼──────────────┘
                            │
                    Security Scenario
                            │
                            ▼
                      LLM Inference
                            │
                            ▼
                  Structured Assessment
                            │
             ┌──────────────┼──────────────┐
             │              │              │
       Classification   Evidence State   Explanation
             │
             ▼
       Deterministic Assertion
             │
             ▼
        Numerical Score

The implementation deliberately isolates:

Dataset
    ↓
Prompt Construction
    ↓
Model Inference
    ↓
Structured Parsing
    ↓
Assertion
    ↓
Scoring

This separation is not cosmetic.

It establishes distinct failure domains.

A dataset defect is not a model failure.

A packaging failure is not a reasoning failure.

A schema violation is not necessarily a classification failure.

A scoring-interface defect is not a model-performance measurement.

That distinction became one of the most important engineering principles in the project.

1. One-Fact Perturbation Testing

One of the core mechanisms in ProofSec is controlled one-fact perturbation.

Instead of constructing completely unrelated questions, ProofSec creates scenarios where a security-relevant fact changes while much of the surrounding semantic structure remains invariant.

For example:

Case A

Authenticated user = 2841
Invoice owner = 2841

Authorization rule:
invoice.owner_id == authenticated_user.id

Case B

Authenticated user = 2841
Invoice owner = 9127

Authorization rule:
invoice.owner_id == authenticated_user.id

The semantic surface can remain substantially similar.

The critical security relationship changes.

That gives us a controlled counterfactual:

Δ Input
   ↓
Δ Security-Relevant Fact
   ↓
Expected Δ Security State

This is fundamentally more informative than asking:

"Is this an IDOR?"

The actual evaluation question becomes:

Did the model detect the fact that causally changes the security state?

This introduces a form of counterfactual sensitivity testing.

If one authorization-relevant fact changes and the model preserves the same classification, the benchmark exposes a specific failure mode.

The model may understand the terminology.

It may understand the vulnerability class.

But it may not be correctly conditioning its conclusion on the evidence that actually determines the security property.

2. Evidence-State Modeling

ProofSec explicitly models the evidentiary state of a scenario.

The benchmark uses states including:

WEAK
PARTIAL
DECISIVE
CONTRADICTORY
NEGATIVE
UNKNOWN

This creates an evidence hierarchy rather than treating every security-relevant observation as equally probative.

A simplified progression is:

Predictable identifier
        │
        ▼
Weak security indicator
        │
        ▼
Ownership relationship established
        │
        ▼
Cross-principal access demonstrated
        │
        ▼
Authorization invariant violated
        │
        ▼
DECISIVE evidence

Consider the difference between:

Predictable ID

and:

Unauthorized cross-user object access

These observations possess radically different evidentiary weight.

The first may justify investigation.

The second can establish a concrete violation of an authorization property.

ProofSec therefore attempts to prevent a model from collapsing the following distinction:

Suspicion
    ≠
Evidence
    ≠
Proof

That distinction is foundational to trustworthy security analysis.

3. Contradictory Evidence

Real security investigations are not monotonic.

Evidence can conflict.

Telemetry can be stale.

Caches can contain misleading artifacts.

Fixtures can resemble production responses.

A preliminary observation can subsequently be invalidated by a higher-authority observation.

ProofSec therefore incorporates contradiction-resolution scenarios.

A simplified reasoning sequence might look like:

Initial observation
        │
        ▼
HTTP 200 response
        │
        ▼
Potential authorization anomaly
        │
        ├───────────────┐
        │               │
        ▼               ▼
Synthetic fixture   Actual endpoint
        │               │
        ▼               ▼
Cached response      HTTP 403
        │               │
        └───────┬───────┘
                ▼
       Re-evaluate hypothesis

The important capability is not simply detecting the first suspicious observation.

The model must determine whether later evidence changes the validity of the original hypothesis.

This introduces a critical distinction between:

Evidence accumulation

and:

Evidence revision

A model that merely accumulates confirming signals can behave very differently from one capable of hypothesis revision under contradictory evidence.

ProofSec deliberately tests that boundary.

4. Negative Evidence

Many vulnerability benchmarks are heavily oriented toward positive findings.

ProofSec deliberately gives negative evidence first-class status.

For example:

User A requests User B's object
                │
                ▼
Authorization rule verified
                │
                ▼
Cross-user request returns 403
                │
                ▼
Unauthorized access not demonstrated

A security reasoning system must be able to update its hypothesis in both directions.

It must identify:

Evidence supporting vulnerability

and:

Evidence weakening or falsifying vulnerability hypothesis

This matters because security investigation is fundamentally adversarial.

The objective is not to collect evidence that confirms the initial hypothesis.

The objective is to determine whether the hypothesis survives attempts to falsify it.

That makes negative evidence an important component of hypothesis discrimination.

5. Authority Bias and Terminology Robustness

Another failure mode targeted by ProofSec is authority and terminology bias.

Cybersecurity vocabulary carries enormous semantic weight.

Terms such as:

critical
confirmed
exploit
CVE
researcher
privilege escalation
authorization bypass
security issue

can exert disproportionate influence over LLM outputs.

ProofSec therefore evaluates whether the model remains anchored to technical evidence when the surrounding terminology changes.

The underlying principle is:

Terminology describes a claim. Evidence substantiates it.

Consider two descriptions:

Possible authorization issue

and:

Confirmed critical authorization vulnerability

If the underlying technical evidence is unchanged, the semantic framing should not arbitrarily alter the actual security state.

This probes linguistic framing sensitivity and authority-induced classification drift.

The benchmark is therefore not only testing cybersecurity knowledge.

It is testing whether that knowledge remains subordinate to the evidentiary record.

Structured Outputs

I intentionally avoided unconstrained free-form text as the primary evaluation interface.

Each model response is constrained into a structured schema containing:

classification
evidence_state
supporting_evidence
missing_evidence
safe_verification
impact

This creates a deterministic machine-readable boundary between inference and evaluation.

The primary scoring target is:

classification

The remaining fields provide diagnostic observability.

For example:

classification:
Vulnerable

could be accompanied by:

supporting_evidence:
"The identifier is sequential."

That exposes a potentially significant evidentiary defect:

Predictable Identifier
        ≠
Proven Unauthorized Access

The final classification alone cannot reveal whether the model arrived at the conclusion through valid or invalid reasoning.

Structured assessment therefore provides a richer diagnostic surface.

Deterministic Ground Truth

I deliberately avoided making another LLM the primary arbiter of the classification.

ProofSec uses canonical ground-truth labels and deterministic assertions.

The core evaluation chain is:

Scenario
   ↓
Model
   ↓
Structured Classification
   ↓
Canonical Ground Truth
   ↓
Exact Assertion
   ↓
Score

rather than:

Scenario
   ↓
Model A
   ↓
Model B judges Model A
   ↓
Score

For the core classification metric, deterministic evaluation establishes a cleaner experimental boundary.

It also reduces the possibility of evaluation contamination, where the judgment model introduces another layer of model-dependent interpretation into the primary metric.

Dataset Integrity

A benchmark is only scientifically useful if its evaluation corpus is reproducible.

ProofSec v0.2 uses a frozen dataset representation with a SHA-256 integrity digest:

422501a4db424c30c8ef24b61183351ec8a4bd2096e2671cf0e6bdf91e133a80

The project maintains a manifest and an integrity-verification workflow.

The intended reproducibility invariant is:

Same benchmark version
        +
Same task corpus
        +
Same evaluation protocol
        =
Comparable experiment

Without corpus integrity, two apparently identical experiments can silently evaluate different datasets.

That means dataset provenance, versioning, and integrity verification are part of the experimental methodology itself.

The Benchmark Itself Became an Experiment

One of the most valuable discoveries during the project was that benchmark engineering is itself part of evaluation science.

My initial task implementation used:

-> None

with assertions.

That produced pass/fail-style behavior.

I initially expected the benchmark to expose a numerical accuracy metric.

That assumption was incorrect.

I subsequently changed the task interface to:

-> float

and explicitly returned:

accuracy = correct / total
return float(accuracy)

This exposed a broader principle:

The semantics of the evaluation harness are part of the experimental design.

A benchmark can execute successfully while still reporting an invalid measurement if the scoring contract is incorrectly defined.

In other words:

Correct Inference
        ≠
Correct Evaluation

A trustworthy benchmark requires both.

The Kaggle Runtime Was Another Experimental Variable

One early published iteration failed with:

NameError: name 'ALL_TASKS_JSON' is not defined

This was not a model reasoning failure.

The model never reached the reasoning stage.

The task failed during execution before the first model inference step.

That forced a more rigorous separation of:

Dataset Integrity
        │
        ▼
Task Packaging
        │
        ▼
Runtime Execution
        │
        ▼
Model Inference
        │
        ▼
Structured Parsing
        │
        ▼
Assertion
        │
        ▼
Scoring

This distinction is fundamental.

A benchmark must not silently transform:

Runtime Error

into:

Model Error

Those are different failure classes with different remediation paths.

It also exposed an important deployment boundary.

My local development environment contained the canonical task source under:

C:\Users\User\Documents\ProofSec\kaggle\tasks

but a Kaggle-hosted execution environment cannot directly access that Windows filesystem.

The benchmark therefore has to package the necessary evaluation corpus and task implementation into the executable artifact available to the remote runtime.

That became an important lesson in benchmark portability and execution-environment isolation.

Versioned Benchmark Development

The benchmark went through multiple iterations while I validated the evaluation infrastructure.

One development version displayed assertion counts including:

Claude Sonnet 4.6    0 Pass
GPT-5.5             75 Pass
Gemini 3.5 Flash    95 Pass
Gemini 3.7 Flash    87 Pass

A subsequent controlled version successfully executed its smoke-test assertions across the evaluated models.

These development-stage measurements should not be interpreted as a universal model ranking.

They represent different stages of benchmark engineering.

During this process I was simultaneously validating:

dataset construction
task implementation
structured-output schema
assertion semantics
scoring semantics
remote runtime packaging
model inference

This separation is essential for responsible interpretation.

A benchmark-development artifact is not automatically equivalent to a final scientific measurement.

Local Validation

Before treating Kaggle execution as authoritative, I also used local evaluation to inspect the complete 110-case corpus.

Representative local validation produced:

Model	Correct	Total	Accuracy
Claude Sonnet 4.6	80	110	72.73%
Gemini 3.7 Flash	86	110	78.18%
Gemini 3.5 Flash	84	110	76.36%
GPT-5.5	67	110	60.91%

These figures are local validation measurements under the specific execution conditions used during development.

They should not be interpreted as immutable measurements of model capability.

LLM inference can vary because of:

model revisions
provider routing
stochastic inference
runtime configuration
API behavior
execution environment
infrastructure conditions

This variability is itself relevant to reproducibility.

The objective was therefore not to reduce the benchmark to:

"Model X is better."

The objective was to characterize failure modes, evidentiary behavior, and reasoning sensitivity under a controlled corpus.

A Particularly Interesting Failure Mode

One of the recurring failure patterns motivating ProofSec can be represented as:

Security Vocabulary Recognition
              ↓
       Semantic Activation
              ↓
     Premature Classification
              ↓
 Evidence Is Never Actually Tested

The desired reasoning pathway is:

Scenario
   ↓
Extract Security-Relevant Claims
   ↓
Identify Actors and Authorization Boundaries
   ↓
Identify Directly Observed Facts
   ↓
Separate Indicators from Decisive Evidence
   ↓
Evaluate Contradictory Evidence
   ↓
Evaluate Negative Evidence
   ↓
Determine Evidentiary Sufficiency
   ↓
Classify

The distinction is fundamental.

A model can possess extensive cybersecurity knowledge while still being unreliable at evidence-grounded adjudication.

This is the capability ProofSec attempts to isolate.

Why "Insufficient Evidence" Matters

This is one of the most consequential design decisions in ProofSec.

A conventional benchmark might ask:

Is the system vulnerable?

YES / NO

ProofSec instead represents:

Vulnerable
Not Vulnerable
Insufficient Evidence

This explicitly models epistemic uncertainty.

The model must distinguish between:

Observed

and:

Established

That distinction matters operationally.

Consider:

Unsupported Finding
        ↓
Analyst Investigation
        ↓
Engineering Interruption
        ↓
Potential Remediation
        ↓
Operational Cost

A system that generates large quantities of unsupported security findings can impose substantial downstream cost even when its vulnerability-recognition capability appears impressive.

Abstention is therefore not merely a failure to classify.

Under uncertainty, abstention can be a legitimate and necessary security behavior.

The Benchmark Is Essentially Testing Epistemic Discipline

At a deeper level, ProofSec is not simply a cybersecurity benchmark.

It evaluates whether an LLM can maintain an explicit boundary between:

What do I know?
        ↓
What does the evidence establish?
        ↓
What remains unknown?
        ↓
What conclusion is justified?

This is fundamentally an epistemic reasoning problem.

Cybersecurity provides a particularly concrete environment in which to measure it because security conclusions often have direct operational consequences.

The same methodology could potentially extend to:

code review
incident response
fraud detection
compliance analysis
threat intelligence
infrastructure diagnostics
financial anomaly investigation
legal document analysis
scientific reasoning

The security domain is simply where I chose to operationalize the problem.

Engineering Stack

ProofSec is implemented using:

Python
Kaggle Benchmarks
kaggle_benchmarks
Pydantic
structured model outputs
deterministic assertions
JSON-based frozen corpus
SHA-256 integrity verification
controlled perturbation generation
evidence-state annotations
local validation
Kaggle-hosted execution

The architecture maintains explicit boundaries between:

DATA

PROMPT CONSTRUCTION

MODEL INFERENCE

STRUCTURED PARSING

EVALUATION

SCORING

This makes the system substantially easier to:

debug
version
reproduce
audit
extend
attribute failures within

The benchmark is therefore treated as an engineered evaluation artifact rather than merely a collection of prompts.

Reproducibility and Development Workflow

The development workflow became:

Security Hypothesis
        ↓
Construct Controlled Scenario
        ↓
Define Canonical Ground Truth
        ↓
Generate Adversarial / Paired Variant
        ↓
Validate Dataset
        ↓
Freeze Corpus
        ↓
Generate Integrity Hash
        ↓
Implement Structured Task
        ↓
Run Local Validation
        ↓
Validate Kaggle Packaging
        ↓
Execute Against Multiple Models
        ↓
Inspect Assertions
        ↓
Analyze Failure Modes
        ↓
Revise Benchmark Infrastructure

This treats the benchmark as a versioned research artifact.

The project structure is maintained as an engineering repository containing the benchmark implementation, evaluation infrastructure, integrity controls, and supporting documentation.

The objective is to make the benchmark auditable rather than opaque.

What I Learned

1. Security reasoning is not vulnerability keyword matching

The presence of:

ID
endpoint
HTTP 200
user
invoice

does not automatically establish IDOR.

The authorization relationship is the security property that matters.

2. Counterfactual sensitivity is extremely valuable

If a single security-relevant fact changes, the model should respond to that change.

Controlled perturbation therefore provides a much stronger probe of reasoning sensitivity than simply increasing the number of unrelated benchmark questions.

3. Negative evidence deserves first-class treatment

A security reasoning system must be capable of recognizing evidence that weakens or falsifies its initial hypothesis.

4. Contradictions expose hypothesis-update behavior

A model that performs well under clean, monotonic evidence may behave very differently when observations conflict.

Contradiction handling is therefore a meaningful robustness dimension.

5. Abstention is measurable

"Insufficient Evidence" can be treated as an explicit epistemic state rather than an evasive response.

6. Structured outputs improve diagnosability

A classification tells us:

What did the model decide?

A structured assessment can additionally expose:

What evidence did it identify?
What evidence did it consider missing?
What verification did it propose?

That creates a substantially richer diagnostic surface.

7. The benchmark harness must be engineered like software

Dataset defects, packaging failures, schema violations, runtime exceptions, parsing failures, and model reasoning failures require separate failure categories.

8. Reproducibility is part of validity

A benchmark without corpus integrity, version control, and execution controls can silently drift.

A score without provenance is considerably less useful than a score attached to a reproducible experimental configuration.

What I Want to Measure Next

ProofSec v0.2 focuses on controlled evidence reasoning.

The next iteration can extend the methodology considerably.

Temporal Reasoning

Instead of presenting all evidence simultaneously:

T0 → Initial Telemetry
T1 → Additional Observation
T2 → Contradictory Artifact
T3 → Authorization Test

I want to evaluate whether the model updates its hypothesis correctly as evidence arrives.

This transforms static classification into a sequential evidence-update problem.

Confidence Calibration

Instead of evaluating only:

classification

evaluate:

classification
+
confidence

Then measure whether confidence is statistically aligned with empirical correctness.

A model that is wrong with extremely high confidence represents a fundamentally different operational risk profile from a model that identifies uncertainty.

Evidence Attribution

Require the model to answer:

Which exact fact changed your classification?

This enables a deeper evaluation:

Correct Conclusion
        +
Correct Evidentiary Basis

versus:

Correct Conclusion
        +
Incorrect Evidentiary Basis

This distinction matters because a model can occasionally arrive at the correct answer for the wrong reasons.

Cross-Domain Generalization

The framework can be extended into:

SSRF
Authentication Bypass
Privilege Escalation
CSRF
Path Traversal
Race Conditions
Business Logic Flaws
Cloud IAM
API Authorization
Multi-Tenant Isolation

The broader research question becomes:

Is evidence-grounded security reasoning a generalizable capability, or is it primarily a collection of domain-specific semantic associations?

The Bigger Question

When I started building ProofSec, the obvious question was:

Which LLM is better at cybersecurity?

After engineering the benchmark, that no longer seemed like the most interesting question.

The more fundamental question is:

Can we construct evaluations that determine whether an AI security conclusion is actually justified by the evidence available to it?

A leaderboard such as:

Model A - 82%
Model B - 79%
Model C - 76%

is useful.

But it is incomplete.

For an AI security system, I want to know:

What evidence did the model use?

What evidence did it ignore?

Did it infer facts that were never provided?

Did it confuse an indicator with proof?

Did it recognize negative evidence?

Did it resolve contradictions?

Did terminology alter its classification?

Did it respond to a one-fact perturbation?

Did it revise its hypothesis?

Did it recognize when the evidence was insufficient?

Those questions are much closer to the requirements of a real security-analysis system.

My Benchmark

ProofSec - Security Evidence Reasoning Benchmark

Open ProofSec on Kaggle

ProofSec v0.2 contains 110 security reasoning cases designed around controlled evidence manipulation, structured classification, deterministic ground truth, integrity verification, and reproducible multi-model evaluation.

The benchmark is built around one principle:

Security conclusions should be proportional to the evidence supporting them.

ProofSec operationalizes that principle through:

controlled one-fact perturbations
evidence-state modeling
contradiction resolution
negative evidence
authority and terminology robustness
structured outputs
deterministic assertions
frozen corpus integrity
SHA-256 verification
cross-model execution

Open-Source Implementation

The benchmark implementation and supporting engineering work are available on GitHub:

ProofSec GitHub Repository

The repository is intended to make the benchmark more than a Kaggle submission.

It provides an engineering artifact around the evaluation methodology, allowing the benchmark to be inspected, versioned, reproduced, and extended independently of the Kaggle interface.

The broader objective is to preserve the chain:

Research Hypothesis
        ↓
Dataset Construction
        ↓
Ground Truth
        ↓
Evaluation Harness
        ↓
Integrity Verification
        ↓
Model Execution
        ↓
Results
        ↓
Failure Analysis

That provenance matters.

A benchmark should not only report a number.

It should make it possible to understand where that number came from.

Final Takeaway

The most important result from building ProofSec was not a single model score.

It was discovering how easily an evaluation can appear to measure security reasoning while actually measuring something else.

A benchmark can accidentally measure:

Keyword Recognition

instead of:

Evidence Reasoning

It can measure:

Runtime Correctness

instead of:

Model Correctness

It can measure:

Confidence

instead of:

Evidence-Justified Certainty

It can measure:

Semantic Familiarity

instead of:

Causal Sensitivity

And it can report a precise-looking number while the underlying dataset, runtime, scoring semantics, or evaluation protocol is not actually controlled.

ProofSec is my attempt to make those boundaries explicit.

The objective is not simply to build an LLM that identifies the largest number of possible vulnerabilities.

The objective is to construct an evaluation framework capable of answering a substantially harder question:

When an AI claims that a security vulnerability exists, did the available evidence actually justify that claim?

That is the standard I want AI-assisted security tooling to eventually be held to.

Not:

Did the model recognize the vulnerability vocabulary?

But:

Did the model establish the security claim from the evidence?

That distinction is the entire premise of ProofSec.

Resources

ProofSec Benchmark

ProofSec on Kaggle

ProofSec Source Repository

ProofSec on GitHub

Kaggle Benchmarking Challenge

Kaggle Benchmarking Challenge on DEV

kagglechallenge #kaggle #ai #machinelearning #llm #cybersecurity #securityresearch #benchmarking #airesearch #python #softwareengineering #llmsecurity #applicationsecurity #securityengineering #aisafety #airesearch #llmreasoning #securityautomation #researchengineering

Weaponizing a CVSS 10.0 PoC: Building a Zero-Touch RSC Recon Engine with GitHub Copilot

Anuththara Wickramasekara — Thu, 24 Sep 2026 09:34:22 +0000

This is a submission for the MLH x DEV Writing Challenge

Every security researcher has a "Project Graveyard" - a dusty local directory where dormant Proof of Concepts (PoCs), 2 AM exploit scripts, and ephemeral bypasses go to die.

For me, that script was a raw, 50-line Python snippet authored in December 2025. A CVSS 10.0 vulnerability (CVE-2025-55182 / React2Shell) had just dropped, exposing a critical unauthenticated Remote Code Execution (RCE) attack vector via unsafe deserialization of the React Flight protocol.

My script successfully fingerprinted the vulnerability, but its architecture was archaic: synchronous, entirely ephemeral, and reliant on manual DevTools injection to scan a target DOM. It was a functional PoC, but terrible for actual field reconnaissance.

When the MLH Finish-Up-a-Thon launched, I decided to pull this script out of the archive and operationalize it. Armed with GitHub Copilot as my autonomous platform engineer, I spent 48 hours transforming a brittle terminal script into a highly resilient, event-driven, Manifest V3 Chrome Extension.

Here is how we turned digital duct tape into a production-grade vulnerability reconnaissance asset.

What I Built: The RSC Reconnaissance Engine

I engineered the RSC Fingerprint Detector, a Manifest V3 (MV3) Chrome extension designed for passive and active telemetry gathering on React Server Components (RSC) and the React2Shell deserialization vulnerability class.

To eliminate false positives during target enumeration, the extension deploys a strict, dual-phase heuristic pipeline:

Passive Reconnaissance (Zero-Touch): Silently hooks the global execution context (window.__next_f), performs script asset introspection (react-server-dom-webpack), utilizes DOM attribute scanning (data-rsc), and monitors network traffic for specific HTTP response headers (Vary: RSC).
Active Probing: Fires a cross-origin, asynchronous fetch with a crafted X-RSC-Probe header payload. It then parses the resulting content-type entropy to extract the exact React Flight protocol signature without triggering target backend alerts.

The Architecture Upgrade: We migrated from a synchronous, single-vector Python script that lost its execution context on page reload, to a declarative MV3 architecture. The new engine features IndexedDB ACID transactions for persistent telemetry, cross-context Inter-Process Communication (IPC), and DOM isolation via Shadow DOM to prevent CSS cross-contamination.

Telemetry & Visual Proof

GitHub Repository: anuththara2007-W/CVE-2025-55182-Exploit-extension

There is nothing quite as satisfying as the visual proof of operationalizing a raw exploit script into a deployable tool.

December 2025 (The Ephemeral PoC)	May 2026 (The Operationalized Asset)

Raw console execution. Highly manual. Zero state persistence.	Isolated UI, real-time threat badges, and active CVE mapping.

Partner Technologies: GitHub Copilot as a Cyber-Operations Ally

Writing the core heuristic logic for vulnerability detection is the fun part. Wrestling with Chrome’s restrictive Content Security Policies (CSP), isolated execution worlds, and MV3 lifecycle deprecations is the grueling friction that kills side projects.

Without GitHub Copilot, this asset would still be a dormant script. I didn't just use it for autocomplete; I utilized it as a senior platform architect who understood the Chrome Sandbox limitations intimately. Here is how Copilot acted as an elite force multiplier:

1. Bypassing the Manifest V3 Sandbox Constraints
Background pages are dead in MV3. Copilot flawlessly orchestrated my manifest.json schema, registering the Service Worker and migrating my network interception logic to declarativeNetRequest instead of the blocking webRequest API. It completely bypassed the dreaded "persistent flag" runtime errors that plague MV3 migrations.

2. Bridging the Asynchronous IPC Gap
When setting up chrome.runtime.sendMessage across isolated execution contexts (content script to background worker), Copilot automatically injected the mandatory return true directive inside the onMessage listener.

The Technical Impact: In MV3, failing to return true immediately terminates the IPC message channel, causing silent failures for asynchronous callbacks. Copilot patched this race condition before the code even compiled.

3. Architecting Persistent Telemetry
I needed persistent local storage for scan logs but wanted to avoid bloated wrapper libraries. Copilot generated a raw IndexedDB implementation from scratch, engineering the onupgradeneeded handler, object store schemas, and timestamp indexing. It proactively wrapped the database transactions in try/catch blocks, implementing idempotency and retry logic for transient database lock errors.

The "Mic-Drop" Network Interception

The most impressive moment occurred while configuring the passive packet-sniffing logic. I wrote a single comment:
// Capture RSC responses via webRequest

Copilot instantly synthesized this interception block:

chrome.webRequest.onHeadersReceived.addListener(
  function(details) {
    const isRSC = details.responseHeaders.some(h => 
      h.name.toLowerCase() === 'content-type' && h.value.includes('text/x-component')
    );
    if (isRSC) {
      saveFlight(details); // Trigger telemetry extraction
    }
  },
  { urls: ["<all_urls>"] },
  ["responseHeaders"] // <-- Copilot successfully identified this mandatory MV3 bypass!
);

Injecting "responseHeaders" into the extraInfoSpec array is an obscure, critical MV3 security requirement. If omitted, Chrome silently sanitizes the headers from your extension's execution context. Copilot's deep contextual awareness of Chrome's security model saved me hours of debugging silent telemetry drops.

Hackathon Experience

Participating as a Solo Developer in the MLH Finish-Up-a-Thon completely shifted my perspective on software development in the security space.

Standard hackathons prioritize reckless velocity-getting a prototype to compile by any means necessary. This event prioritized craftsmanship and operational readiness. It forced me to analyze the "last mile" of software engineering: error boundaries, state persistence, UI reactivity, and adhering to strict platform security models.

We often abandon security research tools because the friction of engineering a deployable interface outweighs the adrenaline of writing the initial exploit logic. GitHub Copilot eradicated that friction. It handled the exhausting platform boilerplate, allowing me to focus 100% of my cognitive load on the vulnerability architecture.

My biggest takeaway? A dead PoC doesn't have to stay in the graveyard. With the right AI tooling and a structured deadline, you can resurrect a forgotten script and weaponize it into a professional-grade asset.

Disclaimer: This reconnaissance tool is intended strictly for authorized vulnerability scanning, threat intelligence, and educational purposes. Use only on infrastructure you own or have explicit, documented authorization to test.

CVE-2026-20127 Defensive Companion

Anuththara Wickramasekara — Sun, 12 Jul 2026 04:29:34 +0000

This is a submission for Weekend Challenge: Passion Edition

What I Built

CVE‑2026‑20127 Defensive Companion - a fully offline, enterprise‑grade browser extension (Manifest V3) that transforms raw exploit intelligence into a real‑time threat analysis laboratory. It provides deep‑dive visibility into the authentication bypass vulnerability in Cisco Catalyst SD‑WAN Controllers (vSmart), enabling SOC teams and defensive researchers to dissect the DTLS handshake flaw, extract Indicators of Compromise, and deploy mitigation strategies without ever exposing operational data to external networks.

The extension delivers:

An offline NLP knowledge engine that answers natural‑language queries about the vulnerability, detection, and remediation - powered entirely by 17 segmented JSON intelligence domains parsed from the original exploit research.
An interactive Protocol Explorer that renders the legitimate vs. exploited DTLS handshake sequences, highlighting the exact memory corruption in vbond_proc_challenge_ack_ack() at 0x38AB7.
A Packet Anatomy Viewer breaking down the forged CHALLENGE_ACK_ACK message byte‑by‑byte.
A report generator that compiles on‑demand Markdown threat summaries for downstream distribution.
Strict air‑gapped design: zero external API calls, rigid Content Security Policy, and the minimal possible permissions (activeTab, sidePanel, storage).

The goal was to encapsulate months of passion for reverse engineering and defensive research into a single, elegant tool that makes sophisticated protocol analysis accessible to every security professional - anywhere, with no cloud dependency.

Demo

The extension’s side panel is the persistent analyst workspace. Below is a capture of the dashboard showing the interactive protocol explorer, local assistant, and IOC extraction.

Visual walk‑through: left pane - protocol sequence diagram with the vulnerable state transition highlighted; right pane - offline NLP assistant answering a detection query; bottom bar - quick IOC summary (malicious verify_status byte, targeted ports, anomalous log patterns).

Code

anuththara2007-W / CVE-2026-20127-Exploit-Extension

Executive Summary

The CVE-2026-20127 Defensive Companion is a premier, enterprise-grade Browser Extension engineered for proactive threat intelligence and defensive analysis. This tool encapsulates authoritative technical research concerning the authentication bypass vulnerability in the Cisco Catalyst SD-WAN Controller (vSmart). By delivering an offline, heavily localized intelligence layer, it empowers Security Operations Centers (SOC) and defensive researchers to rapidly analyze packet structures, trace authentication flows, and instantly extract Indicators of Compromise (IOCs) without exposing active investigative data to external networks.

Technical Background

The Cisco Catalyst SD-WAN architecture relies on the vdaemon service to establish and maintain control-plane DTLS connections between edge devices and the controller. The underlying vulnerability (CVE-2026-20127) exploits a failure in state machine validation during the DTLS handshake sequence, explicitly impacting ports typically associated with vSmart control traffic (e.g., 12346, 52521). The vulnerability allows unauthorized actors to bypass X.509 certificate validation and forge an authenticated state within the internal controller…

View on GitHub

How I Built It

The Spark

Cisco’s disclosure of CVE‑2026‑20127 - an unauthenticated DTLS handshake bypass in the vSmart controller’s vdaemon service - immediately ignited my passion. A single missing state‑machine check in the vbond_proc_challenge_ack_ack() function allowed an attacker to forge an authenticated session without ever presenting a valid certificate. The original exploit repository laid bare the raw mechanics; I wanted to turn that knowledge into a defensive force multiplier that would outlive the patch cycle.

This challenge provided the perfect forcing function. In one weekend, I built an extension that doesn’t just describe the vulnerability - it teaches the protocol weakness, arms defenders with live detection rules, and does so in a completely offline package that respects the operational security of the most sensitive environments.

Architecture: Isolated Intelligence, Maximum Impact

The extension adheres to the strict separation enforced by Manifest V3, while delivering a reactive, near‑instant experience.

Data Layer (src/knowledge)
The raw exploit research was parsed into 17 domain‑specific JSON objects covering everything from the DTLS state machine (message types 8,9,10) to IOC fingerprints in vsyslog and auth.log. This structured intelligence is the foundation of the offline engine.

Offline NLP Engine (src/services)
A custom, zero‑latency pipeline runs entirely in the browser:

Tokenization & stop‑word removal on the user’s natural‑language query.
Heuristic intent detection - mapping phrases like “how to detect” to the Detection knowledge domain.
Semantic retrieval against the pre‑built inverted index (a Set<string> intersection over tokens).
Template‑driven synthesis that assembles the matched JSON fragments into human‑readable Markdown, complete with memory addresses, log signatures, and ACL recommendations.

All indexing happens synchronously at side‑panel load, keeping query response times sub‑millisecond.

Presentation Layer (React 19 + Vite + Tailwind)
The UI is built as a single‑page application mounted in the sidePanel. Key design choices:

Lazy‑loaded components ensure instant first paint; the Protocol Explorer and Packet Viewer are loaded only when their tabs are activated.
Framer Motion hardware‑accelerates the DTLS sequence animations, making the state‑transition attack path intuitively visible.
Lucide React provides a professional icon set that scales across Chrome’s light/dark modes.

Browser Integration

A service worker (background.ts) acts as the central event bus, routing messages between the popup, side panel, and storage layers with mandatory return true for async chrome.runtime.onMessage listeners.
chrome.storage.local persists user preferences and the most recent report snapshot.
The manifest requests only activeTab, sidePanel, and storage - no webRequest, no declarativeNetRequest. All intelligence is self‑contained; the tool makes zero network requests, guaranteeing that no sensitive investigation data leaks.

Technical Deep Dive: Why the Vulnerability Exists

The extension’s Protocol Explorer visualises the core flaw. In the legitimate flow:

Client sends DTLS ClientHello with X.509 cert.
Server issues CHALLENGE (msg_type=8).
Client responds CHALLENGE_ACK (msg_type=9) with cryptographic proof.
Server validates proof, sends CHALLENGE_ACK_ACK (msg_type=10), and sets *(BYTE*)(peer_state+70) = 1.

The exploit skips step 3 entirely. The attacker intercepts the CHALLENGE and immediately crafts a CHALLENGE_ACK_ACK with a non‑zero verify_status byte. Because vbond_proc_msg() exempts msg_type=10 from authentication checks, the packet reaches the vulnerable handler, which blindly trusts the client‑supplied status and marks the peer as authenticated.

The extension renders these two flows side‑by‑side, highlighting the missing validation gate, and overlays the exact binary patch (upgrade to v20.12.6.1) that introduces the state‑machine enforcement.

Detection & Mitigation Built In

Beyond visualisation, the Knowledge Engine codifies actionable detection guidance directly from the exploit research:

vsyslog anomaly: Alert on control-connection-state-change with peer-vmanage-system-ip of 0.0.0.0 followed by immediate down state.
auth.log anomaly: Unexpected SSH key acceptance for vmanage-admin on port 830 immediately after DTLS instability.
Network ACLs: Strictly restrict inbound DTLS ports (12346, 52521) to known edge routers.
Patch imperative: Upgrade to v20.12.6.1+, which adds the missing verify_status validation.

Every answer is synthesized from the local JSON store; the assistant is not a generative AI that hallucinates. It’s a deterministic retrieval engine that guarantees factual accuracy.

Passion in Every Component

This wasn’t a casual build. The extension reflects a deep obsession with protocol‑level security research. I manually disassembled the vdaemon logic, traced the memory writes, and translated the binary analysis into interactive UI elements. The NLP engine itself was hand‑crafted - no TensorFlow, no external models - because I wanted the intellectual satisfaction of building a pure, dependency‑free reasoning core that respects the air‑gap.

The project is a love letter to the art of defense: taking an attacker’s blueprint and forging a shield that educates and protects simultaneously. The World Cup of cybersecurity is happening every day in data centers and SOCs, and this tool is my seat in the stands, cheering for the blue team.

From Zero to Manifest V3: How GitHub Copilot Helped Me Finish an RSC Vulnerability Detector for CVE‑2025‑55182

Anuththara Wickramasekara — Sun, 31 May 2026 14:20:47 +0000

This is a submission for the GitHub Finish-Up-A-Thon Challenge

From Zero to Manifest V3: How GitHub Copilot Helped Me Finish an RSC Vulnerability Detector for CVE-2025-55182

What I Built

RSC Fingerprint Detector - a Manifest V3 Chrome extension that performs passive and active reconnaissance of React Server Components (RSC) and the React2Shell vulnerability class (CVE-2025-55182, CVE-2025-66478). These vulnerabilities stem from unsafe deserialisation of the React Flight protocol, enabling unauthenticated remote code execution on any Next.js App Router endpoint.

The extension implements a dual-phase detection pipeline:

Passive fingerprinting: Heuristic analysis of the global namespace (window.__next_f), script asset introspection (react-server-dom-webpack), DOM attribute scanning (data-rsc, data-reactroot), and response header inspection (Content-Type: text/x-component, Vary: RSC).
Active probing: A cross-origin fetch with a custom X-RSC-Probe header, followed by content-type entropy analysis and Flight protocol signature extraction.

The original project was a 50-line ephemeral script that relied on manual DevTools injection. The finished version is a fully isolated, event-driven Chrome extension with a declarative ruleset, persistent IndexedDB storage, and a reactive UI - all built with GitHub Copilot as the primary force multiplier.

Demo

Repository: https://github.com/anuththara2007-W/CVE-2025-55182-Exploit-extension

Visual proof of completion arc:

Before (abandoned console script)	After (production extension)
Only raw console output, no UI, single detection vector	Modern popup, status badges, active probe results, CVE mapping

Before screenshot: https://github.com/anuththara2007-W/CVE-2025-55182-Exploit-extension/raw/main/images/img1.png
After screenshot: https://i.postimg.cc/wT2vX4xH/Annotation-2026-05-31-184616.png

The Comeback Story

December 2025 - The CVE Disclosure

When CVE-2025-55182 was publicly disclosed (CVSS 10.0, unauthenticated RCE via Flight protocol deserialisation), I wrote a minimal Python script that checked for window.__next_f. It worked, but it was a single-vector, synchronous, non-persistent tool. Each scan required manual injection of the script into the target page's console. The script was abandoned after 48 hours because the effort to productionise it (manifest, permission handling, cross-context messaging, error recovery) exceeded the perceived value.

May 2026 - The Revival

The Finish-Up-A-Thon provided a forcing function. Using GitHub Copilot as a context-aware pair programmer, I transformed the script into a production extension with the following architectural improvements:

Component	Original (Dec 2025)	Finished (May 2026)	Technical Depth
Detection vectors	1 (`window.__next_f`)	4 (globals, script asset fingerprinting, DOM attributes, response headers)	Multi-heuristic fusion reduces false negatives
Active fingerprinting	None	`fetch` with `X-RSC-Probe`, content-type parsing, Flight protocol signature detection	AbortController timeout, CORS-aware, idempotent retry
Execution context	Console injection	Isolated content script + background service worker	Cross-context message passing with `chrome.runtime.sendMessage` and mandatory `return true` for async response
State persistence	None (ephemeral)	IndexedDB schema (object stores: `detections`, `flights`) with ACID transactions	Schema versioning, index creation (`timestamp`), FIFO eviction
UI/UX	Raw terminal	Web-based popup with reactive event handlers and status badges	CSS Grid/flex, card design, shadow DOM isolation
Distribution	Local script	Unpacked Chrome extension with `manifest.json` (MV3)	`declarativeNetRequest` permission, `host_permissions` for `<all_urls>`

The completion arc is not just feature addition - it is a migration from an ad-hoc, single-tenant script to a reusable, multi-tenant security utility that respects Chrome's extension security model (CSP, isolated worlds, least privilege).

My Experience with GitHub Copilot - Technical Breakdown

Without Copilot, this extension would still be a forgotten folder. Copilot acted as a semi-autonomous engineering partner, handling the following high-complexity tasks:

Technical Area	Specific Copilot Contribution	Why a Human (or I) Would Have Struggled
Manifest V3 permissions	Generated the entire `manifest.json` with `activeTab`, `scripting`, `declarativeNetRequest`, `host_permissions`, and the correct `service_worker` registration	MV3 deprecated background pages. Copilot knew the new schema and avoided the `persistent` flag error.
Cross-context message passing	Wrote `chrome.runtime.onMessage` listeners with the required `return true` to keep the message channel open for asynchronous `sendResponse`	I would have forgotten `return true`, causing silent failures in the popup.
IndexedDB schema design	Produced the `initDB` function with `onupgradeneeded` handler, object store creation, and index definitions (`timestamp`)	I had never written IndexedDB without a wrapper. Copilot generated transaction boundaries and error recovery.
Declarative DOM scanning	Suggested `document.querySelectorAll('[data-rsc], [data-reactroot], #__next')` and the fallback to `window.__next_f`	I did not know all RSC markers. Copilot extracted them from its training corpus of React codebases.
Active probe fetch with AbortController	Generated a `fetch` with a 2-second timeout, custom headers, and response header extraction, including CORS-aware error handling	Writing a timeout with `AbortController` and cleaning up the abort listener is error-prone. Copilot did it correctly.
Popup UI reactivity	Built the entire HTML/CSS card layout, plus the `addEventListener` wiring and state synchronisation with `chrome.storage.local`	I am not a frontend developer. Copilot produced a modern, accessible UI that works across Chrome versions.
Error recovery and idempotency	Added `try/catch` blocks to every async operation and implemented retry logic for transient IndexedDB lock errors	Without this, the extension would freeze on first error. Copilot made it resilient.

The single most impressive Copilot moment:

When I started typing // Capture RSC responses via webRequest, Copilot auto-completed the entire chrome.webRequest.onHeadersReceived listener, including the filtering condition (details.responseHeaders.some(h => h.name.toLowerCase() === 'content-type' && h.value.includes('text/x-component'))) and the asynchronous saveFlight call. It even added a comment explaining that MV3 requires 'responseHeaders' in the extraInfoSpec array. That single completion saved me 90 minutes of reading Chrome's migration guide.

Copilot did not write the whole extension. I architected the detection heuristics, chose the CVE-2025-55182 mapping, and performed manual testing on live targets (e.g., nextjs.org, vercel.com, local Next.js sandboxes). But Copilot handled the boilerplate, the edge cases, and the Chrome-specific quirks - turning a 50-line script into a 500-line production extension in two evenings.

Technical Deep Dive - How Copilot Solved Specific Problems

Asynchronous idempotency in message handlers

Copilot correctly added return true inside chrome.runtime.onMessage to indicate that the response will be sent asynchronously. Without this, the popup would time out.
IndexedDB versioning and schema migration

Copilot generated the onupgradeneeded block with conditional object store creation, preventing ConstraintError on subsequent extension updates.
Closure serialisation for injected scripts

When injecting the detection function via chrome.scripting.executeScript, Copilot used func instead of code, avoiding CSP violations and maintaining lexical scope.
DeclarativeNetRequest vs WebRequest

Copilot recommended using declarativeNetRequest for passive header inspection, which is more performant and aligns with MV3's shift away from blocking webRequest.
Heuristic entropy reduction

Copilot suggested combining multiple low-certainty signals (e.g., data-rsc + #__next + __next_f) into a confidence score, reducing false positives.

Without Copilot, this extension would not exist. With Copilot, it is a robust, field-ready security tool.

Team submission: Solo developer - Anuththara Wickramasekara

Disclaimer: This tool is intended for authorised security testing and educational purposes only. Use only on systems you own or have explicit permission to test. The author assumes no liability for misuse.