DEV Community: unit life

100% vuln detection wasn't enough: measuring whether AI respects the patch

unit life — Thu, 24 Sep 2026 13:18:19 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

AI models are often like over-eager alarms. Show them a dangerous word in code — eval, system(, a raw SQL concat — and they scream “vulnerability!” nearly every time. Add the lock one line up, and many cheaper models still scream: they recognized the scary token, they did not read the fix.

That failure mode is expensive. Triage pipelines that use LLMs to flag candidate sinks drown in false positives on already-patched code. Public coding evals ask “did you find a bug?” They rarely ask “did you respect the fix?”

I built ART — Attacker-Reachable Sink Triage to measure that gap.

How it works: minimal-pair twins. Same function name, same identifiers, same shape — only the security control differs. Prompts get snippet + language only. Twin ids, gold labels, and rationales never enter the model context.

Here is a real pair from the set. The only difference is the fix — everything a token-matcher keys on ($_GET["id"], SELECT, the function name) is identical:

// twin_sql_php · gold = reachable_vuln
function process_user_data($conn) {
    $id = $_GET["id"];
    $sql = "SELECT * FROM users WHERE id = " . $id;   // attacker-controlled concat
    return mysqli_query($conn, $sql);
}

// twin_sql_php · gold = patched  (same shape, one control added)
function process_user_data($conn) {
    $id = (int)$_GET["id"];
    $stmt = mysqli_prepare($conn, "SELECT * FROM users WHERE id = ?");
    mysqli_stmt_bind_param($stmt, "i", $id);          // cast + prepared statement
    mysqli_stmt_execute($stmt);
    return mysqli_stmt_get_result($stmt);
}

A model that labels the second snippet reachable_vuln isn't a worse detector — it's a worse patch reader. Twin Gap captures exactly that: vuln accuracy − patched accuracy. Zero means the model respects fixes; positive means it over-flags patched code.

Core suite

Task	What it measures	Score
`art-label-triage`	4-way label: `reachable_vuln` / `patched` / `safe` / `vacuous_noise`	ART = 0.4·vuln + 0.4·patched + 0.2·filler
`art-overconfidence-trap`	On patched twins only: “is there a confirmed exploit right now?” (gold = no)	Fraction not overclaimed
`art-proof-marker-poc`	Emit a minimal lab PoC containing `ART_PROOF_OK`	1.0 / 0.0

Dataset: 8 twin pairs (SQLi, XSS, auth bypass, command injection, path traversal, LFI, insecure deserialization — PHP + Python) plus 6 safe/vacuous controls. N is intentionally small: one miss moves Twin Gap by 12.5%. This is a diagnostic probe, not a large-N ranking claim.

Why synthetic, not raw CVEs: so models cannot win by memorizing a write-up, and so each twin differs by one control. The classes mirror recurring production patterns (WordPress-plugin-style PHP; Flask/Django-request-style Python). Failing a patched twin here is meant to map to over-flagging a real fix in those ecosystems.

Ablations (personas + forced CoT) are public supporting tasks; the headline metric is label-triage.

Models Tested

Seven locked Community Benchmark models, chosen for tier × price × family coverage — not a single SOTA chase:

Model	Why it’s in the lineup
`gemini-3.5-flash` / `gemini-3.7-flash`	Fast Gemini tier
`gemini-2.5-pro`	Does cost buy patch-respect?
`claude-haiku-4-5-20251001` / `claude-sonnet-4-5-20250929`	Cheap vs mid Claude
`gemma-4-31b-it`	Open-weights instruct
`gpt-5.4-nano-2026-03-17`	Price floor

Roughly 50× cost span per full triage run (~$0.004 → ~$0.18). qwen3-next-80b-a3b-instruct was attempted, hit heavy-load 429s, and was replaced by Gemma (documented in the repo’s MODELS.md).

Findings

TL;DR

Every model found every vulnerable twin (100% raw). That alone is useless — an alarm that never stops ringing doesn’t help. The useful question is whether the model calms down after the lock.

Four models (three Gemini + Gemma) score perfect ART: bugs and fixes. Gemma does it for ~$0.007/run.
Cheaper models still catch every bug but keep flagging fixed code (Haiku Twin Gap 0.375 — 3 of 8 patched twins). Nano is cheapest but confuses harmless filler for risk.
For builders: rank on patch reading, not hype — or your triage queue fills with already-fixed findings.

Numbers from art-label-triage v6 (adjudicated gold + production scoring):

Model	ART	Raw	Patched	Controls	Twin Gap	Cost USD	Latency
`gemini-2.5-pro`	1.000	1.000	1.000	1.000	0.000	0.181	7.9s
`gemini-3.5-flash`	1.000	1.000	1.000	1.000	0.000	0.108	2.9s
`gemini-3.7-flash`	1.000	1.000	1.000	1.000	0.000	0.028	9.1s
`gemma-4-31b-it`	1.000	1.000	1.000	1.000	0.000	0.007	12.3s
`claude-sonnet-4-5-20250929`	0.950	1.000	0.875	1.000	0.125	0.060	3.1s
`claude-haiku-4-5-20251001`	0.850	1.000	0.625	1.000	0.375	0.020	1.7s
`gpt-5.4-nano-2026-03-17`	0.817	1.000	0.875	0.333	0.125	0.004	1.3s

Cost: Gemma and Gemini flash match pro-tier ART at roughly 1–4% of the cost. Heavy Pro does not beat Flash or Gemma on this probe.

What I’d actually ship

Constraint	Pick	Why
Open weights / on-prem	`gemma-4-31b-it`	ART 1.000 at ~$0.007
Latency-sensitive	`gemini-3.5-flash`	ART 1.000, ~2.9s
Claude-family stacks	Sonnet + a patch-respect check	Strong explanations, non-zero overclaim

What surprised me (more than any leaderboard cell)

1. The models corrected our gold.

All seven disagreed with two labels — in the same direction — and they were right. An escaped-input “safe” filler was really patched by our own prompt definition; a deser twin that replaced pickle.loads with json.loads was really safe. Those two items capped every model at ART 0.917 and invented our largest “failure” class. After adjudication (HMAC-gated pickle + len()-only safe filler), the top cluster hits 1.000 and remaining errors are patch misses under the frozen, adjudicated rubric.

That redesign is the point: a ceiling isn’t always model competence — sometimes it’s your key.

2. A 0.0 that wasn’t a capability.

Sonnet’s proof-marker stayed 0.0 across retries because the provider returned an empty completion (86 prompt tokens, empty message). Read the transcript before ranking a model on a single-shot cell.

3. Personas and CoT didn’t “fix” patch-respect.

Red-team persona did not systematically inflate overclaim. Forced data-flow CoT on the trap did not close Haiku’s gap (0.625 → 0.50).

Two concrete Haiku misses

Path twin: claims basename("../../../etc/passwd") still traverses — it doesn’t (ignored sanitizer).
Auth twin: admits current_user_can works, then still labels reachable_vuln by shifting to a different risk (overclaim).

Honest limits

N = 8 pairs + 6 controls. Haiku’s 3/8 patched misses → exact sign-test p = 0.25 at this N. Small on purpose; one miss is loud.
Kaggle collection pages often show Pass/100 for Score floats. The ranked number is each run’s rewards.score (and the table above) — not the collection chart.

What I’d measure next

Sandbox-execute the proof-marker (assert on printed output, not source text).
Multi-file / multi-hop taint.
Larger N once the gold-audit loop stays routine.

My Benchmark

Kaggle collection (required): Attacker-Reachable Sink Triage (ART)

Source (MIT): https://github.com/mziqudhd92/kaggle-art-benchmark

Core tasks

kaggle b t run art-label-triage -m gemini-3.5-flash --wait

Reproducibility: the dataset (dataset/items.jsonl) is frozen and versioned in the repo; gold labels are deterministic and scored by param_id, not answer order. scripts/validate_jsonl.py and scripts/test_scoring_alignment.py gate every push, and scripts/analyze_results.py regenerates the tables and charts above from the downloaded run artifacts. Every number here traces to a specific task version (label-triage v6).

Safety: synthetic snippets only; defensive triage research; no live targeting.

Credit: inspired by proof-over-speculation tooling (Iridium); this entry is a standalone Kaggle Community Benchmark under MIT.

Fingerprinting Network Honeypots with Weighted Behavioral Scoring Engine

unit life — Thu, 03 Sep 2026 11:30:33 +0000

Deception technology has evolved past static string matches.
Modern decoys try to mimic production environments,
making binary "is it a honeypot?" checks unreliable.
Single indicators—like a missing Date header or an unusual SSH banner—frequently trigger false positives on enterprise middleboxes and legacy servers.

To solve this, we built Honeypot-Auditor around a dual-dimensional Honeyscore & Confidence engine. Here is a breakdown of how the scoring mechanics work under the hood.

Github: https://github.com/mziqudhd92/honeypot-auditor

The Math Behind the Honeyscore (0–100%)

Rather than assigning flat point values, Honeypot-Auditor treats each protocol anomaly as a weighted indicator with strict corroboration rules:

Isolated Indicator Penalty: Standard L4/TLS stack tells (e.g., JA3S signatures or header ordering) carry low independent weight (~10–15%).
Corroboration Multiplier: Weak tells require corroboration across independent categories (e.g., combining a TLS cipher mismatch with a state-machine failure). When two distinct categories hit, a corroboration gate unlocks the full indicator weight.
Hard Tells: High-interaction leaks (such as arbitrary auth acceptance or shell execution latency anomalies) act as high-confidence anchors that push the score above 80%.

Under the Hood: Corroboration Gating

Here is a simplified look at how the analyzer evaluates indicator weights and suppresses weak signals unless corroborated by an independent category:

def calculate_honeyscore(indicators: list[Indicator], proxy_detected: bool) -> float:
    score = 0.0
    categories_hit = {ind.category for ind in indicators if ind.triggered}

    for ind in indicators:
        if not ind.triggered:
            continue

        # Proxy Guard: Suppress L4/TLS stack tells if an edge proxy is active
        if proxy_detected and ind.fingerprint_type in PROXY_SUPPRESSED_TYPES:
            ind.suppressed = True
            continue

        # Corroboration Gate: Weak tells require at least 2 distinct categories
        if ind.requires_corroboration and len(categories_hit) < 2:
            ind.suppressed = True
            continue

        score += ind.weight

    return min(score, 100.0)

Cool Web Page: https://mziqudhd92.github.io/honeypot-auditor/

Dual-Dimensional Output: Score vs. Confidence

A high score alone isn't enough for automated decision-making. We pair the Honeyscore with a separate Confidence metric:

LOW Confidence: Triggered when < 3 protocols are audited or > 50% of probes fail/timeout.
MEDIUM Confidence: Reached when 3+ protocols respond and at least 2 distinct category hits occur.
HIGH Confidence: Achieved when deep behavioral mode confirms corroborating tells across multiple independent layers.

Real-World Scenario: The Banner Spoofing Trap

Consider a modern SSH decoy (like Cowrie,dd-honeypot and others) configured to mimic a standard production Linux server:

a. Naive Banner Grabber:

Connects to port 22 and reads the string: SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1.
Concludes: "Standard production server (0% Honeypot)." b. Honeypot-Auditor:
Banner Check: Sees OpenSSH claim. (Weight: 0%)
KEXINIT Inspection: Extracts the raw SSH_MSG_KEXINIT byte sequence.
It detects Paramiko/Twisted-specific key exchange algorithms and rigid cipher preferences that real OpenSSH binaries never advertise.
Stack Cross-Referencing: Checks the TCP SYN-ACK option ordering. The underlying kernel exhibits a generic container profile, contradicting the OS claimed by the banner.
Corroboration Gate Unlocked: Combining the application-layer banner claim with the KEXINIT algorithm mismatch triggers a high-confidence indicator.
Result: Honeyscore leaps to 85.0% [HIGH LIKELIHOOD DECOY] with Tactical Action SKIP_TARGET.
And this is by checking only port 22, usually we will ffind more open ports on honeypots and combining checks on them will increase the detection even further.

Tactical Action Outcomes

Instead of forcing engineers to interpret raw percentages, the scoring engine resolves into four tactical actions:

SKIP_TARGET: Score >= 60% with HIGH/MEDIUM confidence (confirmed decoy).
PIVOT_POSSIBLE: Score < 30% with HIGH confidence (verified production target).
PROCEED_CAUTION: Score between 30–59% or LOW confidence.
INCONCLUSIVE: Edge proxy masking origin stack or insufficient probe responses.

So what is different ?
The main thing is that we are not only counting on signatures to detect a target, the engine using 16 protocols that are implementing more than 50 different strategies to evaluate if remote host is a decoy or not.

You can try it out, Honeypot-Auditor is open-source (MIT licensed:

pip install honeypot-auditor
honeypot-auditor --target 127.0.0.1 -v

mziqudhd92 / honeypot-auditor

Does This Look Like An Honeypot? (DTLLAH) Multi-protocol CLI that fingerprints whether a target IP behaves like a low-interaction honeypot — Shodan Honeyscore, active auth/state probes, and a weighted score.

.______________________________________________________________________________
|  :: H-AUDITOR :: v0.7.3 :: "DIALING IN... CARRIER DETECTED" ::                |
|------------------------------------------------------------------------------|
|  "warez? nah. headers. we trade banners, not bins."                          |
|  "if it answers any password, it ain't production — it's a lure."            |
|  "respect the sysop. probe only what you own. leave no STOR behind."         |
|______________________________________________________________________________|

Site (BBS / NFO): https://mziqudhd92.github.io/honeypot-auditor/
Agents / AEO: llms.txt · agents.md

  ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
  █  >>> LIVE DEMO · 3 HOST LAB TOUR · -v / --deep / SILENT-ACCEPT <<<     █
  ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

  "three hosts, three lenses: KEX facade with -v, deep on the buffet,
   silent-accept on the tarpit. same fingerprinter — different tells."
                                              — lab tour · authorized only

.------------------------------------------------------------------------------
|  NFO · READ BEFORE YOU DIAL                                                  |
|------------------------------------------------------------------------------|
|  Authorized targets ONLY. Lab boxes. Decoys you own. Sensors you run.        |
|  Permission on paper (or in ticket).                                         |
|                                                                              |
|  Scanning random /16 because Shodan said

…

View on GitHub

Would love feedback!