DEV Community: Sushant Poudel

Google's OKF cut LLM hallucination 81% on DFIR telemetry — because the format, not the model, was the problem

Sushant Poudel — Fri, 13 Nov 2026 14:00:59 +0000

DFIR tooling still hands its telemetry to LLMs as one big JSON dump. At scale that is a context-exhaustion engine: latency climbs, the model starts hallucinating, and the analyst has to re-verify everything it said.

Our new paper asks a narrow question: does the shape of the serialization matter more than the model?

We benchmarked Google's Open Knowledge Format (OKF v0.1) — a graph-native, selectively traversable format — against flat JSON, using 7,680 inference cycles across five open-source model architectures with escalating decoy densities (four difficulty tiers).

Results, relative to JSON:

Hallucination suppressed by 81.1% at the hardest tier
Token overhead cut by 33.6%
Noise-invariant recall sustained at 0.94–0.99 as decoys increased

The useful part is not "OKF is better". It's the mechanism: when the model can traverse to the relevant subgraph instead of ingesting a monolithic dump, context stays small and grounded — so the failure mode that produces hallucination never gets loaded in the first place.

If your incident-response agent ingests JSON telemetry, this is a format decision you can test before you touch the model.

Paper: https://sushantpoudel2028.com.np/publications/okf-dfir

More writing, including the negative results: https://sushantpoudel2028.com.np/writing

security # ai # dfir # llm

Your router can read your keyboard: 85% keystroke recovery from Wi-Fi, and the firmware fix that cut it to 9.8%

Sushant Poudel — Fri, 13 Nov 2026 12:29:12 +0000

Your Wi-Fi router can read your keyboard. Not by breaking encryption — by measuring the air.

Wi-Fi was designed to move data. Paired with modern deep learning it has quietly become something else: a sensing fabric that detects motion, infers behaviour, and reconstructs physical activity from signal reflections alone, on hardware cheaper than a home router. Enterprise and IoT security was never built for that pairing.

Our paper measures how bad it is, in realistic interference-prone environments, using a bidirectional LSTM trained to correlate Channel State Information (CSI) sequences with keystroke signatures.

A passive adversary — commodity hardware, no network authentication, no physical access, not one injected packet — reaches F₁ 85.4% in line-of-sight office conditions. Through a standard 15 cm drywall partition it still reaches F₁ 61.2%. For context, a ten-class random guess floor is 10%. That is over six times chance, through a wall, passively.

Then we tried to kill it at the source. RF-Guard is a deterministic, stateful CSI obfuscation framework at the firmware level. Across three testbeds it cut the observer's accuracy to 9.8% — below the random-guess floor.

Two takeaways:

If you deploy Wi-Fi for connectivity only, you are also deploying a sensor, and your threat model does not include a passive RF observer.
The mitigation that worked was not "detect the attack". It was making the signal uninformative at the point it is radiated.

Presented at IC-AICSE 2026 (abstract book p. 33).

Paper and details: https://sushantpoudel2028.com.np/publications/wifi-sensing

More writing, negative results kept in: https://sushantpoudel2028.com.np/writing

security # ai # iot # privacy

A 3.8B model on a laptop caught 202 of 208 bad agent actions — and the safe-looking variant was the unsafe one

Sushant Poudel — Fri, 13 Nov 2026 12:07:23 +0000

Our paper measured something that inverted the assumption we started with: making an AI agent's safety checker output JSON made it less safe, not more.

The setup. An autonomous agent that executes actions rather than proposing them sits outside role-based access control — RBAC authenticates an identity, but says nothing about whether a specific action should happen. Routing every proposal to a cloud frontier model closes that gap semantically, but adds round-trip latency and is often prohibited outright in regulated settings.

So we built an edge-native semantic firewall: Phi-3-mini, 3.8B parameters, 4-bit quantized, held under a 4.2 GiB VRAM ceiling, sitting in the evaluator stage of a Generator–Evaluator pipeline on a single consumer laptop. No cloud calls at any stage.

The mechanism is a structured Chain-of-Thought JSON schema that forces the evaluator to name the governing policy rule and justify the match before it may emit a decision. That converts an opaque verdict into a trace an operator can audit.

We evaluated it on a 600-scenario corpus: 155 adversarial scenarios across eight prompt-injection techniques, 60 compound multi-step scenarios, and 30 boundary cases — 1,800 generations, all run on the model itself at temperature 0.

The result that matters: constraining the output format without requiring the reasoning step produced the least safe evaluator of the three conditions. Requiring the reasoning cut hard-denial approvals from 71 to 6, and 6 of 208 got through.

The takeaway is narrow but useful: if you are bolting a model onto an agent as a policy check, the JSON schema is not the safety property. The forced justification is. A checker that can say "deny" quickly without showing its work is not a checker; it is a second opinion with the same blind spots.

Published in ICICSET Vol. 3, No. 1, pp. 228–235 (2 October 2026) — DOI 10.65091/icicset.v3i1.75.

Paper, code, corpus and raw outputs: https://sushantpoudel2028.com.np/publications/agentic-verification

More of the negative results, kept in: https://sushantpoudel2028.com.np/writing

ai # security # llm # showdev