DEV Community: Aditya Tiwari

I built Klarion, a secret scanner that asks an LLM before it alerts. Here's how it did against 4 others on 61k files

Aditya Tiwari — Sun, 04 Oct 2026 17:46:30 +0000

Secret scanners have a noise problem. Point one at a big, healthy codebase and you get hundreds of alerts: test fixtures, docs examples, long identifiers that happen to look random. People stop reading them, and then the real leak slips through.

That gets worse with AI coding agents, which write files faster than anyone reviews them. So I built Klarion, a scanner that asks an LLM before it raises an alert, and then benchmarked it against the usual tools to see whether that actually helps.

The test

Five scanners, four large open-source repos in four languages, all pinned to a commit:

spring-boot (Java), terraform (Go/HCL), next.js (TypeScript), symfony (PHP)
61,514 files in total
Control set: flask and rails (5,054 files)

Versions: klarion 0.4.3 (with Claude Haiku), gitleaks 8.30.1, trufflehog 3.97.5, detect-secrets 1.5.0, ripsecrets 0.1.11.

These repos have no live credentials, so every alert is something a human would have to check and throw away.

Tool	Alerts on the 4 repos	Per 1k files	Control (flask + rails)
Klarion + model	11	0.18	1
ripsecrets	299	4.86	61
trufflehog	344	5.59	9
gitleaks	361	5.87	33
detect-secrets	12,073	196.26	247

That's 33x fewer alerts than gitleaks and over 1,000x fewer than detect-secrets on the same code. Each of Klarion's 11 comes with the model's verdict and a one-line reason, so checking them takes minutes, not an afternoon.

Why entropy scanners blow up on typed languages

Entropy-based detection, including Klarion's own first stage, raises 3–4x more alerts per file on Java, Go, TypeScript and PHP than on Python and Ruby: trufflehog jumps 3.1x and detect-secrets 4.0x. gitleaks and ripsecrets, which lean on rules, stayed flat.

The reason: Java, Go, TypeScript and PHP are full of long identifiers like SseCustomerKeySHA256AttrName that look high-entropy. Entropy alone can't tell them apart from a credential. Reading the surrounding code can, and that's what the LLM step does. After it, Klarion's rate is 0.18 per 1k files on the typed repos and 0.20 on the control, so the language effect is gone.

Does it miss real secrets?

Fewer alerts are useless if real leaks go missing, so I also used Samsung's CredData: labelled lines from 337 real repositories.

On code outside test folders:

	Recall	Precision
Klarion + model	0.35	0.89
gitleaks 8.30	0.21	0.91
detect-secrets 1.5	0.35	0.34

That's about 1.7x the real secrets gitleaks finds, at similar precision.

How it works

Detect: keyword gate, 81 regex rules, normalized Rényi entropy.
Judge: an LLM reads each candidate with its surrounding code and returns secret / false positive / uncertain, with a reason.

It needs a model: Anthropic, OpenAI-compatible, a local Ollama model (nothing leaves your machine), or your Claude Code login.

Where it runs

Claude Code: a plugin hook blocks the write before the file exists (file edits and Bash)
Cursor, Cline or any MCP agent: through its MCP server
CI: a GitHub Action that scans only what a PR adds; GitLab CI works too
Git hooks: klarion protect or the pre-commit framework
Locally: klarion scan .

Try it, and tell me where it's wrong

Free and open source (MIT): https://github.com/0x1Adi/Klarion

The full method, pinned commits and every finding are in benchmark/REPORT.md.

Chatlectify: turn your chat history into a writing style your LLM can reuse

Aditya Tiwari — Mon, 20 Apr 2026 06:21:56 +0000

Two years of daily Claude + ChatGPT. They've seen probably a million tokens of my writing. Every response still opens with "Certainly!" or "Great question!" and closes with "In conclusion…".

Nobody writes like that. The model has no idea who you are — you're just another session.

So I built chatlectify. Point it at your exported chat history (Claude / ChatGPT / Gemini JSON, or a folder of your own writing — blog posts, emails, notes). It outputs a SKILL.md + system_prompt.txt that makes the model write like you.

How it works

Extracts ~20 stylometric features from your messages — sentence-length distribution, contraction rate, bullet usage, hedge words, typo rate, punctuation histograms, question-vs-imperative ratio, top sentence starters
Picks a stratified sample of your messages across length buckets as exemplars
One LLM call distills it all into a portable style file

Privacy

Runs locally. Exactly one outbound LLM call to your configured model — the synth step that writes the style file. That call includes your feature summary and ~40 exemplar messages (the stratified sample). Nothing else leaves your machine. No telemetry, no cloud backend, no account.

Usage

pip install chatlectify
chatlectify all ./conversations.json --out-dir ./my_skill

Drop the folder into ~/.claude/skills/ or paste system_prompt.txt into any model that takes one.

Repo: https://github.com/0x1Adi/chatlectify

Curious what people think. Also — which export formats should I add next? Slack, iMessage, email, Discord, Obsidian vault?

🎭 Slopsquatting: The Supply Chain Attack Hiding in Plain Sight

Aditya Tiwari — Fri, 07 Nov 2025 11:21:33 +0000

Your AI coding assistant just suggested a package that doesn't exist. An attacker is about to register it with malware.
Researchers analyzed 576,000 AI-generated code samples and found something terrifying:
→ 205,474 unique "phantom packages" that don't exist in PyPI/npm
→ 43% repeat PERFECTLY across identical prompts
→ Commercial AI (GPT-4, Claude, Copilot): 5.2% hallucination rate
→ Open-source LLMs: 21.7% hallucination rate

This isn't typosquatting. It's slopsquatting - exploiting systematic AI behavior.

The attack is trivial:
Query AI assistant with common prompts
Collect hallucinated package names
Register them on PyPI/npm with malicious code
Wait for developers to pip install your malware

Here's what makes this surreal:
Despite 6 months of security research, 205K identified targets, and trivial exploitation... zero confirmed attacks exist in the wild.
The window for defense is open. But it's closing fast.
I wrote a deep-dive on:
✓ Why AI reliably hallucinates the same phantom packages
✓ Which security tools detect this (spoiler: almost none)
✓ A 15-minute scanner you can deploy today
✓ Why zero attacks won't last much longer

https://lnkd.in/dw6R2qSN

If you're using AI coding assistants (and you probably are), this affects you.

Read it before the first confirmed attack makes headlines.

Disclaimer: Personal analysis based on my cybersecurity background. Not legal advice. Views are my own.

hashtag#CyberSecurity hashtag#AppSec hashtag#AI hashtag#DevSecOps hashtag#SupplyChainSecurity hashtag#SoftwareDevelopment hashtag#InfoSec hashtag#DevOps

Are you fuzzing?

Aditya Tiwari — Fri, 07 Nov 2025 11:17:02 +0000

Why Fuzzing Matters More Than Ever in the AI Code Generation Era

By 2025, nearly half of all code in AI-assisted projects is generated by LLMs. 63% of developers now use AI tools daily.

We've embraced the productivity gains without updating our testing practices.

Result? We're testing AI-generated code with techniques designed for human cognitive biases.

The data is stark: Google's AI-powered fuzzer found a vulnerability in OpenSSL that had existed for 20 years. Another AI system found a SQLite bug that 150 CPU-hours of traditional fuzzing missed entirely.

These aren't edge cases. As of May 2025, OSS-Fuzz has identified 13,000+ vulnerabilities across 1,000+ projects. The 26 vulnerabilities found by AI-enhanced fuzzing were all unreachable by human-written test harnesses.

I spent time researching what actually works for testing AI-generated code. The answer: automated fuzzing. Not because it's trendy, but because it's the only technique that doesn't rely on human assumptions about how code should behave.

Wrote up the full analysis with implementation guide and cost breakdown: https://lnkd.in/dYRNQxEB

Tools are free. Techniques are proven. What's missing is organizational adoption.

Disclaimer: Views are my own.

hashtag#CyberSecurity hashtag#ApplicationSecurity hashtag#Fuzzing hashtag#AI hashtag#SoftwareEngineering hashtag#DevSecOps hashtag#VulnerabilityResearch

Announcing SlopGuard — Open-Source Defence Against AI Supply Chain Attacks

Aditya Tiwari — Fri, 07 Nov 2025 11:16:02 +0000

Your AI coding assistant just suggested installing a package. It doesn't exist. You install it anyway. Now you're compromised.

This isn't hypothetical—AI models hallucinate non-existent package names in 5-21% of generated code. Research analyzing 576,000 code samples found 205,000+ unique phantom packages, with 58% recurring predictably across sessions.

Attackers exploit this by monitoring AI outputs, registering these hallucinated packages with malware, and waiting for developers to blindly install them. It's called "slopsquatting."

While exploring AI supply chain risks (wrote about it here: https://lnkd.in/dS3D-zwt), I built SlopGuard to detect these attacks before they reach production.

🔍 Technical approach:
• 3-stage lazy-loading trust scorer (downloads → dependents → maintainer/GitHub)
• 87% of packages exit at stage 1 (basic trust), only 3% need full analysis
• Detects typosquatting (Levenshtein distance), namespace squatting, download inflation, ownership changes
• Automated scoring from verifiable signals—no manual whitelists that break
⚡ Validated performance:
• <3% false positives (tested against 500 legitimate + 500 malicious packages)
• 96% detection rate on documented supply chain attacks
• 7 seconds to scan 700+ packages (warm cache)

Built in Ruby, ~2,500 lines, fully open source (MIT).
━━━━━━━━━━━━━━━━━━━━━━━━
Current stage: Early development, personal research project. RubyGems/PyPI/Golang for now. Metadata-based detection—can't analyze behavioral patterns.

Looking for technical peer review:
• Are trust thresholds (80/70/40) optimal for production? • What attack patterns am I missing?
• Would you deploy this in CI/CD, or does it solve the wrong problem?

Try it: https://lnkd.in/dHyvucVQ
GitHub: https://lnkd.in/d2TsnQ3s

If you work on supply chain security or AI security research—where are the blind spots in this approach?

Disclaimer: This is my personal project in early development. The algorithms are based on academic research (TypoSmart, Sonatype data).