DEV Community: Prasanna Nattuthurai

Stop Reading Your AI Agent. Start Talking to It.

Prasanna Nattuthurai — Thu, 12 Nov 2026 01:42:09 +0000

How I gave my coding agent a free, fully local voice for my personal projects, and why I think this is where agentic coding is headed.

🏡 Everything here was built and used on my own side projects, on my personal laptop.

Here's a confession: I've stopped reading most of what my coding agents write.

Not because it's bad. Because there's too much of it. An agent kicks off, reads twelve files, runs the tests, edits four modules, and then hands me a 900-word wall of text with tables, file paths and a bulleted summary of a bulleted summary. By paragraph three my eyes are skimming. By paragraph five I've wandered off to another tab.

That's the quiet problem with agentic coding nobody talks about.

The agents got faster. Our attention didn't.

The bottleneck isn't the model anymore. It's me, scrolling.

So I tried something different. I gave my agents a voice and ears. Now I ask a question out loud, the agent goes off and does the work, and comes back with two spoken sentences: "Tests pass, I changed the retry logic in two files. Want the details on screen?"

💻 It runs entirely on my laptop. 💸 It cost me nothing. I built it in Claude Code, but nothing about it is Claude-specific: any agent that can use MCP tools can get the same thing.

Here's how, and why I think you should try it.

🎙️ Why voice, why now

Three things changed my mind.

1. Agents produce output faster than humans can read it.
A single agentic task can generate more text than a short blog post. Reading all of it is exhausting; skimming it means you miss the one line that mattered. Speech forces the agent to summarise. You can't speak a table. You can't speak a 40-line diff. So the agent has to tell you what actually matters, and put the rest on screen if you want it.

2. Talking keeps you in the loop without keeping you glued to the screen.
When I'm in voice mode I can lean back, look away, think. The agent works between turns and reports back by voice. It feels less like supervising a log file and more like pairing with a friend who's sitting next to you.

3. It's finally free and local.
A year ago this meant cloud speech APIs, API keys, and paying per minute. Today, open models like Whisper (speech-to-text) and Kokoro (text-to-speech) run comfortably on a laptop. No account, no key, no audio leaving your machine.

🔌 The idea: one voice layer, any agent

The trick is that the voice doesn't live inside the agent. It lives in an MCP server, the standard plug-in protocol that Claude Code and a growing list of other coding agents already speak.

The MCP server I use is VoiceMode (open source, MIT licensed). It exposes one main tool, converse: speak something, then listen and return what the user said. Any agent that can call MCP tools can now hold a conversation.

That's the whole point: build the voice once, plug it into every agent you use. When the next coding assistant comes along, it gets a voice for free too.

⚡ Setup: about ten minutes

I'm on an Apple Silicon Mac with 16 GB of RAM. Nothing exotic.

1. Install VoiceMode and the local speech services

uvx voice-mode-install
voicemode service install whisper   # speech-to-text
voicemode service install kokoro    # text-to-speech

2. Pin it to local only

By default VoiceMode can fall back to a cloud API if it finds an API key. I don't want that:

voicemode config set VOICEMODE_STT_BASE_URLS http://127.0.0.1:2022/v1
voicemode config set VOICEMODE_TTS_BASE_URLS http://127.0.0.1:8880/v1

Now nothing you say leaves the machine.

3. Plug it into Claude Code

One line:

claude mcp add --scope user voicemode -- ~/.local/bin/voicemode

4. Add a /talk command

A tiny custom slash command: a markdown file in ~/.claude/commands/ whose prompt says start a voice loop with the converse tool, keep replies short, stop when I say “stop”. Type /talk, and you’re talking.

That's it.

🧩 Using a different agent?

The same recipe works in any coding agent that supports MCP. The pieces are always the same:

Run the two local speech services (Whisper and Kokoro). They don't care which agent calls them.
Register VoiceMode as an MCP server in your agent's MCP config, pointing at the voicemode binary.
Allow its converse tool, if your agent asks you to approve tools.
Add a reusable prompt (a slash command, custom prompt or rules file, whatever your agent calls it) with the voice-loop rules below.

Check your agent's docs for where its MCP config lives. Some read one global file; others want the server listed per agent or per project.

💬 Making it feel like a conversation, not a robot

Getting audio in and out is the easy part. Making it pleasant took some prompt tuning. The rules that made the biggest difference:

One or two sentences, ~25 words. The agent offers to say more instead of explaining everything. This alone fixes the wall-of-text problem.
Never read code, paths or tables aloud. Speak the summary; put the detail on screen.
A live transcript. Before each turn the agent prints 🎤 You: … and 🔊 Friday: …, so the screen matches what you hear. Handy when the speech recognizer mishears you.
A persona. In voice, my agent answers as "Friday": calm, concise, a little dry. Put the persona in the shared prompt and every agent you wire up sounds like the same assistant. Small thing, surprisingly nice.
Sound human. Vary the acknowledgements. Nothing kills the vibe like hearing "Great question!" eleven times.

🛑 The safety rule you must not skip

Speech recognition mishears things, especially names, numbers and jargon. "Delete the test branch" and "delete the best branch" are one syllable apart.

So my rule is simple:

Read-only questions ("what failed?", "summarise the last change"): the agent just answers.
Anything that changes or sends something (file writes, deletes, git commits and pushes): the agent repeats back what it understood in one sentence and waits for a spoken "yes".

👉 Never let a voice command run an action unconfirmed. This one rule is what makes voice safe enough to actually rely on.

🪙 "But doesn't voice burn more tokens?"

I wondered the same, so I measured it from my own session logs.

Across 16 voice exchanges:

~350 output tokens per turn (spoken reply + on-screen transcript + reasoning)
~800 uncached input tokens in total. What you say comes back as a tiny bit of text.
The big number, ~110k cached tokens per call, is just the conversation context being re-read, which is exactly the same when you type.

✅ Verdict: a voice turn costs about the same as a typed one. Speech-to-text and text-to-speech run locally, so they're free. The only extra is the transcript line, maybe 50–100 tokens a turn. If anything, voice saves tokens, because short spoken answers replace long written ones.

(Caveat: that's one session, not a controlled typed-vs-spoken comparison. Take it as a strong hint, not a benchmark.)

⏱️ Is it fast enough?

Mostly, yes. The voice gap after the agent finishes thinking is about 2 seconds: ~0.3 s for Whisper to transcribe you, ~1.6 s for Kokoro to start speaking. The real variable is how long the model thinks, which you'd wait for anyway.

One surprise: Kokoro was faster on CPU than on the Mac's GPU. Time to first audio was 0.4 s on CPU versus 2.2 s on GPU, and the GPU never caught up after warm-up. If your local text-to-speech feels slow, try forcing CPU.

And a lesson: when benchmarking TTS, use new text every time. Repeated text gets cached, and my first measurement looked 5× better than reality.

🪤 Gotchas I hit, so you don't

Bluetooth headset as the mic → beeps and robot voice. macOS flips the headset into "call mode" when it starts listening. Use the laptop mic for input and keep the headset for output.
voice-mode-install --model small broke the Whisper install on my setup. Install Whisper on its own with voicemode service install whisper.
Python version mismatch. Kokoro wants Python 3.10; if your machine defaults to newer, rebuild its venv on 3.10.
Corrupt model download. If Kokoro won't start, check the model file size. Mine was 50 MB short.
Not every agent reads a global MCP config. If the voice tools don't show up, add the server to the per-agent or per-project config.
Privacy note: VoiceMode's logs store what you said in plain text. Put them on your cleanup list.

🚫 What I didn't use, and why

I looked around before settling on this:

Claude Code's built-in /voice: great for dictation, but input-only and cloud-transcribed. No spoken replies.
Other voice MCP servers: either barely maintained, macOS-only and stale, or multiple GBs of models per project.
Cloud voice tools: some have nicer live transcripts, but they send your audio to the cloud and some bill separately. Not what I wanted.

VoiceMode was the only option I found that was fully local, actively maintained, and worked across every project and every agent.

🧠 "Is AI making us dumber?"

You've probably seen the term cognitive debt going around, along with cognitive surrender and AI brain fry. The worry is real, and the research is starting to back it up:

In a randomized trial from Anthropic, junior developers who used an AI assistant scored 17% lower on comprehension tests than those who coded by hand. The interesting part is how they used it. People who handed the whole task to the AI scored below 40%. People who asked it to explain things scored above 65%, as well as or better than the no-AI group.
Researchers Shaw and Nave draw a useful line between cognitive offloading (you delegate a task but keep thinking about it) and cognitive surrender (the AI's answer becomes your answer without much scrutiny).
Boston Consulting Group researchers coined "AI brain fry" for the mental hangover of keeping up with tools that produce more than you can absorb.

Here's my take: the wall of text is a big part of the problem. When an agent dumps 900 words on you, the easiest move is to skim, nod and accept. That's surrender by exhaustion.

Voice won't fix this on its own. You can say "yes" to everything just as lazily as you can click "accept". But it changes the default in a few useful ways:

You have to say what you want out loud. Putting your intent into a sentence is thinking. It's hard to explain a task without understanding it.
Short answers invite follow-ups. When the agent says "I changed the retry logic in two files", the natural reply is "why two?" A spoken back-and-forth pulls you into the reasoning instead of past it.
Less to read means less overload. You get the gist by ear and open the detail on screen only when it matters. That's the brain fry problem in reverse.
The confirmation step makes you decide. The agent says back what it's about to do, and you have to actively agree.

A few habits I've picked up to stay on the offloading side of the line:

Say your guess before you ask. ("I think it's the timeout. Check?")
Ask "why" at least once per task.
Ask the agent to explain, not just to do, especially in code you don't know yet.
Every so often, solve something yourself first and use the agent to check it.

Voice makes these habits feel natural, because they're just how you'd talk to a person.

🚀 Where this is going

I don't think we'll read our agents' output forever. As agents take on longer, more autonomous tasks, the human job shifts from reading everything to staying oriented: knowing what happened, what's blocked, and what needs a decision, while still doing the thinking yourself.

Voice is good at exactly that. It's short and in the moment, and you don't have to sit and stare at a screen to follow it. Detail stays on screen for when you want it; the gist comes to you.

The setup is free, local, and about ten minutes of work. If you spend your evenings building side projects with a coding agent, try talking to it for a week. I don't think you'll go back to scrolling.

🔭 Next up for me: trying NVIDIA's Parakeet model for even faster speech recognition, and moving the speech services to a home server so the laptop stays cool.

My AI Trading Bot Looked Profitable. Every Number That Said So Was the Wrong Number.

Prasanna Nattuthurai — Fri, 06 Nov 2026 02:41:32 +0000

Five ways a prediction-market model fooled me, and the one metric I trust now

For five months I ran an LLM-powered forecaster against real prediction markets: Kalshi and Polymarket. It researched a market, estimated a probability, compared it with the price, and logged a "shadow" bet whenever it disagreed enough. Every resolved bet went into a SQLite database.

The dashboards looked great. Directional accuracy was above 60%. Brier scores cleared the bar I'd set. One strategy beat the market on its first 45 trades. Another showed a statistically significant edge.

None of it held up. That isn't because the model was secretly brilliant and unlucky. Each of those numbers was measuring something other than "will this make money." This post is about how I found that out, one metric at a time. It's the most useful engineering lesson I've learned this year, and it applies well beyond betting. If you evaluate any AI system, whether a classifier, a ranker, an agent or a forecaster, you probably have one of these five problems too.

1. Accuracy over everything vs. accuracy on what you act on

The headline number was 60.9% directional accuracy. That was over every market the model looked at.

But most of those were SKIPs: markets where the model's estimate sat close to the price, so no bet would ever be placed. On those, "pick the favourite" is easy and the model agrees with the crowd. That pads the score.

The bets that matter are the ones where the model disagrees with the market by 5 points or more. That's the only subset where money moves. On that subset:

61 of 152 correct. 40.1%.

The corpus number and the actionable number pointed in opposite directions. The model was right when it agreed with everyone and wrong 60% of the time when it had an opinion.

The lesson: score the decisions your system actually makes, not everything it touches. If your agent only acts on 15% of inputs, the other 85% are noise in your eval.

2. Beating a fixed bar vs. beating the baseline you compete with

My go-live gate said "Brier score under 0.25." Soccer came in at 0.2177, which passed.

The market's own price scored 0.2118 on the same games.

The model "passed" while losing to the thing it was betting against. A fixed threshold told me nothing. The only bar that matters for a forecaster is the price, because the price is your opponent. When I re-scored every football category against the market, the market won every time. Over 517 markets it was 0.2376 for the model and 0.2166 for the price. I killed football forecasting.

The lesson: a threshold is a guess. A baseline is a fact. Compare against the best simple thing you could do instead, whether that's the price, the heuristic, the last model or the if/else.

3. Small-sample edge vs. the same edge at 4× the data

Cross-venue "arbitrage" (the same event priced differently on two exchanges) looked like the cleanest win. At n=45, its Brier score was 0.0906 against the market's 0.0928. It was a small edge, but a real one, and it was enough that I wired it up to place real orders.

At n=170, the edge was gone: 0.1951 for the model, 0.1951 for the market. On the 21 bets it actually took, it went 9 for 21.

A negative or zero edge doesn't turn positive with more data. More volume only makes you more certain it's not there.

The lesson: never promote a system on its first good sample. Decide the checkpoint size before you look, and don't let an early number buy production access.

4. Gross edge vs. net edge at the price you can actually get

This one hurt the most because the bug was a single constant.

The "favourites" strategy bought heavily favoured outcomes priced 0.65–0.80. I pre-registered it: run until 100 resolved out-of-sample bets, then judge. At 135 resolved bets:

Hit rate 65.19% against an average price of 68.69%
Gross edge -3.50pp per contract

That was already negative. Then I found my fee model was wrong. I had Polymarket US's taker fee as 0.003 × p. The real formula is 0.0695 × p × (1 − p), about 1.5 points at these prices, or five times what I'd modelled. Net edge was -4.97pp.

Even the one statistically significant positive result had the same problem. Closing-line value on favourites was +1.15pp, with a 95% CI of [+0.37, +1.97] that stayed above zero. But that was measured at the mid-price. Scored at the ask, the price you actually pay, after fees, it came to -1.93pp. The spread plus the fee ate it.

The lesson: measure at the point of execution, with the real costs, from the real docs. "Edge at mid" is a backtest fantasy. For AI systems the equivalent is quality measured without latency, token cost or the retry rate.

5. Data that leaks vs. data the model could actually see

The economics markets ask things like "Will CPI come in above 3.2%?" The model's estimates were wildly off, and for weeks I treated that as a modelling problem.

It was a parsing bug. Any strike with a decimal point got cut off, so the question the model received read "more than -0". 251 rows were estimated from questions with the threshold missing. I was grading a model on a test it could not read.

The opposite leak showed up in the backtest. The walk-forward calibration was quietly using information from after the forecast time (look-ahead), and it counted some markets more than once. Fixing both made the numbers worse, and they were also the first honest ones.

The lesson: before you blame the model, read the exact input it received. Then check that your evaluation can't see the future.

The one metric I trust now

After all five fixes, one number decides whether real money is allowed:

Realised edge, net of fees, at the executable price, one row per market, with a bootstrapped 95% confidence interval whose lower bound must clear zero.

The all-time figure today: -0.61pp, CI [-8.01, +6.88]. Not cleared.

Then there's the part nobody puts on a dashboard: statistical power. My earlier gate said I needed "about 400 more bets." That was a 50%-power figure built on an effect size that turned out to be inflated. At 80% power, proving a real 3-point edge takes about 2,000 bets. At my volume, that's a long time.

So the bot doesn't trade. Specifically:

Real orders are hard-disabled. Every command that can place one refuses unless an explicit environment flag is set for that single run.
There's a lifetime loss cap of $50 per venue, enforced in code, not by willpower.
Kill criteria are pre-registered, and "kill" is the default. The economics strategy gets judged on a fixed date against a fixed test. If the result is ambiguous, it dies. I wrote the rule down before seeing the data so I can't negotiate with myself later.
No refitting a dead hypothesis. When "favourites" failed, the 0.70–0.80 slice looked mildly positive. That slice came from cutting up a failed experiment, so it doesn't get a test of its own.

Why I'm publishing a failure

Every "I built an AI trading bot" post I've read ends with an upward-sloping equity curve. Mine ends with a confidence interval that straddles zero and an honest power calculation. That's the more useful post.

The bot didn't fail because LLMs can't reason about probabilities. The evaluation failed five times, and each failure flattered the model. That pattern is everywhere in AI engineering right now: a leaderboard number that doesn't match your traffic, a demo scored on the easy cases, a cost model off by 5×, a benchmark leaking into the training data.

The forecaster was the easy part. Building a measurement system that couldn't lie to me took months, and that's the part I'd hire for.

If you're building evals for agents or forecasters and want to compare notes on baselines, power and pre-registration, I'd love to hear how you're doing it.

Your Developer Toolchain Is a Supply Chain Attack Surface — Here's How I Hardened Mine

Prasanna Nattuthurai — Sat, 22 Aug 2026 17:14:58 +0000

A practical guide to applying SLSA, Sigstore, and SBOM-based verification to the tools on your local machine.

The Problem Nobody Talks About

We obsess over dependency scanning in CI/CD. We pin versions in package-lock.json. We run Dependabot. But what about the tools that build your software?

Every morning, you run brew upgrade. npm updates your global packages. rustup update refreshes your Rust toolchain. Each of these is an implicit trust decision:

"I trust that the binary I just downloaded is exactly what the publisher built, from the source code I'd expect, without tampering in transit."

For most developers, that trust is blind. I decided to make it explicit.

Threat Model: What Can Go Wrong

Threat	Real-World Example	Likelihood
Malicious package in registry	`event-stream` (npm, 2018), `ua-parser-js` (npm, 2021)	High
Compromised Homebrew tap	Typosquatting tap overrides official formula	Medium
npm postinstall script RCE	`eslint-scope` credential theft (2018)	High
Binary replaced after install	Compromised local system, PATH hijack	Low
DNS/BGP hijack on update channel	PyPI CDN poisoning (hypothetical)	Low
Dependency confusion	Internal package name claimed on public registry	Medium

The attack surface is your entire toolchain: compilers, linters, security scanners, CLI tools, language runtimes. If an attacker compromises trivy, they can hide their own CVEs from your scans.

The Architecture: 6 Defense Layers

┌─────────────────────────────────────────────────────────────────────┐
│                    DEVELOPER WORKSTATION                             │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Layer 1: TRANSPORT SECURITY                                        │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │  All channels: TLS 1.2+ (brew, npm, PyPI, crates.io, Go)    │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                              │                                      │
│  Layer 2: CRYPTOGRAPHIC VERIFICATION                                │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │  brew: SHA256 bottles    │  uv: Sigstore cosign              │  │
│  │  rustup: signed manifest │  npm: ECDSA registry sigs         │  │
│  │  macOS: Apple code-sign  │  Go: sum.golang.org               │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                              │                                      │
│  Layer 3: SOURCE ALLOWLISTING                                       │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │  Homebrew tap allowlist (BLOCK on untrusted)                  │  │
│  │  npm pinned package list + --ignore-scripts                   │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                              │                                      │
│  Layer 4: SYSTEM INTEGRITY                                          │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │  SIP enabled │ Gatekeeper enabled │ No world-writable PATH   │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                              │                                      │
│  Layer 5: AUDIT TRAIL                                               │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │  SHA256 pre/post snapshot │ Timestamped log │ Diff alerting   │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                              │                                      │
│  Layer 6: SUPPLY-CHAIN VERIFICATION (post-update)                   │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │  ┌─────────────┐ ┌───────────────┐ ┌─────────────────────┐  │  │
│  │  │ brew verify │ │ npm audit sig │ │ gh attestation      │  │  │
│  │  │ (Sigstore)  │ │ (provenance)  │ │ verify (SLSA L3)    │  │  │
│  │  └─────────────┘ └───────────────┘ └─────────────────────┘  │  │
│  │  ┌─────────────┐ ┌───────────────┐ ┌─────────────────────┐  │  │
│  │  │ syft→grype  │ │ govulncheck   │ │ cargo audit         │  │  │
│  │  │ (SBOM+CVE)  │ │ (-mode=binary)│ │ (RustSec DB)        │  │  │
│  │  └─────────────┘ └───────────────┘ └─────────────────────┘  │  │
│  │  ┌─────────────┐ ┌───────────────┐                          │  │
│  │  │ codesign -v │ │ SHA diff alert│                          │  │
│  │  │ (macOS sig) │ │ (tampering)   │                          │  │
│  │  └─────────────┘ └───────────────┘                          │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

The Gap Analysis: Industry vs. Reality

I benchmarked my existing toolchain updater against SLSA v1.0, NIST SP 800-218 (SSDF), and CISA's Secure Software Development guidance. Here's what I found:

What Most Developers Have (SLSA L1)

✅ Lock files with checksums (package-lock.json, Cargo.lock, go.sum)
✅ TLS on all download channels
✅ Package manager verifies checksums at install time

What's Missing (SLSA L2/L3)

Gap	Risk	What's actually needed
No independent provenance verification	Medium	Don't just trust the package manager's internal check — verify Sigstore attestations yourself
`npm audit` ≠ `npm audit signatures`	Medium	CVE scanning is different from provenance verification
GitHub Release downloads unattested	High	If you download a tarball from Releases, verify SLSA provenance
No SBOM of installed toolchain	Medium	You can't find vulns inside compiled binaries without an SBOM
SHA recorded but never compared	High	A hash that's never diffed is security theater
Go binaries never scanned	Medium	Most security tools (trivy, gh, gitleaks) are Go — embedded deps have vulns too
Cargo binaries never audited	Low	crates.io has zero Sigstore support — `cargo audit` is all you have

The Implementation

1. Sigstore Provenance Verification (Homebrew)

Homebrew bottles from homebrew-core have had SLSA L3 Sigstore attestations since 2024. Most developers don't know this — or that you can verify them independently:

# Verify your security tools came from the real Homebrew CI
brew verify trivy
brew verify semgrep
brew verify gitleaks
brew verify gh

This checks the Sigstore transparency log and confirms the bottle was built by Homebrew's GitHub Actions workflow, from the expected source commit.

Real output from my system:

[sec]   ✓ provenance verified: trivy
[sec]   ✓ provenance verified: semgrep
[sec]   ✓ provenance verified: gitleaks
[sec]   ✓ provenance verified: osv-scanner
[sec]   ✓ provenance verified: gh
[sec]   ✓ provenance verified: jq
[sec]   ✓ provenance verified: node
[sec]   ✓ provenance verified: python@3.13
[sec]   ✓ provenance verified: go
[ok]    Homebrew provenance: all 9 critical formulae verified

2. npm Provenance Signatures (Not Just `npm audit`)

npm audit checks for known CVEs. npm audit signatures checks that packages were built from their claimed source repo via Sigstore:

# This checks CVEs:
npm audit

# This checks provenance (different!):
npm audit signatures

These are complementary, not interchangeable. A package can have zero CVEs but be a complete forgery.

3. SBOM + Binary Vulnerability Scanning

Here's the insight most developers miss: compiled binaries embed their entire dependency tree. A Go binary built with a vulnerable net/http is vulnerable even after you've updated your system Go.

# Generate an SBOM of all your Homebrew binaries
syft dir:$(brew --prefix)/bin -o cyclonedx-json > toolchain-sbom.json

# Scan for embedded vulnerabilities
grype sbom:toolchain-sbom.json --only-fixed --fail-on high

My result: grype: no high/critical fixable vulns in toolchain binaries ✅

4. Go Binary Vulnerability Scanning

govulncheck -mode=binary is the Go team's official tool for scanning compiled Go binaries. It reads the embedded module info and checks against the Go vulnerability database:

# Scan your security tools (they're all written in Go!)
govulncheck -mode=binary $(which trivy)
govulncheck -mode=binary $(which gh)
govulncheck -mode=binary $(which gitleaks)
govulncheck -mode=binary $(which osv-scanner)

On a first run this surfaced a batch of Go stdlib DoS advisories in one of the scanners above — the binary had been compiled against an older Go toolchain than the one that fixed them. Low risk in a local CLI context, and resolved by a rebuild, but the point is that nothing else I ran would have told me.

5. SHA Diff Alerting (The Most Important One)

Recording SHA256 hashes is meaningless if you never compare them. The real value is detecting unexplained changes:

# After every update run, record hashes
shasum -a 256 $(brew --prefix)/bin/trivy >> binary-shas.txt

# On next run, compare:
# - If SHA changed AND an update was logged → expected ✅
# - If SHA changed WITHOUT a logged update → RED FLAG 🚨

My implementation detects this automatically:

[sec]   Comparing runs: 2026-08-09 → 2026-08-18
[sec]   ✓ go: SHA changed (update logged)
[ok]    Binary diff: 1 changes, all explained by logged updates

If a binary ever changes without a corresponding update in the log, the script flags it as potential tampering.

The Complete Flow

┌──────────────────┐
│   brew upgrade   │──→ SHA256 bottle verification (automatic)
│   npm install -g │──→ --ignore-scripts (kill postinstall RCE)
│   uv self update │──→ Sigstore cosign verification (automatic)
│   rustup update  │──→ Signed manifest verification (automatic)
└────────┬─────────┘
         │
         ▼
┌──────────────────────────────────────────────────────┐
│            POST-UPDATE VERIFICATION                   │
├──────────────────────────────────────────────────────┤
│                                                      │
│  1. brew verify (Sigstore provenance)                │
│  2. npm audit signatures (provenance ≠ CVE scan)    │
│  3. gh attestation verify (SLSA L3 for Releases)    │
│  4. syft → grype (SBOM-based binary vuln scan)      │
│  5. codesign -v (macOS signature validation)        │
│  6. SHA diff alerting (detect unexplained changes)   │
│  7. govulncheck -mode=binary (Go embedded deps)     │
│  8. cargo audit (RustSec advisory DB)               │
│                                                      │
└────────┬─────────────────────────────────────────────┘
         │
         ▼
┌──────────────────┐
│  SHA256 snapshot │──→ Baseline for next run's diff
│  SBOM archived   │──→ Audit trail
│  Log recorded    │──→ Evidence
└──────────────────┘

What This Catches (Real Examples)

Scenario	Which layer detects it
`event-stream` style npm hijack	Layer 3 (`--ignore-scripts`) blocks execution
Typosquatting Homebrew tap	Layer 3 (tap allowlist blocks unknown taps)
Compromised brew bottle	Layer 6 (`brew verify` — Sigstore attestation mismatch)
Binary replaced by attacker on disk	Layer 5/6 (SHA diff alert — unexplained change)
Vulnerable dependency inside `trivy`	Layer 6 (`govulncheck -mode=binary`)
npm package without provenance	Layer 6 (`npm audit signatures` flags it)
PATH hijack via world-writable dir	Layer 4 (detected at script startup)

The Ecosystem Reality Check

Not everything is verifiable yet. Here's the honest state as of 2026:

Ecosystem	Provenance Coverage	What's available
Homebrew	✅ 100% (homebrew-core)	Sigstore attestations on all bottles
npm	⚠️ ~7% of packages	`npm audit signatures` + registry ECDSA
PyPI	⚠️ ~17% of uploads	PEP 740 attestations (growing fast)
Go modules	✅ Checksums (sum.golang.org)	No Sigstore, but tamper-evident log
crates.io	❌ Checksums only	No Sigstore, no provenance
GitHub Releases	⚠️ Opt-in	`gh attestation verify` (if publisher enables)

The strategy is defense in depth: verify what you can cryptographically, scan what you can't verify, and detect tampering as a last resort.

Getting Started (15 Minutes)

# 1. Install the verification tools
brew install syft grype cosign
go install golang.org/x/vuln/cmd/govulncheck@latest
cargo install cargo-audit

# 2. Verify your current Homebrew bottles
for f in trivy semgrep gh node python@3.13 go; do
  brew verify "$f" 2>/dev/null && echo "✓ $f" || echo "✗ $f"
done

# 3. Check npm provenance
npm audit signatures

# 4. Generate your first toolchain SBOM
syft dir:$(brew --prefix)/bin -o cyclonedx-json > ~/toolchain-sbom.json
grype sbom:~/toolchain-sbom.json --only-fixed

# 5. Scan Go binaries
govulncheck -mode=binary $(which trivy)
govulncheck -mode=binary $(which gh)

# 6. Create your SHA baseline
for bin in brew node npm go cargo trivy gh; do
  shasum -a 256 "$(which $bin)" >> ~/binary-shas-baseline.txt
done

What I Learned

Recording hashes is not monitoring. A hash file that's never diffed is security theater. You need automated comparison with explainability (was there a logged update or not?).
Your security tools are attack surface. trivy, semgrep, gitleaks — if these are compromised, they can hide their own findings. Scan the scanners.
npm audit is not npm audit signatures. One checks CVEs, the other checks provenance. You need both.
Bash 3.2 is still the default on macOS. If your security script uses mapfile or associative arrays, it silently fails on every Mac. Ship it or test it.
Timeouts matter. Any network call in a security script that can hang indefinitely will eventually hang indefinitely. I learned this the hard way when Sigstore transparency log queries hung for 90 minutes.
SLSA is a consumer spec too. Most people think SLSA is for producers/builders. It's equally a spec for how consumers should verify artifacts.
A check that fires on a healthy machine is worse than no check. My world-writable PATH test matched any two w characters in the permission string — so it flagged every group-writable directory, which is Homebrew's own normal layout. It also read the wrong column, testing other-read instead of other-write. Result: PATH hijack risk! on a correctly configured Mac, every single run — a warning with no way to act on it, which is the kind you stop reading. A false positive doesn't just waste attention; it trains you to skip the output, and then the true positive scrolls past too. Prefer a false negative you'll notice missing over a false alarm you'll learn to filter.
My tool was broken for every user but me, and only a second machine could tell me. One line read the list of Rust-installed binaries:

   cargo install --list | grep -E "^[a-z]" | while read -r line; do

Under set -euo pipefail, grep matching nothing exits 1, which aborts the entire script. On a machine with no cargo-installed binaries — a fresh Mac, most people's Mac — the dry run died silently a third of the way through. It worked flawlessly for me because I happen to have some. No amount of testing on my own laptop could have surfaced it; a CI runner found it on its first execution. If you're writing a tool that inspects an environment, the one environment you cannot test in is your own.

References

The full implementation is open source — a 1,300-line bash script with no
runtime dependencies beyond Homebrew itself:

brew tap prasannavarshan/tap
brew trust prasannavarshan/tap    # Homebrew gates third-party taps
brew install update-toolchain

update-toolchain --dry            # report; change nothing
update-toolchain --audit          # security checks only

Source, threat model and known limitations:
github.com/prasannavarshan/update-toolchain

Allowlists are config files rather than code, so nothing updates unless you have
named it — and the config is parsed, never sourced, because a tool that decides
what to trust should not let its own config file execute anything.

Tags: #SupplyChain #DevSecOps #SLSA #Sigstore #macOS #Homebrew #SBOM #SecurityEngineering