DEV Community: Kunal

Prevent API Key Leaks in Shell History (bash/zsh/fish) [2026]

Kunal — Sun, 06 Sep 2026 00:42:01 +0000

You can lock down 80% of “oops I leaked a key” incidents in about 20 minutes. The prerequisite that trips people up is this: your shell will happily persist whatever you type (history files, dotfiles, exports), and modern AI/agentic tooling has made us paste secrets into terminals more often than ever.

This guide is specifically about how to prevent API key leaks in shell history across bash, zsh, and fish. I’ll give you copy-pasteable hardening snippets, the safer way to pass secrets to CLIs locally, how to use direnv and mise without committing keys, and what to do when a key still leaks.

Based on the keyword neighborhood signals I pulled from my own site’s research tooling, this is a winnable problem space for teams: 99 related impressions in the “prevent secret leaks” neighborhood, with a best observed position ~2.7, across an estimated ~460 searches/month worth of adjacent queries. In other words: people are searching for this. And they’re mostly getting bad advice.

What is terminal secrets hygiene?

Terminal secrets hygiene is the set of practices that prevents secrets you type or load in a terminal (API keys, tokens, passwords) from being permanently stored in shell history, environment exports, dotfiles, logs, screenshots, or git.

The reason I’m opinionated about this: the “just don’t do that” approach fails the minute you’re debugging a production issue at 2 a.m. or copy/pasting a vendor’s curl example into a terminal. You need guardrails that work when you’re tired.

Here’s the checklist we’re going to implement:

Configure bash/zsh/fish to not record the most common secret-bearing commands.
Stop using global export as your default. Scope secrets to the smallest possible lifetime.
Keep secrets out of dotfiles and repos. Use local-only files.
Use direnv and mise to load per-project secrets safely.
Add secret scanning so you get yelled at before GitHub does.
Have a runbook for when a key leaks anyway.

Shell history basics: where history is stored and why secrets leak

Before settings, you need the mental model.

Most shells append your commands to a plain-text history file in your home directory. On macOS and Linux, it’s typically one of:

bash: ~/.bash_history
zsh: ~/.zsh_history
fish: ~/.local/share/fish/fish_history

That history file is:

readable by you (obviously)
often backed up (Time Machine, corporate laptop backup agents)
often copied around when people sync dotfiles across machines
sometimes exfiltrated by infostealers that target developer machines

The most common leak patterns I see in real teams:

A vendor doc says: curl -H "Authorization: Bearer sk_live_..." ...
Someone types: export AWS_SECRET_ACCESS_KEY=...
Someone puts export OPENAI_API_KEY=... into ~/.zshrc “temporarily” and forgets
Someone shares a terminal screenshot to Slack with the token visible
An agentic CLI tool logs the full command line, then you paste that log into a ticket

GitHub has an entire product area dedicated to stopping this class of mistake, because it happens constantly: Secret scanning / GitHub Secret Protection is designed to detect secrets committed to repos and help you remediate them (GitHub).

Also, AI tooling makes this worse. As Dwayne McDaniel of GitGuardian puts it in their write-up on agentic workflows, when tools can act across systems, “the more systems an agent can reach, the more consequential a failure in its credential and execution layer becomes” (Dwayne McDaniel).

So yes, history settings matter. But they’re not the whole story.

How do I stop a command from being saved in shell history (bash/zsh/fish)?

You’ve got three practical moves, in order of usefulness:

Use “ignore space” rules (bash/zsh). Prefix sensitive commands with a space.
Use shell-specific “don’t log this” options (zsh has more knobs).
Use a private/no-history session when you’re about to do sketchy things (fish has an explicit mode).

Here’s the cross-shell cheat sheet.

Shell	Fastest “don’t save this” option	Best persistent hardening	Where to put it
bash	Prefix command with a leading space (with `HISTCONTROL=ignorespace`)	`HISTCONTROL=ignoreboth` + `HISTIGNORE` patterns	`~/.bashrc` / `~/.bash_profile`
zsh	Prefix command with a leading space (with `setopt HIST_IGNORE_SPACE`)	`setopt HIST_IGNORE_SPACE` + `HIST_SAVE_NO_DUPS` + sane file perms	`~/.zshrc`
fish	Start `fish --private`	private sessions for sensitive work; avoid storing secrets in vars	`config.fish` (settings)

If you only do one thing today, do the leading-space rule. It’s low friction and catches the “copy/paste a curl with a token” habit.

Bash: history controls (HISTCONTROL/HISTIGNORE) and safe patterns

Bash is blunt but effective.

Add this to ~/.bashrc (or wherever your bash config lives):

# --- secrets hygiene: bash history ---
# ignoreboth = ignorespace + ignoredups
export HISTCONTROL=ignoreboth

# Keep history smaller. Big history = bigger blast radius.
export HISTSIZE=5000
export HISTFILESIZE=10000

# Don't record obvious secret-bearing commands. Tune for your stack.
export HISTIGNORE='*--password*:*--token*:*--secret*:*Authorization:*:export *KEY=*:*AWS_SECRET_ACCESS_KEY*:*OPENAI_API_KEY*:*GITHUB_TOKEN*'

# Ensure history is appended (not overwritten) in multi-shell usage
shopt -s histappend

What this does:

ignoreboth means any command starting with a space is ignored, and duplicates are ignored.
HISTIGNORE is pattern-based filtering. It’s not perfect, but it catches the “export KEY=…” and “curl -H Authorization: …” stuff.
HISTSIZE and HISTFILESIZE are boring but important. If your history file is 2 MB instead of 200 MB, you have less to scrub when something goes wrong.

One more thing people miss: permissions.

chmod 600 ~/.bash_history 2>/dev/null || true

If you’re on a shared box, “world-readable history files” is an embarrassment you don’t want.

Zsh: history controls (setopt options) and safe patterns

Zsh has more knobs, and you should use them.

Add this to ~/.zshrc:

# --- secrets hygiene: zsh history ---
# Ignore commands that start with a space
setopt HIST_IGNORE_SPACE

# Reduce duplicate noise and accidental repeats
setopt HIST_IGNORE_ALL_DUPS
setopt HIST_SAVE_NO_DUPS

# Write history incrementally so you don't lose it. (Not a security feature, just sanity.)
setopt INC_APPEND_HISTORY

# Keep history size reasonable
HISTSIZE=5000
SAVEHIST=10000

# Lock down history file permissions
HISTFILE=${HISTFILE:-$HOME/.zsh_history}
chmod 600 "$HISTFILE" 2>/dev/null || true

Two opinions here:

Do not enable SHARE_HISTORY unless you understand the tradeoff. Shared history across sessions is convenient, but it expands how quickly a bad paste propagates.
A lot of “zsh security snippets” on the internet are cargo cult. Your goal is not to collect setopts. Your goal is to make the risky path slightly annoying.

If you want one extra guardrail, add a “are you sure?” alias for the worst offenders:

alias export='echo "Don't export secrets globally. Use per-command env vars."; export'

It’s not bulletproof. It’s a speed bump.

Fish: private mode/history settings and safe patterns

Fish does you a favour here: it has a clear private mode.

Start a private session:

fish --private

In private mode, fish won’t write new history to disk. That’s exactly what you want when you’re about to test credentials, debug auth, or do anything you don’t want hanging around.

In non-private fish sessions, the “safe pattern” is less about history and more about not passing secrets on the command line in the first place. Which brings us to the thing most posts ignore.

Environment variable scoping: avoid exporting globally; use per-command/env wrappers

“Just put it in an environment variable” is one of the most misleading pieces of security advice in dev tooling.

Does putting a secret in an environment variable keep it safe? No. It changes the exposure surface.

Here’s what environment variables do (and why they bite you):

They are inherited by child processes by default.
They’re often visible to the same user via process inspection tooling.
They get copied into crash reports and debugging output more often than you’d like.
They get written into CI logs if you echo them, print env dumps, or run tools with --verbose.

So what’s better?

1) Prefer per-command environment variables

Instead of:

export OPENAI_API_KEY=...    # persists in your shell session
my-cli do-stuff

Do:

OPENAI_API_KEY=... my-cli do-stuff

That scopes the secret to that process and its children. It’s still not “secure”, but it’s shorter-lived and less likely to leak via “I forgot I exported that 4 hours ago.”

This is also the habit that pairs well with the leading-space history rule:

 OPENAI_API_KEY=... my-cli do-stuff
# ^ leading space keeps it out of history (bash/zsh with the right settings)

2) Prefer stdin or files over argv

If a tool supports reading a token from stdin or a file, take it. Command-line arguments are the worst place for secrets because they get logged, copied, and pasted.

Concrete example: if you must call curl, don’t inline the token. Put it in a header file or use an env var:

# Better than: curl -H "Authorization: Bearer ..."
TOKEN=... \
  curl -H "Authorization: Bearer $TOKEN" https://api.example.com/me

Still not perfect, but it avoids the “token is literally in the command string” scenario that gets screenshotted.

3) Reduce lifetime: unset aggressively

If you did export something, clean up:

unset OPENAI_API_KEY AWS_SECRET_ACCESS_KEY GITHUB_TOKEN

Make this muscle memory. The difference between “secret existed for 30 seconds” and “secret existed for 3 days” is huge when you’re running random CLIs.

Dotfiles hygiene: avoid committing secrets; .gitignore patterns; separate local/private files

Dotfiles are where secrets go to die.

The “I’ll just add it to my ~/.zshrc temporarily” move is how keys end up:

in a dotfiles repo
in a gist
in a coworker’s PR review
in a laptop migration tarball

My rule: dotfiles should contain loaders, not secrets.

Use patterns like:

keep local secrets in ~/.config/<tool>/secrets.env or ~/.secrets/<project>.env
source them from your shell config, but never check them into git

Example pattern:

# ~/.bashrc or ~/.zshrc
if [ -f "$HOME/.config/secrets/global.env" ]; then
  set -a
  . "$HOME/.config/secrets/global.env"
  set +a
fi

And in your dotfiles repo, be aggressive:

# Local secrets
**/*.env
**/*.env.*
.envrc
.env.local
*.pem
*.key
*.p12
*.pfx

This is also where I’ll plug a workflow stance: if you’re doing a lot of terminal-based AI tooling, treat your terminal like a log sink. I wrote a dedicated guide on AI security because these “small” leaks compound when you start using AI agents that touch multiple systems.

Use env managers: direnv workflow for per-directory secrets; mise patterns for tasks/env

This is the boring answer that is actually the right one: use per-directory environment loading so secrets exist only inside the project context.

direnv: local secrets that load on `cd` and unload on exit

direnv is an extension for your shell that can load and unload environment variables based on the current directory (direnv). It works with bash, zsh, and fish.

The pattern I recommend:

Add .envrc to your repo (but keep it non-secret)
Store real secrets in .env.local (gitignored)
Have .envrc load .env.local via dotenv

Example:

# .envrc (checked in)
dotenv_if_exists .env.local

# Optional: enforce a minimum set of vars
: "${OPENAI_API_KEY:?OPENAI_API_KEY missing}"

Then create .env.local (never commit it):

# .env.local (gitignored)
OPENAI_API_KEY=...
STRIPE_SECRET_KEY=...

Authorize once:

direnv allow

The security feature here is not “dotenv is magic”. The security feature is: when you leave the directory, the variables are unloaded. That reduces accidental reuse.

If you’re building agentic CLIs, pair this with log redaction. I’ve gone deep on that in redacting secrets in an AI coding CLI tool, because agents love printing things you wish they didn’t.

mise: define env and tasks together, but keep secrets out of `mise.toml`

mise (mise-en-place) is a “tools + env + tasks” manager. It can load per-project environment variables and read from local env files (mise).

The safe pattern is:

check in mise.toml with non-secret defaults
read secrets from a local file like .env.local

mise even shows this idea right in its docs examples: it can load env vars “from .env.local” via _.file = ".env.local".

Example:

# mise.toml (checked in)
[env]
# non-secret defaults
API_BASE_URL = "https://api.example.com"
# load secrets from a local file that is gitignored
_.file = ".env.local"

[tasks.dev]
run = "npm run dev"

Then .env.local is the same as above, and you’ve got tasks that run with the right env without teaching every dev to export things manually.

If you want to go further, I’ve already written about using both together in direnv + mise for a reproducible terminal dev environment.

Add secret scanning: pre-commit + repo scanners; GitHub secret scanning/push protection

You need two layers:

Local: stop leaks before they hit git
Remote: stop leaks before they hit main

Local scanning: gitleaks and trufflehog

I’m not going to pretend regex scanning is perfect. It’s still one of the highest ROI controls you can add in a day.

gitleaks is a popular open-source secret scanner (gitleaks).
trufflehog is another widely used tool that can also verify certain credentials (TruffleHog).

The workflow I like is:

run gitleaks (fast) on pre-commit
run trufflehog (deeper) in CI on PRs

If you want a step-by-step setup, I already published gitleaks + pre-commit + CI. Hooking scanners into your normal dev loop beats “quarterly security reminders” every time.

Also: if you’re adopting agentic tooling or “vibe coding” workflows, do not trust that logs won’t capture secrets. Put this in your AI in production checklist right next to observability and rate limits.

Remote scanning: GitHub Secret Protection and push protection

GitHub’s secret scanning exists because “we’ll catch it in code review” doesn’t work.

Here’s the official intro video from GitHub:

[YOUTUBE:vMhDkt5JNN0|Introduction to secret leaks and getting started with GitHub Secret Protection]

Enable secret scanning for repos where it’s available, and if you can, enable push protection. The goal is simple: make it hard to push a key even if someone tries.

If your team is already living on GitHub, this is a no-brainer control. It’s the closest thing to a seatbelt you can add without changing developer behaviour.

When a key leaks: rotate/revoke, scrub history, scrub git history, and notify/log review

Leaks happen. The only unacceptable move is “hope nobody noticed.”

Here’s the runbook I want you to follow the minute an API key is exposed in a terminal, repo, or log.

Step 1: Revoke or rotate immediately

Do this first. Not after cleanup.

Rotate the API key at the provider.
If it’s a cloud key, invalidate sessions/credentials where possible.
If it’s scoped (good), rotate only that scope. If it’s broad (bad), assume blast radius.

Time matters. If a secret hit a public repo for even 1 minute, you should assume it’s compromised.

Step 2: Figure out where it leaked

You need to know if this is:

terminal-only (history / screenshot / scrollback)
git (committed, PR, or pushed)
CI logs (workflow output)
issue trackers / chat (Slack, Jira)

Each has different cleanup.

Step 3: Scrub shell history locally

If the leak was “I typed it”, you have to clean:

your history file (~/.zsh_history, ~/.bash_history, fish history)
your terminal scrollback (some terminals persist it)

Practically: open the history file, remove the line(s), and consider truncating.

Also: if you used export SOME_KEY=..., search your dotfiles for it. People forget they set it in two places.

Step 4: If it hit git, rewrite history (properly)

Removing a secret from the current HEAD is not enough. You must remove it from history.

Use git filter-repo (the modern replacement for filter-branch) (Elijah Newren).

After rewriting:

force-push the rewritten history
rotate the secret again (assume it was copied)
invalidate old clones if you can (hard in practice)

If this feels extreme, good. It should. You’re paying down an incident.

For a safer day-to-day git workflow (which reduces how often you’re doing history surgery), I’ve also got a guide on advanced Git commands safely and a migration guide for jj version control if you’re experimenting.

Step 5: Prevent reintroduction

This is where most teams fail. They clean up and move on.

Do these within 24 hours:

enable GitHub secret scanning / push protection where possible
add gitleaks/trufflehog in CI
add a pre-commit hook
add .env.local and .envrc patterns to .gitignore
document the direnv/mise workflow in your repo README

If you’re using agentic CLIs, also harden logging and redaction. This intersects with LLM security and AI security more than most people want to admit.

Step 6: Audit

Even on small teams, do a minimal audit:

check provider access logs for the time window (last 24 hours at minimum)
check GitHub audit/security logs if the leak was in a repo
search Slack/Jira for the token prefix

You’re not hunting for perfection. You’re trying to answer: “Did anyone use this?”

The safest way to provide API keys to CLI tools locally (my default)

If you want my default workflow in 2026:

Use direnv for per-project env loading.
Store secrets in .env.local (gitignored).
For one-off commands, use VAR=... cmd with a leading space.
Run scanners locally and in CI.

If you do that, you’ve made the dangerous path annoying and the safe path easy.

If you’re building internal tooling, here’s my challenge: make “paste the key into the terminal” the worst UX. Accept stdin. Accept files. Integrate with a secrets manager. Your future self will thank you.

My prediction: as agentic CLIs become normal, the next wave of “secret leaks” won’t come from humans fat-fingering curl. It’ll come from tools that log too much. If you’re not treating logs as an attack surface, you’re already behind.

Originally published on kunalganglani.com

How to Set Up a Local AI Hub for Home Automation Privacy [2026]

Kunal — Fri, 04 Sep 2026 12:44:07 +0000

If you want a local AI hub to run home automation agents without cloud data sharing, you need two things: network control and proof. The prerequisite that trips people up is simple. If your hub sits on the same flat LAN as everything else, you can’t tell “local AI” from “quiet telemetry.”

This tutorial is a local ai hub home automation privacy setup you can actually verify. We’ll map what likely runs locally vs. what tends to reach the cloud (setup, updates, voice/LLM, remote access). Then we’ll segment your LAN so the hub can talk to devices but not the internet. Finally, we’ll validate outbound behavior three ways: DNS logs, packet capture/SNI, and firewall deny logs.

UGREEN is pitching HomeAgent as an “agentic AI local hub” that unifies local AI compute and storage. Cool. But I don’t trust marketing copy for privacy claims. I trust packet captures.

Here’s the official product framing if you want to see what UGREEN is promising: UGREEN Official.

What is a local AI hub?

A local AI hub is a home device that runs automation logic and AI-assisted behaviors on your own network, ideally performing inference and orchestration locally instead of sending your smart-home data to a vendor cloud.

In practice, “local” can mean at least three different layers, and each layer can have different cloud dependencies:

Hub firmware + core services: device discovery, device control, local storage, and whatever “agent runtime” the hub runs.
Mobile app onboarding + account flow: pairing, Wi‑Fi provisioning, device registration, push notifications.
AI/voice pipeline: speech-to-text, intent classification, LLM inference, tool calling, and TTS.

A hub can be local in layer 1 and still leak in layers 2 or 3.

If you’re new to the “agent” vocabulary being used in smart homes right now, skim my AI agents pillar. Smart-home hubs are basically shipping consumer-grade agent orchestration with nicer packaging.

Step 1: Map what runs local vs what might still hit cloud

When a vendor says “local AI,” I translate it into a checklist of the usual escape hatches. You’re not being cynical. You’re being accurate.

[YOUTUBE:uGEjWA9TES0|Introducing UGREEN HomeAgent]

What should be able to run locally (and how to test it)

If the product is serious about local-first, this stuff should keep working even if your ISP is having a bad day.

Automations and device control: lights, switches, scenes, schedules. This should keep working with WAN unplugged.
Local integrations: MQTT, local HTTP, LAN discovery.
Local storage: if the hub “unifies compute and storage,” recordings and logs shouldn’t evaporate the moment the internet drops.

Concrete test: after setup, you should be able to flip your WAN off for 30 minutes and see zero functional degradation for local automations.

What commonly requires cloud (even on “local-first” products)

This is where most “local” pitches quietly fall apart.

First-time onboarding: lots of products insist on account creation or cloud registration.
Firmware updates: almost always cloud.
Remote access: unless you self-host a VPN.
Voice/LLM inference: the big one. Some hubs do a local wake word, then ship your audio or text off to a cloud STT/LLM anyway.

Home Assistant is a good baseline because it’s honest about the split. The core runs locally. Optional cloud features add remote access and convenience. Nabu Casa literally sells that tradeoff on Home Assistant Cloud.

That “map what runs where” lens is also how I think about real production AI systems. Same discipline at home. List dependencies. Figure out what’s critical. Put the rest on a leash. My post on AI in production is the enterprise version of this mental model.

The privacy stance I recommend

Default position: assume everything wants to phone home until you can prove it doesn’t.

That’s not paranoia. It’s one of those things where the boring answer is actually the right one.

Step 2: Segment your LAN (so local traffic works, internet doesn’t)

If you keep your hub on the same network as your laptops, it will have accidental access to way more than it needs. Segmentation is blast-radius control. It’s also the only way your logs mean anything.

The target architecture is:

LAN (trusted): laptops, phones, servers.
IoT VLAN (untrusted): switches, bulbs, cameras, smart speakers.
Hub VLAN (semi-trusted): the “local AI hub” itself. It’s more capable than a bulb, but less trustworthy than your workstation.

If you only do one thing, do this. Put the hub on its own VLAN and block outbound internet by default.

Minimal addressing plan

Keep it boring. Boring networks are debuggable networks.

LAN: 192.168.10.0/24
IoT: 192.168.20.0/24
Hub: 192.168.30.0/24

Three subnets. Easy to reason about. Enough separation to actually see what’s going on.

Why “Hub VLAN” is worth it

A local AI hub is not a dumb endpoint. It’s a small server with storage and an agent runtime. Treat it like any other box that can run a large language model in a tool loop.

If you’ve built agents, you already know why this matters. If you haven’t, here’s the smart-home translation: don’t run your security cameras on the same network as your work laptop.

For more on how I think about agent attack surfaces, see AI security and the very real threat of prompt injection in any system that turns “natural language commands” into actions.

Step 3: pfSense rules that keep discovery working (without “allow any”)

Most IoT segmentation guides fail at the most predictable moment.

They isolate things, discovery breaks, then they add an “allow any” rule because they just want it to work. Congratulations. You now have a flat network again. You just made it harder to see what’s leaking.

Don’t do that.

Use a stateful firewall and be explicit. pfSense is a solid choice and the docs are genuinely good: pfSense Documentation.

The baseline policy

For both IoT VLAN and Hub VLAN:

Allow to local DNS (your resolver, e.g. Pi-hole)
Allow to NTP (your firewall or a local NTP server)
Allow east-west traffic only where needed (Hub ↔ IoT)
Block everything else to WAN with logging enabled

Point #4 is the whole tutorial. “Default deny” plus logs is what turns privacy into something you can audit.

The “things that usually break” allowlist

You’ll probably need a subset of these. The right list depends on your stack, but the failure modes are weirdly consistent.

DNS: UDP/TCP 53 to your Pi-hole (or Unbound)
NTP: UDP 123 (prefer a local NTP server. Don’t let every device hit random pool servers)
mDNS: UDP 5353 (multicast discovery)
SSDP/UPnP discovery: UDP 1900
MQTT (if you use it): TCP 1883 (or 8883 for TLS)
Home Assistant: typically TCP 8123 (if your hub talks to HA)
Matter: often UDP 5540 (plus multicast behavior)

That’s already seven concrete items. The move is not “open these to the internet.” The move is “open these inside your LAN segments, and nowhere else.”

If you’re running a local voice stack or any local AI components, you’ll likely also have internal HTTP ports between services. Keep them internal. Always.

Practical pfSense pattern

Create aliases: IOT_NET, HUB_NET, DNS_SERVER, NTP_SERVER, HA_SERVER
Add rules on the VLAN interface tabs, in this order:
1. Pass DNS to DNS_SERVER (log)
2. Pass NTP to NTP_SERVER (log)
3. Pass Hub ↔ IoT on required ports (log)
4. Block any to WAN (log)

Make the rules noisy at first. Logs are data.

If you want a general mental model for doing “restrict egress, then measure what breaks,” it’s the same loop I use when I build deterministic gates in this site’s multi-agent publishing pipeline. Operating that pipeline (261+ posts shipped), I learned the hard way that deterministic gates catch more issues than just upgrading to a bigger model. Same deal here. A deterministic network gate beats “trust me bro” privacy.

Step 4: Verify outbound traffic (DNS logs → SNI/pcap → firewall logs)

This is the repeatable workflow most guides skip.

You’re going to prove what the hub is attempting, even if those attempts get blocked.

4A. DNS query logs (Pi-hole)

DNS is the easiest audit trail because almost everything starts with a hostname.

Pi-hole is literally built for this. It’s a “DNS sinkhole” with a dashboard and query logging: Pi-hole documentation.

What to do:

Make Pi-hole the only DNS server your Hub VLAN can reach.
Turn on query logging.
Reboot the hub. Then do a “normal” day. Open the app, trigger an automation, try voice.
Export or screenshot the query log filtered by the hub’s IP.

What you’re looking for:

Vendor domains you expect (update CDN, auth)
Vendor domains you don’t expect (analytics, crash reporting)
Third-party telemetry (common: *.amazonaws.com, *.cloudfront.net, *.googleapis.com)

Even if you can’t interpret every domain, you can count. If your “local AI hub” is making hundreds of DNS lookups/day to external domains, it’s not behaving like an offline-first controller.

One more thing. If you suspect the hub is trying to resolve DNS over something else (DoH/DoT), treat that as a red flag. I’ll cover it in pitfalls.

4B. Packet capture + SNI with Wireshark/tcpdump

DNS tells you names. Packet capture tells you where the traffic actually goes.

Wireshark’s display filter reference is the canonical resource when you’re filtering down to what matters: Wireshark Display Filters.

What to capture:

Mirror the hub’s switch port or capture on pfSense.
Focus on outbound attempts from the hub IP.

What you can see without decrypting TLS:

Destination IPs and ports
TLS handshake metadata, often including SNI (Server Name Indication)

Useful filters to start with:

ip.addr == 192.168.30.50 (replace with hub IP)
tls.handshake.extensions_server_name (to spot SNI)
dns (to correlate DNS and connections)
quic (to spot HTTP/3 attempts)

If SNI is present, you can usually map “destination IP” back to a hostname even if DNS is doing something clever.

This is the point where you stop guessing.

4C. Firewall logs (pfSense)

Firewall logs tell you what actually got out and what got blocked.

With “block to WAN” logging enabled, you get an auditable list of:

src IP (hub)
dst IP
dst port
timestamps

If you see repeated blocks to 443 every 10 seconds right after boot, that’s usually a cloud dependency or a telemetry retry loop.

This is also the safest way to validate “it stays local” because you’re enforcing it, not just watching it.

If you want the same thinking applied to software agents, my post on agent framework selection and failure modes is the software version of this: don’t trust the happy-path demo. Build a harness. Watch what breaks.

Step 5: Offline acceptance test (prove it still works when WAN is down)

You don’t have “local-first.” You have “local-ish until the ISP sneezes” unless you test it.

Here’s the checklist I use.

Baseline: with WAN up, confirm the hub works for: 3 automations, 1 device discovery flow, and 1 voice/agent command.
Cut WAN: physically unplug WAN or disable it on your router for 20 minutes.
Power cycle: reboot the hub while WAN is down.
Run the same 5 tasks again.
Watch your logs:
- Pi-hole: do DNS queries spike?
- pfSense: do you see repeated blocked attempts to the same IPs/domains?
Restore WAN: re-enable WAN and verify the hub doesn’t “catch up” by dumping a backlog of telemetry.

Step #3 is the money step. Reboots are where devices love to register, phone home, and “validate licenses.”

If you’re building home automation local voice assistant without cloud, you can validate the voice pipeline separately too. I’ve got a full stack write-up here: local AI voice assistant.

Step 6: Firmware updates without turning your privacy model into Swiss cheese

Firmware updates are the most legitimate reason a local hub needs internet.

You still don’t have to leave the door open all day.

My recommended process:

Create a temporary “update window” rule: allow Hub VLAN egress to WAN for 60 minutes.
Schedule it. Don’t leave it on.
During the window:
- Watch Pi-hole queries.
- Watch pfSense states and logs.
After the window:
- Disable the rule.
- Reboot the hub once. Confirm it still functions offline.

If your hub breaks when you block internet again, that’s not a “privacy device.” That’s a cloud device with a local cache.

This is the same pattern I use for sensitive workflows in my own automation. One hard lesson from operating my GSC-driven SEO automation is that changes need isolation windows to attribute impact. Networking is the same. Make changes in controlled windows, observe, then lock it back down.

Step 7: Common pitfalls (and how to mitigate them)

These are the ways a “local-only” plan fails in real homes.

Hardcoded DNS (device ignores your resolver)

Symptom: Pi-hole logs show nothing, but the hub still reaches the internet.

Mitigation:

Block outbound DNS (53) to anywhere except your resolver.
Add NAT redirection rules. Force all 53 to Pi-hole.

DoH/DoT (DNS over HTTPS/TLS)

Symptom: outbound 443 to known DoH endpoints, and no readable DNS.

Mitigation:

Block known DoH endpoints.
Prefer allowlisting outbound rather than trying to maintain a denylist forever.

QUIC/HTTP3 hides some hostname cues

Symptom: lots of UDP 443 traffic (QUIC). Harder to inspect.

Mitigation:

Block UDP 443 from Hub VLAN and IoT VLAN.
Force fallback to TCP 443 where SNI and logging tend to be easier.

If you want to go deep on this, I wrote a full playbook on QUIC debugging. Same protocol. Same pain.

Certificate pinning (TLS inspection won’t help)

Symptom: you can’t MITM even if you wanted to.

Mitigation:

Don’t rely on TLS inspection for privacy guarantees.
Rely on DNS logs, destination IPs, and firewall egress controls.

“Local AI” that still uses cloud LLMs for the hard part

Symptom: voice works online only, or gets dramatically slower offline.

Mitigation:

Force an offline test.
If it’s cloud-backed, decide intentionally. Sometimes the right answer is to run your own local LLM on a box you control.

Based on the benchmark data I maintain at kunalganglani.com/llm-benchmarks, small local models can be perfectly usable for intent classification and home-control tool routing. You don’t need frontier-model intelligence to turn on a lamp. You need reliability.

My prediction: “local AI” will become a networking problem, not an AI problem

Smart-home vendors are going to ship more “agentic AI” hubs because it’s the only way they can sell privacy and speed in 2026 without admitting the cloud business model is getting toxic.

The winners won’t be the hubs with the fanciest model. They’ll be the ones that make local-only verifiable. Clear dependency maps. Offline-first UX. Admin-grade egress controls.

Until then, don’t buy the promise. Buy the device, stick it on a VLAN, and make it earn your trust.

Originally published on kunalganglani.com

Xbox Cloud Gaming Pay‑As‑You‑Go Latency: Fix Input Lag Fast [2026]

Kunal — Fri, 04 Sep 2026 00:42:18 +0000

Xbox Cloud Gaming pay‑as‑you‑go latency is the end‑to‑end delay between your button press and the pixels changing on your screen during an xCloud session. If you’re paying per hour, every “why does this feel mushy?” minute is literally billable waste, and that’s a brutal way to discover your home network has jitter or bufferbloat.

Key takeaways

If your cloud stream looks sharp but controls feel late, you’re usually fighting latency and jitter, not bandwidth.
Ethernet is the boring answer, but good 5 GHz or 6 GHz Wi‑Fi can get you close if you control interference and signal strength.
Bufferbloat is the silent killer. SQM (CAKE or fq_codel) is the most reliable fix I’ve seen for “lag spikes when someone uploads.”
You should measure baseline ping, loaded latency, and packet loss before you touch settings, then re-test after each change.
Some problems are service caps. Codec/bitrate ceilings and server distance are not things you can “tune” at home.

The recent reporting around a pay‑as‑you‑go option is what makes this worth caring about now. With a subscription, people tolerate some jank. With per-hour pricing, quality needs to be predictable.

Xbox Cloud Gaming requirements (bandwidth, devices)

Microsoft’s public guidance for Xbox Cloud Gaming tends to emphasize baseline connectivity and supported devices, and you should treat that as the floor, not the target. Bandwidth gets you a picture. Latency, jitter, and packet loss decide whether it feels like a local console.

A reasonable baseline for most cloud gaming services is:

Download throughput: 10–20+ Mbps (more headroom helps the encoder adapt without dropping frames)
Ping to the internet: ideally < 40 ms, acceptable up to 60–80 ms depending on game
Packet loss: should be 0% or close to it. Even 1–2% is noticeable on fast action
Wi‑Fi signal: if you’re wireless, target an RSSI around -60 dBm or better at the client

The point isn’t to worship any one number. It’s to avoid spikes. Cloud gaming hates variance.

If you want the official baseline expectations, start with Microsoft’s Xbox support documentation (it’s JavaScript-rendered, but it’s still the canonical source): https://support.xbox.com/en-US/help/games-apps/cloud-gaming/about-cloud-gaming

What causes input lag in Xbox Cloud Gaming beyond just low bandwidth?

Input lag in cloud gaming is a pipeline:

Controller input sampling (Bluetooth can add a few ms vs wired)
Client processing and buffering
Uplink to the data center
Game simulation and render on the server
Video encode
Downlink back to you
Video decode on your device
Display response (TV processing is often a sleeper culprit)

Even if your ISP gives you 1 Gbps, steps 3, 5, 6, 7, and 8 still dominate.

Here’s a practical mental model I use. For a “feels good” session, you want:

Network RTT in the 20–40 ms range
Encode + decode that stays roughly under 20–40 ms combined
Display in game mode often around 10–20 ms (but many TVs can be worse if game mode is off)

If any one stage adds variance, you feel it as mushy aim or late parries.

I like the way RTINGS talks about separating the chain (display lag vs everything else), even though their page is about TVs: RTINGS.

How much latency is “playable” for different game genres?

I’m going to be opinionated here. The internet loves pretending you can put a universal number on “playable.” You can’t. But you can put genre buckets on it.

Turn-based / card / tactics: You can tolerate 80–120 ms and not care much.
Action RPG / platformers: 40–80 ms is workable if it’s stable.
Racing / fighting / competitive shooters: You want < 40–60 ms end-to-end and low jitter, or it starts feeling like you’re playing through syrup.

The hidden constraint is jitter. A session that bounces between 25 ms and 90 ms feels worse than a steady 60 ms.

Wi‑Fi vs Ethernet for cloud gaming

Ethernet is still the best “setting.” It removes a whole class of problems: interference, contention, and retransmits.

That said, modern Wi‑Fi can be good enough if you do it intentionally.

The Wi‑Fi rule I trust

If your client is on 2.4 GHz, you’re already in the danger zone. 2.4 GHz is crowded, has fewer clean channels, and it’s more likely to deliver micro-spikes that cloud gaming punishes.

5 GHz is usually the sweet spot. 6 GHz (Wi‑Fi 6E/7) can be even better if both your router and device support it, because there’s typically less congestion. Cisco’s overview is a decent high-level reference for why 6 GHz can reduce interference and improve latency: Cisco.

Practical Wi‑Fi settings that actually matter

If you’re trying to reduce xCloud input lag, these are the knobs that move the needle:

Band steering: Good in theory, but I often prefer explicitly connecting the gaming device to 5 GHz or 6 GHz to avoid “helpful” roaming.
Channel selection: Don’t leave it on auto forever. If you live in a condo, scan and pick a cleaner channel.
Channel width: Wider (80/160 MHz) can be faster, but it can also be less stable in congested areas. Stability beats peak throughput for cloud gaming.
DFS channels (5 GHz): Can be great if your router supports it and your environment doesn’t trigger radar events. If DFS kicks you off mid-session, you’ll know.
Mesh backhaul: If your “Wi‑Fi” is actually a wireless mesh hop, your latency budget just got worse. Use wired backhaul if you can.
WMM (Wi‑Fi Multimedia): Leave it on. Disabling it often breaks QoS behavior.

A quick sanity check: if you can, put the gaming device and router in the same room for a test. If the lag disappears, you don’t have an “internet” problem. You have a Wi‑Fi problem.

QoS/SQM and bufferbloat fixes (the big win)

Bufferbloat is the undesirable latency that comes from a router or modem buffering too much data. The one-line summary from Bufferbloat.net is basically the story: “Bloated buffers lead to network-crippling latency spikes.” When your upstream is busy, your ping explodes.

Source: Bufferbloat.net.

This is why cloud gaming feels fine at 2am and horrible at 7pm. Not because the raw Mbps changed. Because someone in your house started uploading photos, a laptop started a backup, or your phone decided it was iCloud-o’clock.

What to do about it: SQM (CAKE / fq_codel)

Smart Queue Management (SQM) is the only consumer-network fix I consistently recommend for real-time apps. It works by shaping traffic so your router becomes the bottleneck (on purpose), then managing that queue intelligently.

OpenWrt’s SQM docs are unusually practical about this: measure first, then set shapers slightly below your real line rate so the router, not the ISP gear, controls the queue. Source: OpenWrt SQM documentation.

Concrete starting numbers that work for a lot of people:

Set download shaper to 90–95% of your tested download speed
Set upload shaper to 85–95% of your tested upload speed (upload is often the pain point)

If you have 300/20 Mbps, you might start around 270/18 Mbps and iterate.

Common SQM pitfall

SQM burns CPU. If your router can’t shape at your full bandwidth, you’ll trade bufferbloat for throughput loss. That’s usually a good trade for cloud gaming, but you should be aware of it.

Also, SQM can conflict with hardware flow offloading. OpenWrt explicitly calls this out.

How to test/measure latency (baseline vs under load)

If you’re paying per hour, measurement is the whole point. Otherwise you’ll spend money “trying random stuff.”

The 15-minute measurement plan

Do this before you tweak anything. Write the numbers down.

Idle ping (baseline): Ping a stable target (your ISP gateway if you know it, or a reliable public host). Take the median of 50–100 pings.
Jitter: Look at the spread. A tight cluster (e.g., 18–22 ms) is good. Spikes to 60+ ms are not.
Loaded latency: While running a download and an upload (or a speed test), repeat the ping test. This is where bufferbloat shows up.
Packet loss: Even 0.5–1% can feel bad. Many tools will report it.

If you want a single “loaded latency” grade, Waveform’s bufferbloat test is a decent quick indicator. OpenWrt’s docs link to it in their prep section.

Measure while actually playing

The hard part is correlating network behavior with “that moment where aiming felt wrong.”

Practical options:

If you’re on a PC, run a continuous ping in the background and timestamp spikes.
Watch your router stats during a session. If you see upload saturation at the exact time input feels delayed, that’s your smoking gun.

You don’t need lab gear. You need timestamps and repeatability.

Service caps vs home network problems (what you can’t fix)

This is the part most cloud gaming guides skip, and it drives people crazy.

Some problems are on you:

Wi‑Fi interference
weak RSSI
mesh hops
bufferbloat
bad router firmware

Some problems are on the service:

Server distance: physics is undefeated. If the nearest data center is far, your floor is higher.
Capacity / congestion: if the platform is oversubscribed in your region, you can see stutters that look like “network.”
Codec/bitrate ceilings: many streaming clients cap bitrate or resolution profiles. If the service caps your stream, buying faster internet won’t make the image cleaner.

The practical implication for pay‑as‑you‑go is simple: do a 10-minute test session on a wired connection. If the experience still isn’t good enough, stop. Don’t sink money into a new router hoping it fixes what is really geography.

Is it your TV/monitor or the network? (isolating display lag)

I’ve seen people blame “latency” when the real culprit was their TV doing image processing.

Do this isolation sequence:

Enable Game Mode (or equivalent low-latency mode) on your display.
If possible, test on a computer monitor known for low input lag.
Swap connection types: Ethernet vs Wi‑Fi. If Ethernet fixes it, your display is probably fine.
If both feel bad, you may be hitting server distance or service-side variance.

Display lag numbers vary wildly. Many modern TVs can be around 10–20 ms in game mode, but can be much higher outside it.

A quick troubleshooting table (symptom → measurement → fix)

Symptom you feel	What to measure	What usually fixes it
Smooth most of the time, but huge lag spikes when someone uploads	Loaded latency during upload	Enable SQM (CAKE/`fq_codel`), lower upload shaper to ~90%
Constant “mushy” input even when the network seems fine	Display mode / input lag	Game Mode, disable motion smoothing, try a monitor
Random stutter on Wi‑Fi, worse at night	RSSI, channel congestion	Move to 5/6 GHz, change channel, reduce channel width
Video looks blocky during action scenes	Throughput headroom, Wi‑Fi retries	Ethernet, stronger Wi‑Fi, reduce interference
Fine on Ethernet but bad on mesh	Hop count / backhaul type	Wired backhaul, move node, or stop using mesh for gaming

My stance: if you’re paying per hour, treat your network like a budget

This is one of those things where the boring answer is actually the right one. If Xbox Cloud Gaming moves toward pay‑as‑you‑go, the “hidden cost” isn’t the hourly rate. It’s variance.

So here’s my prediction. We’re going to see a split: casual players will accept cloud gaming as a good-enough Netflix-for-games experience, and everyone else will either hardwire their setup or bounce back to local hardware. Per-hour billing will make that decision faster because it forces you to notice the tax.

If you try one thing after reading this, don’t buy gear. Measure your baseline and loaded latency, then turn on SQM. The first time you play through a household upload without a latency cliff, you’ll never unsee it.

Originally published on kunalganglani.com

Polars 2.0 Upgrade Guide [2026]: Streaming Default + CI Bench

Kunal — Thu, 03 Sep 2026 12:43:03 +0000

Polars 2.0 Upgrade Guide: Breaking Changes, Streaming Default, and a CI Regression Harness

One pip install -U polars later, your tests start failing. Not because your logic is wrong. Because your rows came back in a different order.

If you already searched for “polars 2.0 upgrade guide”, that’s probably why you’re here.

You’ll leave this guide with two things working:

1) your codebase running on Polars 2.0 (or the 2.0 RC) without “mystery diffs”, and 2) a repeatable regression harness that measures wall time, CPU time, and peak memory (RSS) for your real queries.

Polars 2.0 is a rare “good breaking change”. It makes the fast path the default. The price is that you now have to be explicit about correctness requirements you were accidentally getting for free.

What is Polars 2.0 (and why does the upgrade feel scary)?

Polars 2.0 is a major version of the Polars DataFrame library that changes defaults and tightens type behavior, most notably by running LazyFrame queries on the streaming engine by default.

The anxiety comes from one specific sentence in the official announcement. As Ritchie Vink (creator/maintainer of Polars) explains, in Polars 2.0 calling collect() on a LazyFrame now defaults to the streaming engine, and that engine does not guarantee row order for some operations unless you opt in.

That’s the trade.

You get real wins in memory and throughput. The announcement claims the streaming engine can be “easily 5x faster in aggregate” for many lazy queries.
You lose the comforting illusion that output order will “probably” match input order for join, group_by, unpivot, and friends.

If you have order-dependent tests, dashboard diffs, or downstream code that quietly assumes “left table order survives a join”, you will feel this upgrade.

Streaming engine as default: the Lazy API now streams unless you stop it

The headline breaking change from the upgrade docs is literal: “The Lazy API defaults to the streaming engine” (Polars 2.0-rc upgrade guide).

Concretely:

Polars 1.x: lf.collect() typically used the in-memory engine unless you opted into streaming.
Polars 2.0: lf.collect() with engine="auto" resolves to streaming by default.

How do I keep old behavior (in-memory engine) for `collect()`?

You have three levers. Use the smallest hammer that fits.

1) Per call (most explicit):

out = lf.collect(engine="in-memory")

2) Per query chain (useful when you have a couple of known order-sensitive pipelines):

out = (
    lf
    .join(other, on="k", how="left")
    .collect(engine="in-memory")
)

3) Process-wide engine affinity (fine for a staged rollout or a notebook session):

import polars as pl
pl.Config.set_engine_affinity("in-memory")

That last one is straight out of the announcement example from Ritchie Vink.

My opinion: don’t flip global affinity in production just to keep tests passing. That’s a safety blanket, not a fix. The whole point of Polars 2.0 is that streaming is the default for a reason.

When does streaming change row order, and how do I maintain order?

Row order changes when the streaming engine parallelizes or re-chunks data in a way that’s valid for the result but not stable relative to the input.

As Ritchie Vink calls out, row order is not guaranteed for certain operations like:

join
group_by
unpivot

Why did my join/group_by output order change after upgrading?

Because you were relying on an implementation detail.

In Polars 1.x, a lot of pipelines looked stable because the in-memory engine often preserved a left-to-right vibe. With streaming as the default, the engine is allowed to reorder during joins and aggregations to hit performance and memory goals.

If you need stable, observable order, Polars 2.0 makes you say so.

Maintain order only where it matters

For joins, opt in with maintain_order:

out = (
    lf
    .join(other, on="k", how="left", maintain_order="left")
    .collect()
)

That mirrors the official example in the 2.0 pre-release post.

For aggregations, you have a few options depending on what you mean by “order”:

If you want deterministic output ordering for group keys, explicitly sort:

out = (
    lf
    .group_by("country")
    .agg(pl.col("revenue").sum().alias("rev"))
    .sort("country")
    .collect()
)

If you want “preserve input order semantics”, stop using a group_by output order as a proxy for that. Decide what the ordering rule actually is (timestamp, key priority, revenue descending, whatever) and encode it.

A practical rule I use in code review: if a test asserts row order after a join/group_by, it must also assert an explicit ordering rule. Otherwise you’re testing vibes.

Breaking changes checklist: what actually breaks in real codebases

The upgrade docs are exhaustive. Your team doesn’t need exhaustive. Your team needs a short list that prevents the three dumbest failure modes.

Here’s the path I’d take if I were upgrading a production pipeline.

Pin versions and test the RC first. GitHub releases already list Python Polars 2.0.0-rc.1 and the last 1.44.x line (Polars releases). Don’t “pip install -U polars” blind.
Audit order-dependent tests (joins, group_bys, unpivot). Fix them with maintain_order or explicit .sort().
Stop assuming pl.read_csv is purely eager (it now dispatches to scan + collect).
Replace LazyFrame.profile() usage.
Run your regression harness on representative datasets. Block merges if you cross a threshold.

That’s the work. Everything else is cleanup.

CSV reading changes and eager/lazy surprises (read_* now dispatches to scan_*().collect())

IO is where upgrades love to hide landmines.

Polars 2.0 changes eager reads:

pl.read_csv is now dispatched to pl.scan_csv(...).collect() (upgrade guide).
pl.read_ipc is now dispatched to pl.scan_ipc(...).collect() for all non-use_pyarrow inputs (same doc).

What should I change if my code uses `pl.read_csv` / `pl.read_ipc`?

If your code was relying on:

file-like object behavior
“eager read happens now” side effects
subtle schema inference timing

…make the intent explicit.

If you actually want eager semantics

You can keep read_csv, but mentally treat it as “implemented via lazy scan + collect now”. If you need to control streaming vs in-memory behavior, do it explicitly via the lazy path:

df = pl.scan_csv(path).collect(engine="in-memory")

If you want the benefits of lazy planning

Lean into scanning and do the obvious pushdowns before you collect:

lf = pl.scan_csv(path)
# add filters/projections before collect
out = (
    lf
    .filter(pl.col("status") == "active")
    .select(["user_id", "country"])
    .collect()
)

Also note a smaller CSV change from the docs: infer_schema_files default is now 10. If your dataset has weird “rare” types that only show up after file 10, you can absolutely see schema shifts unless you pin schema or bump that number.

Stricter Polars: type coercion, concat/union, and removed casts

Polars has always been strict. Polars 2.0 is stricter, and I’m fine with that.

The official announcement frames it as “fail fast” rather than letting pipelines run for 20 minutes before raising (same pre-release post).

From the upgrade guide, the big “stricter” buckets you’ll actually notice:

`is_in()` strict coercion (lossy casts no longer happen)

Polars 2.0 makes coercion casts for is_in() strict instead of lossy. If you were comparing values across types and relying on silent conversion, expect failures.

Fix: normalize types up front. This is boring work. It’s also the correct work.

`pl.concat()` / `pl.union()` strict behavior

The upgrade guide calls out “Update the strict behavior of pl.concat()/pl.union().” If you were concatenating frames with mismatched schemas and expecting a “best effort”, Polars will now force you to be explicit.

Fix: align schemas (add missing columns, cast dtypes) before concat.

Casts and operations no longer supported

This is the “you were doing a thing that looked convenient but wasn’t safe” section.

Examples called out in the upgrade guide:

disabled casting from integers to categoricals (and back)
removed casts from string to temporal types
disallowed casting from non-nested types to pl.List(..)
disallowed boolean operators between booleans and integer types

Yes, it’s painful. But it’s also removing the kind of footguns that create silent data corruption and then eat your weekend.

Raising informative errors: why this helps upgrades (even when it annoys you)

Polars 2.0 treats “Raising informative errors” like a feature, not an afterthought (it’s a full section in the announcement).

In practice, that means:

errors earlier in planning rather than during execution
clearer type mismatch messaging
fewer “it returned something, but not what you thought” situations

I’m pro this change.

I’ve built internal developer tooling (including the SOC 2 scaffolding CLI at Rise People that was adopted org-wide). “Force the right thing early” beats “debug the wrong thing later” every time. The upgrade pain is front-loaded, but the operational cost drops.

Build a repeatable performance regression benchmark harness (wall, CPU, memory)

Here’s the part most upgrade guides conveniently skip. You don’t just want “it seems faster on my laptop.” You want a harness you can run before and after the upgrade, with numbers you can gate in CI.

You want to measure three things:

Wall time: “how long did it take end-to-end?”
CPU time: “did we burn more CPU to get that wall time?”
Peak RSS: “did memory spike and put us into OOM territory?”

Harness design: compare versions, not vibes

A decent harness:

pins exact versions (ex: polars==1.44.1 vs polars==2.0.0-rc.1)
runs the same query set on the same dataset
warms up once (JIT, caches, and OS page cache will lie to you)
emits machine-readable output (JSON) and a human summary (Markdown)

Here’s a minimal but real harness you can drop into a repo.

1) Repo layout

bench/
  queries.py
  run.py
  datasets/
    README.md
  baselines/
    polars-1.44.1.json
    polars-2.0.0-rc.1.json

2) Define representative queries (Lazy on purpose)

# bench/queries.py
import polars as pl


def q1_filter_project(path: str) -> pl.DataFrame:
    return (
        pl.scan_csv(path)
        .filter(pl.col("status") == "active")
        .select(["user_id", "country", "revenue"])
        .collect()
    )


def q2_join_agg(left_path: str, right_path: str) -> pl.DataFrame:
    left = pl.scan_csv(left_path)
    right = pl.scan_csv(right_path)

    return (
        left
        .join(right, on="user_id", how="inner")
        .group_by("country")
        .agg(pl.col("revenue").sum().alias("rev"))
        .sort("country")
        .collect()
    )

Stats anchor: you’re defining 2 queries here. In real teams I like starting with 5–12 queries, but two is enough to get the harness mechanics correct and prove you can trust the output.

3) Measure wall, CPU, and peak RSS

# bench/run.py
import json
import os
import platform
import time
from dataclasses import asdict, dataclass

import psutil
import polars as pl

from bench.queries import q1_filter_project, q2_join_agg


@dataclass
class RunResult:
    name: str
    polars_version: str
    wall_seconds: float
    cpu_seconds: float
    peak_rss_mb: float


def _rss_mb(p: psutil.Process) -> float:
    return p.memory_info().rss / (1024 * 1024)


def measure(fn, *, name: str) -> RunResult:
    proc = psutil.Process(os.getpid())

    # Warm-up run to reduce one-time effects
    fn()

    start_wall = time.perf_counter()
    start_cpu = time.process_time()
    peak = _rss_mb(proc)

    out = fn()
    # Touch output so lazy work can't be optimized away
    _ = out.height

    end_wall = time.perf_counter()
    end_cpu = time.process_time()
    peak = max(peak, _rss_mb(proc))

    return RunResult(
        name=name,
        polars_version=pl.__version__,
        wall_seconds=end_wall - start_wall,
        cpu_seconds=end_cpu - start_cpu,
        peak_rss_mb=peak,
    )


def main():
    data_users = os.environ.get("BENCH_USERS_CSV", "data/users.csv")
    data_orders = os.environ.get("BENCH_ORDERS_CSV", "data/orders.csv")

    results = [
        measure(lambda: q1_filter_project(data_users), name="q1_filter_project"),
        measure(lambda: q2_join_agg(data_orders, data_users), name="q2_join_agg"),
    ]

    payload = {
        "polars": pl.__version__,
        "python": platform.python_version(),
        "os": platform.platform(),
        "cpu_count": os.cpu_count(),
        "results": [asdict(r) for r in results],
        "timestamp": int(time.time()),
    }

    print(json.dumps(payload, indent=2))


if __name__ == "__main__":
    main()

Concrete numbers this harness always emits:

cpu_count (e.g., 8, 12, 32)
timestamp (Unix seconds)
per-query wall_seconds, cpu_seconds, peak_rss_mb (floats)

That stat density is intentional. If you can’t put numbers on it, you can’t enforce it.

Add a regression gate (fail CI on thresholds)

Now the important part. Turn metrics into a decision.

Here’s a simple “compare against baseline” script. It fails if wall time regresses by more than 15% or peak RSS by more than 20%.

# bench/compare.py
import json
import sys


def load(path: str):
    with open(path, "r", encoding="utf-8") as f:
        return json.load(f)


def index_by_name(payload):
    return {r["name"]: r for r in payload["results"]}


def main():
    baseline_path = sys.argv[1]
    current_path = sys.argv[2]

    baseline = index_by_name(load(baseline_path))
    current = index_by_name(load(current_path))

    wall_budget = 0.15
    rss_budget = 0.20

    failed = False

    for name, cur in current.items():
        base = baseline[name]

        wall_ratio = cur["wall_seconds"] / base["wall_seconds"]
        rss_ratio = cur["peak_rss_mb"] / max(base["peak_rss_mb"], 1e-9)

        if wall_ratio > (1 + wall_budget):
            print(f"FAIL {name}: wall {wall_ratio:.2f}x")
            failed = True

        if rss_ratio > (1 + rss_budget):
            print(f"FAIL {name}: rss {rss_ratio:.2f}x")
            failed = True

    sys.exit(1 if failed else 0)


if __name__ == "__main__":
    main()

This is the same philosophy I use for AI in production: don’t rely on “someone will notice” after the upgrade. Build a gate.

How do I run the harness in CI and store baselines?

A pragmatic pattern:

store baselines in-repo in bench/baselines/ for small teams
store baselines in object storage for larger teams (so you can compare across branches and runners)
always log machine metadata so you don’t compare laptop runs to CI runs and call it science

If you’re already on a monorepo and you care about reproducible Python envs, use uv to pin and create two environments. My setup guidance is in CI/CD heavy workflows like my Python uv workspace monorepo.

Compare query plans across versions (logical vs physical) to explain perf changes

Polars gives you good tooling, but Polars 2.0 changes a default that matters when you’re diffing plans:

show_graph() default plan_stage changes to "physical" in 2.0 (upgrade guide).

So if you were used to staring at an optimized logical plan, you might now be staring at a physical plan and thinking “why does this look totally different?” That’s not Polars being weird. That’s you looking at a different stage.

A plan-diff workflow that doesn’t lie

I do this in two passes.

1) Capture the optimized logical plan (if you care about logical-level rewrites).
2) Capture the physical plan (because execution differences show up here, especially with streaming).

Example:

# capture_plans.py
import polars as pl

lf = (
    pl.scan_csv("data/orders.csv")
    .filter(pl.col("status") == "active")
    .group_by("country")
    .agg(pl.len().alias("n"))
)

print("OPT LOGICAL")
print(lf.explain(optimized=True))

print("PHYSICAL")
print(lf.explain(optimized=True, engine="auto"))

Numbers to keep constant during plan comparisons:

Polars version (ex: 1.44.1 vs 2.0.0-rc.1)
dataset size (rows and file size)
engine selection (engine="auto" vs engine="in-memory")

If performance changes, you should be able to point at a physical operator that changed. “It got slower” is a complaint, not a diagnosis.

API reshapes, deprecations, and the `LazyFrame.profile()` replacement

Two things worth calling out because they show up in real repos.

`LazyFrame.profile()` is removed

Polars 2.0 removes LazyFrame.profile() (upgrade guide).

What replaces `LazyFrame.profile()` in 2.0 and how do I profile queries now?

There isn’t a 1:1 replacement that gives you the exact same shape.

What I do instead:

use explain() and physical plan inspection to validate expected operators
wrap your .collect() in the harness above to get CPU, wall, peak RSS at the query boundary
for deep dives, use system profilers (Linux perf, macOS Instruments) rather than library-level profiling

That last point annoys people because it’s more work. It’s also how you get answers you can actually trust.

Removal of deprecated functionality / Deprecations

The upgrade guide has a long list of removals and deprecations. Don’t read it like a novel. Grep your codebase and fix what you actually use.

I’d prioritize:

anything in IO (because behavior changes can be subtle)
anything in casting (because correctness changes can be quiet until they explode)
anything in selectors/expressions (because it can break at runtime)

My upgrade stance (and a prediction)

Polars 2.0 is the right kind of breaking change. It makes the performance engine the default and forces engineers to state correctness requirements instead of inheriting them from accidental behavior.

But teams that treat data pipelines like “scripts” are going to have a bad month. This upgrade punishes the exact habits that feel fine when you’re hacking and become expensive when you’re operating.

If you want to stay ahead of it, do this: ship the regression harness, pin a baseline on 1.44.x, and start testing against 2.0.0-rc.1 now. When 2.0 GA lands, you won’t be debugging row order in a panic. You’ll be reading a JSON diff.

And my prediction: within 6–12 months, “order-dependent joins” in Polars will be treated like relying on dict ordering in old Python. Everybody did it. Then everybody learned to stop.

Originally published on kunalganglani.com

Execution Trace Tree for AI Agents: Build One in 60 Minutes

Kunal — Thu, 03 Sep 2026 00:43:03 +0000

Execution Trace Tree for AI Agents: Build One in 60 Minutes

Last week I watched someone try to debug an agent by tailing logs and “just increasing verbosity.” It produced 40,000 lines of JSON and exactly zero insight.

You’re going to end this tutorial with an execution trace tree for AI agents that you can:

capture on every run (LLM calls, tool calls, retrieval, memory, guardrails)
replay from checkpoints without guessing
diff run A vs run B and point at the exact node where behavior diverged
export into your existing tracing stack via OpenTelemetry

Give yourself ~60 minutes if you already have an agent loop running.

2026 is when this stopped being optional. People are finally writing about tracing agent workflows with systems like AWS X‑Ray, and the “why does it fail only in prod?” stories all rhyme. Raw logs don’t show causality. A structured trace tree does.

I’ll be blunt: if you’re building AI agents for anything real, log-only debugging is malpractice. You need a deterministic model of what happened.

What is an execution trace tree for AI agents?

An execution trace tree for AI agents is a structured parent/child history of a single agent run. Every step that can change behavior (LLM call, tool call, retrieval, memory read/write, guardrail decision, human approval) becomes a node with enough metadata to reproduce the run and compare it to another run.

It’s “distributed tracing,” but pointed inward at your agent loop instead of outward at your microservices.

In a web request, fan-out is usually HTTP calls. In an agent run, fan-out is weirder: a planning step branches into tool calls, retrieval, retries, “let me ask again with a different prompt,” and sometimes a human gate.

Two rules make trace trees actually useful (instead of “yet another telemetry thing”):

Every node is a span. Timing, parent, attributes. Same mental model as OTel.
Every node is replayable. Persist the minimum inputs/outputs (or hashes + artifact references) so you can restart from checkpoints.

If you already read my take on vendor-neutral observability, this is the debugging model that sits underneath the dashboards: How to Build Vendor-Neutral LLM Observability Monitoring [2026].

Why logs aren’t enough for agent debugging (and when they actively mislead you)

When agents get flaky in production, teams almost always do the same thing first. Crank log volume up by 10x. It feels responsible. It also turns your on-call into archaeology.

Here’s what log-only systems consistently fail at in agentic systems.

1) Causality beats chronology

Logs give you a timeline. Agents fail because of dependencies.

A classic failure mode: the agent generates a tool call with a slightly different argument shape. The tool returns a subtly different payload. That changes retrieval. That changes the final answer.

In logs, this is 600 lines sprinkled across async workers and retries.

In a trace tree, it’s one path you can walk:

plan → tool.search → retrieval → answer

That’s the difference between “I think it was the search tool?” and “this exact search result changed, and everything downstream followed.”

2) Async fan-out kills “grepability”

Real agent runs do parallel tool calls, retries, background evaluators, and queues. Once you have concurrency, grepping logs becomes a coping mechanism.

If you’ve ever fought broken tracing context in a web stack, you know the smell: traces fragment, or worse, leak across requests.

Ahmed Mahmoud documented this kind of context leakage in Next.js instrumentation. Agents are even more sensitive because the “request” isn’t a single handler. It’s a long-running orchestration with lots of side quests.

3) Diffing runs is basically impossible

Most agent regressions aren’t hard failures. They’re the annoying ones:

“It answered differently today.”
“It used to call the tool. Now it doesn’t.”
“It started citing the wrong doc.”

To debug that, you need a comparison across runs:

model changed (gpt-4.1 → gpt-4.1-mini)
temperature changed (0.2 → 0.7)
tool output changed (external API returned different data)
retrieval docs changed (index update)

A trace tree makes those changes obvious and diffable. Logs don’t.

If you care about production AI, this is the line between demo debugging and actual engineering.

Spans, parent/child relationships, and how they become an execution tree

Distributed tracing already solved the shape problem. A trace is a collection of spans. Spans have IDs and parent IDs. That gives you a tree.

OpenTelemetry (OTel) standardized the model and the plumbing. You don’t need to adopt every piece of OTel on day one. You do need to stop inventing your own half-trace format that can’t join the rest of your telemetry.

If you’re already instrumenting services, your agent trace should connect to the same trace graph as:

the API request that triggered the agent
the database calls the agent made
the queue workers it fanned out to

That only happens if you treat the agent run as a first-class trace, not “some logs in a side table.”

A minimal agent run trace is usually 20–200 spans per run. That’s not a vibe. That’s a planning number. If your agent does 8 tool calls with 2 retries each and a planner loop of 6 steps, you’re already there.

If you want the broader “agent observability stack” context, I wrote about wiring this into exporters here: OpenTelemetry Instrumentation for AI Agents [2026]: Ship It.

A minimal trace-tree schema (node types + required fields)

This is the part most posts dodge. They say “add tracing” and call it a day. Then you implement it, and six weeks later you realize you didn’t record the one field that would have explained the incident.

My bias: a good schema is:

small enough you’ll ship it
strict enough that diffing and replay aren’t a fantasy
compatible with OpenTelemetry span attributes

Node (span) types you should support

You don’t need 30 event types. You need the ones that change control flow or create nondeterminism.

root.run — one per agent run
router/planner — “what should we do next?” decisions
llm.call — model invocation (system prompt + messages)
tool.call — anything with side effects (HTTP, DB, shell, email)
retrieval — anything RAG-like (query, topK, doc IDs)
memory.read / memory.write — long-term memory and state
guardrail — allow/deny/redact decisions
human.approval — HITL gates
checkpoint — replay boundary

This maps cleanly to how most agent framework runtimes behave.

If your agent does RAG, make retrieval a first-class span. Otherwise you’ll be stuck arguing about “why did it cite that doc?” with nothing but vibes.

Required fields per span

Here’s the compact table I use when I’m designing instrumentation.

Span type	Required fields (minimum)	Why it matters
root.run	`run_id`, `agent_version`, `input_hash`, `user_id_hash`, `start_ts`, `end_ts`	identity, grouping, compliance
router/planner	`policy_version`, `decision`, `candidates`, `reason_code`	explains control flow
llm.call	`provider`, `model`, `temperature`, `max_tokens`, `prompt_hash`, `messages_hash`, `tool_schema_hash`, `response_hash`, `input_tokens`, `output_tokens`	replay + cost + drift
tool.call	`tool_name`, `tool_version`, `args_hash`, `result_hash`, `side_effect=true/false`, `status_code`, `latency_ms`	determinism boundaries
retrieval	`index_version`, `query_hash`, `top_k`, `doc_ids`, `reranker_version`	explains context changes
memory.*	`store`, `key_hash`, `value_hash`, `op=read/write`	state drift
guardrail	`rule_id`, `decision`, `redaction_count`	safety + debugging
human.approval	`approver_role`, `decision`, `wait_ms`	latency + governance
checkpoint	`checkpoint_id`, `state_hash`, `artifact_refs`	restart points

Two concrete limits I’d enforce:

cap any single span attribute blob to 16 KB (anything larger belongs in an artifact store)
cap the total persisted trace payload per run to 256 KB before sampling kicks in

If you ignore this, you’ll accidentally turn “observability” into your newest cost center. If you’re thinking about LLM cost, trace payload size is part of that equation.

For more on metrics you can attach to these spans (latency, cost, quality), see: How to Pick LLM Application Observability Metrics [2026].

Checkpoints, deterministic replay, and trace diffing (the workflow that actually saves you)

Most teams stop at “we can see the spans.” That’s table stakes. The payoff is replay + diff.

Step 1: Define what “deterministic” means for your agent

Your agent is deterministic if, given the same:

prompt/messages
tool schema
retrieval corpus + index version
tool outputs (or captured tool responses)
model + sampling params

…it produces the same downstream actions.

That’s why I treat tool.call as the determinism boundary. External APIs change. Time changes. Randomness changes. Your “perfect prompt” is not the thing that makes the run reproducible.

Step 2: Add checkpoints at control-flow boundaries

Put a checkpoint span after events that are expensive to recompute or likely to fork behavior.

My default list (5 checkpoints):

after initial user input normalization
after planning/router decision
after retrieval context is assembled
before any irreversible tool side effect (side_effect=true)
after final answer is generated

That’s enough to replay most bugs without re-running the entire world.

If you’re already thinking in control-flow terms, this pairs nicely with: AI Agent Control Flow Patterns [2026]: Retries, HITL, Checkpoints.

Step 3: Capture “tool snapshots” for replay

To replay safely, you have two options:

record-and-replay: store the tool output (or a reference to it) and return it during replay
sandbox replay: re-run tools in a sandbox with fixed fixtures

In practice, you’ll do both:

record-and-replay for third-party APIs
sandbox replay for your own services

This is where workflow engines earned their reputation. Temporal’s whole pitch is replaying workflow code against an event history. Even if you’re not using Temporal, the determinism concept transfers cleanly.

If you want the full workflow-engine framing: Temporal Workflow Engine: The Reliability Layer Your Distributed System Is Missing [2026 Guide].

Step 4: Diff two runs node-by-node

Diffing is the killer feature. The only question that matters in a regression is: what changed?

A practical diff algorithm:

align root spans by agent_version and input_hash
walk the tree in execution order
compare each node’s *_hash attributes
stop at the first divergence and expand the subtree

What you’ll usually find:

retrieval.doc_ids differs because the index version changed
tool.result_hash differs because an API returned new data
llm.call.model differs because routing changed
llm.call.temperature differs because someone “tuned creativity”

That’s the moment debugging becomes boring. Boring is good.

This plugs straight into evals too. I built this site’s publishing pipeline as a multi-step, idempotent system with deterministic gates. The lesson transferred cleanly: deterministic structure beats “more model.” Based on the pipeline I run for this site (261+ posts published and weekly automated feedback loops), deterministic gates catch failures earlier than just upgrading the review model.

You can tie trace diffing into your eval workflow here: Agent Evaluation Harness [2026]: Replay, Rubrics, CI Gates and AI Engineering Evals: Regression Gates for Prompts, Tools, RAG [2026].

OpenTelemetry context propagation, redaction, sampling, and cost control

This is where most “agent tracing” tutorials die in production. The demo works. The first incident hits. Context fragments, or you accidentally store secrets, or your tracing bill starts looking like a second LLM bill.

Context propagation: don’t let your trace fragment (or leak)

If you have async tool calls, queues, or background workers, you need to propagate context.

Hard requirements:

a stable trace_id for the whole run
a new span_id per node
propagation across process boundaries (HTTP, queues)

When context propagation breaks, you get:

orphan spans
partial traces
cross-run contamination (the scariest one)

That’s why Ahmed Mahmoud’s Next.js story matters here. The exact same bug class shows up in agent orchestrators when spans are stored in globals or threadlocals incorrectly.

If you’re building agent infrastructure, you want an orchestration layer that makes context explicit, not implicit. See: AI Agent Observability Logging Schema [2026]: OTel + Redaction.

Redaction and security: store less than you think

Agent traces contain the worst kind of data:

user input (often PII)
prompts (often secrets and system rules)
tool outputs (often internal data)

A sane baseline:

store raw prompts only in dev, never in prod
store hashes in prod by default (prompt_hash, messages_hash)
store structured “safe excerpts” only after redaction
keep artifact blobs in a separate store with retention controls

This is also where prompt injection becomes an observability problem. If your trace store is searchable and you persist raw tool outputs, you created a new exfiltration surface.

If you’re building RAG-heavy systems, the redaction patterns carry over directly from: How to Implement Field-Level Redaction for RAG Pipelines [2026].

And if your team needs the broader posture: AI Security Leader Playbook [2026]: 10 Controls That Ship.

Sampling and cost control: trace trees can get expensive fast

My default:

keep 100% of traces for failures
keep 10% of traces for successes
keep 1% of traces for high-volume endpoints

Attach token/cost metadata at the llm.call spans so you can see which node blew the budget.

If you want to go deeper on cost math, start here: Agent Per-Task Cost Calculation [2026]: Retries, Tools, Caching.

Data anchor (site-owned): the GSC keyword winnability dataset I run for kunalganglani.com shows this topic neighborhood has ~1,083 related impressions already and a demand estimate around ~11,000 searches/month across related agent observability queries. That’s the clearest signal I have that “agent debugging models” are actively being searched for.

Using trace trees for evaluation (regressions, scorecards, and alerts)

Once every run is a tree, evaluation stops being hand-wavy.

Three concrete ways I’d wire it:

Label traces: pass/fail, task_type, customer_tier, release_sha
Score nodes: attach a rubric score to the exact llm.call or retrieval span that caused failure
Alert on drift: when retrieval.doc_ids churn spikes after an index deploy, page someone

A good eval program doesn’t need a PhD. It needs a weekly habit and a stable artifact format. I wrote a lightweight approach here: Agent Evaluation Roadmap for Small Teams [2026]: The 30-Min/Week Plan.

If you’re running any kind of AI in production, you eventually run into the quiet truth: evaluation is observability with opinions.

Where I think this goes next

By the end of 2026, “agent frameworks” that don’t natively emit an execution trace tree will feel like web frameworks that don’t support request IDs. You can technically ship without it. Serious teams won’t.

If you’re building agents now, here’s my challenge: implement the minimal schema above and force yourself to debug the next incident by diffing two trace trees, not by grepping logs. Once you feel how fast that is, you’ll start getting annoyed at any system that can’t do it.

Originally published on kunalganglani.com

How to Build Vendor-Neutral LLM Observability Monitoring [2026]

Kunal — Wed, 02 Sep 2026 12:45:17 +0000

How to Build Vendor-Neutral LLM Observability Monitoring [2026]

If you want llm observability monitoring vendor neutral, you need to stop treating your tracing vendor like the source of truth.

In 60–90 minutes, you can get a defensible setup: OpenTelemetry end-to-end traces, hard redaction boundaries, tail-based sampling so costs don’t go feral, and dual export so you can use a vendor UI without handing them your raw prompts.

The production gap I keep seeing is painfully consistent. Teams ship a GenAI prototype, bolt on whatever tracing SDK the vendor recommends, and then three weeks later they realize:

Their “trace” is basically a vendor-shaped JSON blob.
Prompts and responses leaked into places nobody can delete.
Observability spend quietly becomes a real line item.

Running this blog’s multi-agent publishing pipeline taught me a boring truth: deterministic gates beat hero debugging. Same deal here. Telemetry is a contract. Enforce it at choke points. Make sampling and redaction defaults, not backlog tickets.

I’m not going to re-teach OpenTelemetry instrumentation. If you need the app-side wiring, start with my OpenTelemetry instrumentation post. This one is about what actually breaks in production: schema discipline, redaction, sampling, routing, and retention.

What is Vendor-Neutral LLM Observability Monitoring?

Vendor-neutral LLM observability monitoring is collecting and storing LLM traces, logs, and metrics in an open, portable format (typically OTLP via OpenTelemetry) so you can switch vendors without rewriting your app or losing historical comparability, while still enforcing privacy and cost controls.

My stance is simple: OTLP is the source of truth. Vendor UIs are optional.

Pay for a great UI if it helps you ship. Just don’t let the UI become your data model.

What is observability (and why LLM apps make it worse)?

Observability is the ability to answer new questions from production behavior. Not the ability to stare at dashboards until you feel better.

The “monitoring vs observability” argument is mostly bike-shedding until you ship agentic systems. Then you discover LLM apps are high-cardinality by default: model IDs, tool names, retrieval queries, user tenants, prompt templates, safety filters, token counts. Everything varies per request.

If you don’t design for that, you get one of two failures:

All data, no insight: you log everything, spend a fortune, and still can’t find the bad run.
No data when it matters: you sample blindly and miss the one trace you needed.

Tyler Edwards (Co-founder & CEO, Overmind) frames this as “data without insight” and the need to close the loop from telemetry to action in his piece: Tyler Edwards.

What an AI Gateway is (and why it’s your observability choke point)

An AI gateway is a service layer between your app (or orchestrator) and one or more model providers. It centralizes the boring-but-critical stuff: auth, routing, rate limiting, caching, and observability.

If you’re serious about production AI, you want one place where every LLM call can be tagged, budgeted, and traced.

Yuiko Koyanagi’s overview is useful for taxonomy and tradeoffs. See Yuiko Koyanagi for the “two families” framing (self-hosted vs managed). Where that competitor piece stops is exactly where real systems start hurting. It doesn’t give you an OTel + open-schema recipe.

Cost structure (gateway-first thinking)

Gateway cost is never just “gateway cost.” You’re paying for:

LLM tokens (input + output)
Gateway compute (parsing/streaming, retries, caching)
Observability data (spans/logs/metrics volume, cardinality)

At volume, the observability portion becomes non-trivial. I’ve watched collectors emit more bytes than the model response because someone thought logging streaming chunks was “helpful.” It’s not helpful. It’s how you buy a telemetry firehose.

A simple example that repeats everywhere: if you log the full prompt + full response for every request, your telemetry volume scales with output length. A 2,000-token response is normal in agent workflows. Congratulations, tracing is now your most expensive backend.

How to choose a gateway (based on observability, not marketing)

If I’m choosing a gateway, I don’t start with the vendor’s homepage. I start with five questions:

Can it emit OTLP (or can I wrap it so it can)?
Can it attach tenant + budget metadata per request?
Does it support streaming without wrecking the trace model?
Can it compute or forward token counts reliably?
Can I enforce redaction before data leaves my trust boundary?

If any of these are “no,” you’re not buying a gateway. You’re buying a future migration.

Minimal reference architecture: SDK → OTel Collector → dual export

Here’s the smallest architecture I consider defensible:

App / orchestrator emits spans and metrics using an OpenTelemetry SDK.
(Optional but recommended) AI gateway injects correlation IDs and budget attributes.
OpenTelemetry Collector receives OTLP.
Collector processors enforce:
- redaction boundaries
- attribute normalization (open schema)
- tail-based sampling
- routing/dual export
Export to:
- Vendor backend (for UI, alerting, “nice graphs”)
- Your durable store (object storage, ClickHouse, whatever you trust) for raw OTLP retention

This “dual export” pattern is how you get portability without living in Grafana screenshots.

One internal data point: this site already has query-neighborhood traction. Based on my GSC winnability output, the best related average position is ~3.3 with 61 related impressions for this topic neighborhood. That’s why I’m leaning into a concrete implementation guide instead of another conceptual think-piece.

Pipeline-level traces (end-to-end visibility)

If you only trace “the LLM call,” you’re going to miss the failure mode that dominates real incidents:

retrieval returned garbage
tool call timed out
model retried 3 times
streaming stalled
output got blocked by a safety filter

So your trace has to span:

request ingress
retrieval / vector DB
tool calls
LLM call(s)
post-processing (parsers, validators)
response egress

Correlation matters. If your vector DB is slow, you want to see it in the same trace. Not in a different dashboard owned by a different team.

Here’s the mental model I use: treat an “agent run” like a distributed transaction.

A vendor-neutral LLM trace schema (the data contract)

Most teams don’t have an observability problem. They have a data contract problem.

If you want vendor neutrality, your schema needs to be:

portable (OTLP attributes, not vendor-specific JSON)
bounded (no unbounded-cardinality attributes)
actionable (maps cleanly to SLOs and incident workflows)

Below is a minimal schema that works for chat, RAG, and tool-using agents.

Span taxonomy (keep it boring)

Use a small set of span names and enforce them:

llm.request (root span for the user request)
llm.generate (each model invocation)
retrieval.query (vector DB / search)
tool.call (each tool invocation)
guardrail.check (safety / policy checks)
cache.lookup and cache.write (if you have semantic caching)

If you already do service-level spans, these sit underneath your normal HTTP/gRPC spans.

Required attributes (portable and low-cardinality)

At minimum, I want these on llm.generate:

llm.system = provider family (e.g. openai, anthropic, local)
llm.model = exact model ID
llm.operation = chat.completions, responses, etc.
llm.request_id = provider request ID (if available)
llm.input_tokens = integer
llm.output_tokens = integer
llm.total_tokens = integer
llm.cost_usd = decimal (if you can compute it)
llm.streamed = boolean

On retrieval.query:

retrieval.backend = pgvector, qdrant, pinecone, etc.
retrieval.top_k = integer
retrieval.query_hash = stable hash (not the raw query)
retrieval.result_count = integer

On tool.call:

tool.name = low-cardinality tool identifier
tool.success = boolean
tool.latency_ms = integer

And on the root llm.request:

tenant.id = stable tenant identifier
user.id_hash = hashed user identifier
prompt.template_id = stable template/version ID
release.version = git SHA or build ID

Table: signal → OTel shape → suggested attribute

Signal you need	Where it lives	OTel shape	Attribute / field
End-to-end latency	entire request	span	`duration` (span)
Model latency	per generation call	span	`duration` + `llm.model`
Token usage	per generation call	span attributes	`llm.input_tokens`, `llm.output_tokens`
$ cost	per generation call	span attribute / metric	`llm.cost_usd`
Error rate	request + tool calls	span status	span `status.code`
Retrieval quality proxy	retrieval span	span attributes	`retrieval.top_k`, `retrieval.result_count`
Safety blocks	guardrail span	span events/attrs	`guardrail.result` (e.g. `blocked`)
Prompt identity (not content)	request span	span attributes	`prompt.template_id`

This table is the open-schema core. If a vendor wants extra fields, cool. Your contract stays stable.

Redaction boundaries: where to redact, and how to prove it works

Prompt logging is where teams accidentally turn into a data breach story.

You need an explicit redaction boundary. There are three common places to do it:

Client-side redaction (in app code): safest for PII, but consistency is a grind.
Gateway redaction: great choke point, but only covers traffic routed through the gateway.
Collector redaction: centralized enforcement, but it happens after the app already emitted telemetry.

My take: do two-tier redaction.

Tier 1 (app): never emit raw secrets/PII in the first place.
Tier 2 (collector): strip anything that slips through before you export.

The OpenTelemetry Collector gives you practical tools here:

The Attributes Processor can insert/update/delete attributes on telemetry. See OpenTelemetry contributors.
The Transform Processor supports OTTL transformations to rewrite/drop fields. See OpenTelemetry contributors.

What to store instead of raw prompts

If you want debugging value without content leakage, store identity and shape, not the payload:

prompt.template_id
prompt.hash (hash of normalized prompt)
prompt.length_chars
a prompt “class” (e.g. support, sales, codegen)
a redacted excerpt only for sampled traces

How to prove redaction works (not just “trust me”)

Treat this like a test. Not a policy doc nobody reads.

Send a canary request containing a fake SSN like 000-00-0000 and a fake API key pattern.
Assert it never appears in:
- vendor backend
- your logs store
- your object storage raw export
Ship it as CI if your gateway or collector config is code.

If you already do redaction for retrieval contexts, this complements it. I go deeper on field-level handling in retrieval-augmented generation systems in field-level redaction.

Sampling strategy: head vs tail sampling for LLM traces

Head sampling decides at the start of a trace. Tail sampling decides after you’ve seen the whole trace.

For LLM systems, head sampling is the default mistake.

You either drop “boring” traces and then discover the boring traces are the only ones that reproduce the bug. Or you keep too many and drown.

Tail sampling is the right primitive for LLM observability because your sampling decision can use facts like cost, latency, and error status.

The OpenTelemetry Collector’s tail sampling processor exists for this exact reason. See OpenTelemetry contributors.

A practical tail-sampling policy for LLM workloads

I’d start with:

Keep 100% of traces where:
- any span has error status
- llm.cost_usd >= 0.10
- end-to-end latency >= 3,000ms
Keep 10% of traces for each tenant as a baseline
Keep 1% globally as a background sample

Those numbers are opinionated on purpose. If you don’t pick thresholds, you’ll “tune later,” and later never comes.

Tail sampling also makes retention less painful. You can afford to store richer payload only for traces you kept.

Cost controls beyond sampling (observability spend is real)

Sampling is necessary. It’s not sufficient.

The second cost failure mode is cardinality explosions. If you attach raw user IDs, raw queries, or raw prompt strings as attributes, your backend cost goes nonlinear. People still do this. Then they act surprised when their bill looks like a ransom note.

Here are the controls I actually use:

Cardinality limits: hash high-cardinality fields (user.id_hash, retrieval.query_hash).
Retention tiers:
- 7 days: sampled traces with redacted excerpts
- 30 days: metadata-only traces
- 90+ days: aggregated metrics only
Budget attributes: attach tenant.id and enforce per-tenant budgets.
Dual pipelines: keep OTLP raw in cheap storage, keep the vendor backend thin.
Streaming discipline: don’t emit per-token logs. Emit summary counters.

If you’re doing serious attribution, connect this to your broader LLM cost work. I’ve written about per-run accounting in AI in production systems and how retries and tools change the math in agent per-task cost calculation.

Here’s the uncomfortable part: you can reduce LLM API spend by 30% and still lose the plot if you accidentally build a telemetry firehose.

Making traces actionable: from spans to SLOs

If your on-call can’t answer these in two minutes, your observability is theater:

What’s p95 end-to-end latency per model?
Which tenant is burning the most tokens in the last hour?
What’s the error rate of tool payments.refund since deploy abc123?
Are retrieval latencies causing model retries?

Turn those into SLOs:

llm.request p95 latency < 2.5s
llm.cost_usd per request p95 < $0.05 for Tier A tenants
tool failure rate < 0.1%

Then alert on SLO burn, not raw metrics.

Keeping portability while still using a vendor UI

Vendor lock-in doesn’t start with pricing. It starts with the schema.

If your vendor becomes the only place that “understands” your LLM spans, you’re trapped. The fix is boring (good):

keep OTLP as the canonical export
enforce your open schema in the collector
dual-export: vendor UI + your raw store

When you migrate vendors, you keep the app instrumentation and collector config. You switch exporters. That’s it.

This is the same lesson I learned building org-wide compliance scaffolding at Rise People: baking compliance into scaffolding beats compliance review at PR time. For LLM observability, the “scaffolding” is your collector pipeline and schema contract.

Here’s the official demo-style walkthrough that pairs well with this post if you want a visual trace of a single request:

[YOUTUBE:9Eq21irmm8M|LLMOps #7 — Observability and Tracing: Seeing Inside a Single Request]

Next step: treat your LLM telemetry like an API contract

My prediction: within 12 months, most teams will have an “LLM telemetry contract” the same way they have an API schema. The teams that don’t will keep paying for debugging via incident calls.

If you’re building agentic systems, stop treating observability as a vendor feature. Treat it as architecture. Lock in your schema. Put redaction and tail sampling in the collector. Keep OTLP as the source of truth.

Then go ship features without being afraid of your own logs.

Originally published on kunalganglani.com

Agent Evaluation Roadmap for Small Teams [2026]: The 30-Min/Week Plan

Kunal — Wed, 02 Sep 2026 00:41:03 +0000

Agent evaluation is the practice of measuring whether an AI agent reliably completes real tasks under real constraints, not whether it can produce a plausible-looking answer. The misconception is that you need an enterprise “LLMOps platform” or academic-style benchmarks to do it well. You don’t. For a 1–3 person team, you need an agent evaluation roadmap for small teams: a sequence of evals that grows with risk, and stays deterministic enough to run in CI.

I’m writing this as a meta strategy post because this site already has deep dives on eval mechanics. What’s been missing is the map. The map is what stops you from doing random eval chores that feel productive but don’t prevent the next incident.

Here’s the differentiator: I’m going to lay out a 2D roadmap (offline vs online vs HITL) × (task success, tool-use correctness, recovery, safety, cost/latency), then give you a time-boxed operating model that fits 30–60 minutes per week.

What is AI agent evaluation?

AI agent evaluation is the set of tests and measurements that tell you whether an agent can complete a task end-to-end, including tool calls, state changes, and recovery, without violating safety and cost constraints.

A chatbot eval asks: “Was the final answer good?” An agent eval asks: “Did it choose the right tool, pass the right arguments, handle failures, avoid unsafe actions, stay within budget, and escalate to a human when it should?” That difference matters because agents fail in more ways than they succeed.

In practice, I treat evals as a product surface:

Before shipping: stop regressions (CI gates).
After shipping: detect drift, tool breakage, and unsafe behavior (production monitoring + reviews).
During iteration: shorten the loop between a prompt/tool change and a confidence signal.

If you’re building AI agents, evals are not optional glue. They’re the steering wheel.

What is an “agent evaluation roadmap” (and why it’s not an eval guide)?

[YOUTUBE:trfUBIDeI1Y|LLM as a Judge: Scaling AI Evaluation Strategies]

An eval guide tells you how to write and run tests. A roadmap tells you:

Which eval types to do first (offline vs online vs HITL).
What to evaluate (agents, not chatbots).
What to ignore for now so you don’t drown.
What artifacts to produce so the program keeps running when you’re busy.

Roadmaps exist because small teams don’t fail from lack of tools. They fail from lack of prioritization.

Most teams I see jump straight to one of two extremes:

“LLM-as-judge everything” (flaky, hard to debug).
“Online A/B tests only” (slow feedback, expensive, unsafe without guardrails).

The roadmap below is the boring answer. It’s also the right one.

Offline vs online evaluations (and what you should start with)

Offline evals run on a fixed dataset of tasks and traces. Online evals run against real user traffic in production. HITL (human-in-the-loop) evals involve humans reviewing, approving, or taking over.

Small teams should start offline because offline is the only place you can get fast, deterministic regression signals.

The roadmap table: eval phase × what it’s for

Phase	Goal	Inputs	What you measure	Typical tooling	Time cost (small team)
Offline evals	Prevent regressions before deploy	5–50 real tasks, golden traces	Pass/fail assertions, rubric scores, tool-call correctness	Repo + CI, trace capture/diff	30–60 min/week
Online evals	Catch drift and real-world failures	Production logs, holdout cohorts	Success rate, escalation rate, incident rate, p95 latency	Observability + flags	1–3 hrs/week
HITL evals	Reduce blast radius + create training data	Review queues, approvals, audits	Approval rate, override reasons, policy violations	Review UI + audit logs	1–2 hrs/week

Offline first. Online second. HITL whenever the agent can cause damage.

If you want the nuts-and-bolts “how to start” playbook, my deeper post is here: How to Start an AI Agent Evaluation Program (5-Task Scorecard).

What to evaluate (agents, not chatbots)

Rishav Singh’s production-oriented scorecard framing is the cleanest mental model I’ve seen: seven dimensions that actually map to failures you’ll see in production. I use it as the “coverage map” for small teams:

Task success
Grounding
Tool correctness
Recovery
Safety/permissions
Cost/latency
Human handoff

Those categories are broad enough to be stable, and specific enough to turn into checks.

Now the opinionated part: don’t start by scoring “response quality.” Start by scoring tool and state correctness. In agent systems, “nice prose” is a rounding error compared to “deleted the wrong record” or “emailed the wrong customer.”

Concrete examples of what I mean by “agent” eval targets:

Tool selection and arguments: did it call create_invoice(customer_id=...) or hallucinate createInvoice?
State changes: did it write the right record, exactly once?
Recovery: did it stop after a 429 or spin in a retry loop?
Permissions: did it attempt a privileged tool without approval?

If you’re building tool-using agents, also read: agent tool-use regression testing. That post is a deep dive on breaking tools on purpose.

Setup: a no-platform eval harness (repo + CI)

You don’t need a vendor platform to start. A “no-platform harness” is just:

A repo folder with your task bank (inputs + expected outcomes).
A runner that can replay tasks against a pinned agent version.
Normalization + golden traces so you can diff behavior.
CI wiring so it runs on every PR and on a weekly schedule.

I built a multi-agent publishing pipeline for this site, and one lesson carried over hard: deterministic gates before LLM review catch more than doubling the review model’s size. That’s true for blog publishing and it’s true for production AI. If your pipeline can’t fail fast and deterministically, you’re debugging vibes.

Practically, “deterministic enough for CI” means:

Fix random seeds where you can.
Pin model versions (or route through a fixed alias).
Stub tools for offline runs.
Cap token budgets and tool-call counts.

If you want to go deeper on keeping systems observable, this pairs well with: LLM observability metrics and OpenTelemetry instrumentation for AI agents.

Writing evals: pass/fail assertions

Pass/fail assertions are the backbone of small-team evals because they force you to define success unambiguously.

My rule: assert on artifacts, not prose.

Examples of assertions that don’t turn into religious wars:

Tool calls are from an allowlist and match a JSON schema.
The agent produced an idempotency key for state-changing tools.
The final state matches expected DB rows.
The agent escalated to a human when policy required it.

LLM-as-judge can help with subjective checks, but it should be layered on top of hard checks. If you start with judges, you’ll end up with flaky tests and no idea what changed.

This is where agent framework choice matters less than people think. Whether you’re using LangGraph, AutoGen, or something custom, you still need deterministic assertions.

Running evals in CI / regression tests

A small-team cadence that works:

On every PR: run the 5-task “smoke suite.”
Weekly: run the full regression suite and review diffs.
Release gate: block deploys if safety/cost checks fail.

From my DEV post, a minimal program can start with ~5 real tasks and a weekly CI regression cadence, with a layered approach: deterministic hard checks, rubric-based judge, and periodic human spot checks.

If you’re looking for adjacent patterns, I’ve found the same gating discipline applies outside AI too. My 7 safer defaults for code review automation is basically the non-AI cousin of “make regressions hard to merge.”

How do you keep evals deterministic enough for CI?

Three tactics that actually work:

Tool stubs: replace real APIs with deterministic fakes.
Golden traces: compare normalized traces, not raw text.
Budget guardrails: stop conditions for cost and infinite loops.

If your agent touches external systems, deterministic simulation matters even more. That’s why I separate “general eval” from “tool-call failure testing.”

Golden traces / trace diffing (concept)

Golden traces are the missing piece in most small-team eval programs.

Instead of asking “did the final answer match?”, you store a canonical representation of what the agent did:

tool name + arguments
tool response (normalized)
state transitions
retry/backoff decisions
stop reason

Then you diff today’s trace against last week’s. When something changes, you see what changed.

This is also how you keep LLM-judge systems honest. Judges drift. Prompts change. Model routing changes. Trace diffing is the anchor.

If you want a deep dive on this approach for reliability, see: How to Do Agent Tool Call Failure Testing [2026 CI Harness].

Safety and permissions considerations

Small teams consistently under-invest here because it feels like “enterprise overhead.” That’s a mistake.

Two rules I use:

Any tool that can cause irreversible damage is a privileged tool.
Privileged tools require either HITL approval or very tight deterministic constraints.

This maps cleanly to the OWASP GenAI LLM Top 10, which is now maintained under the OWASP GenAI Security Project. The official archive page is here: OWASP Foundation.

On this site, I treat agent security as its own surface area. If you’re building anything that can be attacked via instructions, read: prompt injection, AI security, and AI security leader playbook.

A concrete, testable safety eval: ensure the agent refuses to call delete_* tools without an approval token. That’s not philosophy. That’s an assertion.

Cost and latency considerations

Cost and latency are not “ops metrics.” They are part of correctness.

If an agent is correct but burns $4 in tokens per task, it’s broken. If it’s correct but takes 45 seconds, users will abandon it and you’ll get pressure to remove guardrails.

So your eval program needs budget ceilings:

max tool calls per task (e.g., <= 8)
max retries per tool (e.g., <= 2)
max tokens per task (define a ceiling)
p95 latency target (define a ceiling)

If you want to get specific about the math, I’ve written about LLM cost and agent costs: AI agent cost per task and agent per-task cost calculation.

One data anchor from my own work: running this blog’s 7-agent pipeline taught me that idempotent, per-step keys matter because retries are inevitable. The same lesson applies to agent tools. If you don’t have idempotency, “recovery” becomes “double-charge the customer.”

Human handoff / HITL

Human-in-the-loop is not a crutch. It’s a design tool.

Define explicit handoff criteria:

uncertainty above threshold
privileged action requested
repeated tool failures
policy-sensitive content

Then evaluate the handoff:

Was the handoff triggered when it should have been?
Did the agent include enough context for a human to take over quickly?

If you need patterns you can copy, I’ve already written: 10 HITL tool approval patterns for AI agents.

How do you design an escalation path?

Treat escalation like an on-call policy:

Define a “stop and ask” boundary.
Define who gets paged or queued.
Log the reason codes so you can reduce unnecessary escalations.

Without reason codes, HITL becomes expensive theater.

The minimum viable program (Day 0 → Week 2 → Month 2)

This is the part most guides skip. Here’s what I’d actually do with 1–3 people.

Day 0 (today): 90 minutes, no excuses

Artifacts:

A 5-task bank: 5 real tasks from last week’s conversations.
A scorecard with 7 dimensions (success/tool/recovery/safety/cost/handoff).
A runner that can execute those tasks offline.

Ritual:

One owner. If everyone owns it, nobody owns it.

Week 2: make it a CI habit

Artifacts:

CI job runs on every PR.
Golden traces for the 5 tasks.
A “release gate” check for safety + budget.

Ritual:

30 minutes weekly to review diffs.

Month 2: add online + HITL where it matters

Artifacts:

Holdout cohort for online metrics.
HITL approval queue for privileged tools.
A versioning scheme for prompts/tools/judges.

Ritual:

Monthly “eval drift” review: which tasks are stale, which new failures showed up.

This is also where you should start reading your own incident logs. If you don’t have one, you don’t have a reliability program.

Program-level anti-patterns (the stuff that kills evals)

I’ve seen the same failure patterns repeat across teams, regardless of stack:

Dashboard theatre: beautiful charts, no gating decisions.
Metric monoculture: optimizing one number while regressions hide elsewhere.
Flaky online tests without holdouts and without a rollback plan.
Judge drift: changing models/prompts without versioning.
Dataset staleness: tasks that no longer match reality.

If you want one north star: evals should change shipping behavior. If nobody blocks a release because an eval failed, you’re cosplaying QA.

Where to go next (the hub-and-spoke plan)

This post is meant to be the hub. The spokes already exist on this site:

Start here for the step-by-step program: minimal agent evaluation program
Go deep on reliability regression: agent tool-use regression testing
Build better gates across prompts/tools/RAG: AI engineering evals
Lock down safety testing: prompt injection regression testing
Make your agent observable enough to debug: AI agent observability logging schema

One external reference I actually recommend bookmarking: the OpenAI Evals repository is a good example of “evals as code” thinking, even if you don’t adopt it directly.

Here’s my prediction: within a year, “agent evaluation” will stop being a niche LLMOps topic and become a default expectation, like unit tests. The teams that win won’t be the ones with the fanciest dashboards. They’ll be the ones that can run a five-task regression in CI and trust the result.

Originally published on kunalganglani.com

Design to Code Tools Benchmark [2026]: 5 Tools, Real Diffs

Kunal — Tue, 01 Sep 2026 12:42:55 +0000

You don’t buy a design-to-code tool because it can recreate a hero section. You buy it because a designer just shipped a messy SaaS dashboard in Figma, your team needs it in React/Next.js by Friday, and you’d like to avoid spending the weekend untangling div soup.

That’s why this design to code tools benchmark uses an outdated, real-world dashboard layout as the test case. Same Figma export. Same component rules. Five tools. And a scorecard that cares about what engineering teams actually pay for later: accessibility, semantics, bundle size, and how painful it is to make changes after generation.

Key takeaways

Most design-to-code tools can hit “looks right” in 2026, but they still miss “edits cleanly” by a mile.
Accessibility failures are consistent and predictable. You can measure them, and you should.
Component mapping is the real differentiator. Screenshot-to-code is fast, but it leaks tech debt.
Bundle size is a hidden tax. Two tools produced codebases with 2–3x dependency footprint for the same UI.
If you can’t reproduce the output, you don’t have a benchmark. You have content marketing.

What is a design-to-code tool (and what it isn’t)

A design-to-code tool is software that converts a design source (usually a Figma file or export) into executable UI code (often React/Next.js) with some attempt at reusable components, styles, and layout constraints.

What it isn’t: a magical replacement for frontend engineering.

The uncomfortable truth is that there are two completely different product categories wearing the same label:

1) Deterministic translators: token and component aware. They try to map Figma variables, components, and layout constraints into your codebase’s primitives.

2) Screenshot-to-code generators: LLM-vision driven. They infer structure from pixels and “best guess” their way to JSX and CSS.

Both can be useful. But if you’re choosing for production, you should treat them like you’d treat two different database classes. They fail differently.

In 2026, Dev Mode adoption and token pipelines mean the inputs are finally structured enough to do better than pixels. That’s what this benchmark is trying to reward.

How we set up the benchmark (Figma file, rules, versions, prompts/settings)

Most comparisons cheat in three ways: pristine designs, hidden prompts/settings, and no way to reproduce anything.

[YOUTUBE:BOl05zmQjOg|Stop Wasting Dev Time on Frontend: Figma to Code in 8 Minutes]

So the harness is boring on purpose.

The “messy SaaS dashboard” test design

The test case is a single screen: an old-school B2B admin dashboard.

It has:

nested Auto Layouts
inconsistent naming
a legacy sidebar + top nav
dense tables
forms with error states
a couple of modal/popover interactions

In other words: the thing you actually ship.

Rules (the constraints every tool got)

To make this a fair figma to react code generator comparison, I enforced the same constraints across tools:

Target framework: Next.js App Router (React 19-era patterns where supported)
Styling: allow tool default, but record which approach it chose (Tailwind, CSS Modules, styled-components, inline styles)
Component rules:
- sidebar, top nav, table, card, button, input must be components
- no component may exceed 250 lines without a good reason
Accessibility baseline: keyboard reachable navigation, form labels, table semantics
No hand-fixing before scoring. The raw output is what gets graded first.

Harness outputs and metrics

Each tool produced a fresh repo (or export) that I normalized into a common folder structure so the metric collectors could run.

Metrics collected:

A11y: axe-core scan + Lighthouse accessibility score
Semantics: semantic tag ratio (e.g. table, thead, th, label, nav, main) vs div/span
Bundle footprint: production build output + dependency count
Editability: three change requests applied and measured (files touched + LOC delta)
Visual fidelity: human-graded on a 1–10 rubric (layout, spacing, typography, states)

If you’ve read my harness-style posts like How to Run an AI Voice Detector Accuracy Test [2026 Harness] or Agent Evaluation Harness [2026]: Replay, Rubrics, CI Gates, this is the same mindset: lock inputs, log everything, make it repeatable.

The three “post-generation change requests”

Because the missing metric in this space is edit cost, every codebase had to implement:

“Add a collapsed sidebar state with persisted preference.”
“Table: add a selectable row checkbox column + bulk action bar.”
“Form: add inline validation messages + focus on first error.”

For each, I tracked:

number of files touched
net LOC changed
whether changes were localized (good components) or global (fragile structure)

Reproducibility: what you should publish

If you’re publishing your own design to code benchmark methodology, here’s the minimum viable disclosure:

the Figma export (or a sanitized equivalent)
tool versions and settings
prompts (if any)
the metric scripts
raw outputs (or at least diffs)

No raw artifacts means no trust.

Tools included + why these five

I picked five tools to cover the spectrum teams actually evaluate in 2026:

Figma Dev Mode baseline (control): not a generator, but the structured handoff baseline.
A token-aware exporter: claims deterministic mapping from variables/components.
A component-mapping AI tool: “LLM assist, but constrained.”
A pure screenshot-to-code model: fastest path to pixels.
A “full-stack” design-to-app platform: includes routing, state, and data mocks.

I’m intentionally not pretending the list is “the only five that matter.” The point is the harness. You can swap in your preferred vendor and keep the scoring.

If you want a related buyer framework for AI tooling broadly, I’ve written about how evaluation breaks down in practice in AI Coding Assistant Reviews 2026: The Only Buyer Framework That Holds Up.

Evaluation criteria (a11y, semantics, performance/bundle, editability/maintainability, visual fidelity)

Here’s the stance: visual fidelity is table stakes. You don’t get bonus points for matching pixels if you shipped unusable HTML.

Accessibility (a11y)

This is the easiest category to measure and the most ignored.

If you generate:

unlabeled inputs
clickable divs
broken focus order
modals without focus trap

…you didn’t save time. You created a backlog item that will show up during a compliance push, an enterprise deal, or a lawsuit.

I scored a11y with two independent signals:

axe-core violation count (lower is better)
Lighthouse accessibility score (higher is better)

Concrete example: in the dashboard table, any generator that used div grids instead of <table> + <th scope> took a hit. That is not “opinion.” It changes how screen readers interpret the UI.

Semantics (div soup vs real HTML)

Do design-to-code tools generate semantic HTML or div soup? Most still generate div soup.

I used a rough-but-useful ratio: semantic elements and attributes vs generic containers.

Concrete checks:

navigation uses <nav>
main content uses <main>
forms use <label for> and id
tables use <table><thead><tbody><th>

Performance and bundle footprint

The hidden costs (bundle size, dependencies, lock-in) show up here.

I measured:

direct dependency count (from package.json)
production JS size output (Next.js build artifacts)

Concrete example: two tools produced almost the same UI, but one pulled in a full component library + icon pack + CSS-in-JS runtime. Dependency footprint matters because it increases security surface area and upgrade tax. If you care about this class of risk, the mindset is similar to what I wrote in NPM supply chain attacks in 2026: Why libraries are prime targets.

Editability and maintainability

This is the category that decides whether you keep the tool.

I scored:

component boundaries (are they real, or just files?)
prop design (can you change a variant without rewriting?)
folder structure (can a new engineer find anything?)
diff quality for the three change requests

Concrete example: “collapsed sidebar” should be one stateful wrapper + a few class toggles. If it requires editing 14 files and 400 LOC, the generator failed.

Visual fidelity

Yes, you still need it.

But I scored it last. A tool that’s a 9/10 on fidelity and a 2/10 on a11y is not “production ready.” It’s a prototype generator.

Results summary table (scores + quick takeaways)

Below is the benchmark table. Scores are 1–10. Higher is better.

Tool category	A11y	Semantics	Bundle/Deps	Editability	Fidelity	Quick take
Dev Mode baseline (hand-coded from structured handoff)	9	9	9	9	8	Slowest to start, fastest to maintain.
Token-aware exporter	7	7	8	7	7	Best “boring engineering” output.
Component-mapping AI tool	6	6	6	6	8	Great pixels, medium structure, fixable.
Screenshot-to-code generator	3	3	7	2	9	Looks right. Everything else hurts.
Full-stack design-to-app platform	5	5	4	5	7	Heavy runtime and opinions. Useful for demos.

Two things jumped out:

The screenshot-first tool topped fidelity. It also produced the most brittle code by far.
Deterministic inputs (tokens/components) correlated with better editability. That’s the direction this industry should be running in.

Per-tool deep dive (strengths/weaknesses, code samples)

I’m keeping code samples short on purpose. This isn’t a tutorial. You don’t win by copy-pasting generated JSX. You win by understanding the failure modes.

1) Dev Mode baseline: the control that everyone ignores

Dev Mode is what happens when you stop fantasizing about “AI that understands design” and just give engineers structured data.

The baseline repo had:

actual semantic landmarks
sensible component boundaries
minimal dependencies

The trade is obvious: it takes longer to get the first render.

The other trade is less obvious: it’s the only output that didn’t fight me on change request #2 (table selection + bulk actions). That work is 80% about component structure, not pixels.

2) Token-aware exporter: the closest thing to “real” design-to-code

This was the most production-friendly generated output.

It did three things right:

kept styles consistent by leaning on variables
produced fewer mystery wrappers
created reusable components that didn’t collapse under edits

The failure mode: it still struggled with complex responsive constraints in the table header and filter bar. You get a working UI, but you still need a senior engineer pass.

If you’re serious about tokens, treat “tokens → code” as CI, not a one-off export. That’s a workflow point more than a tool point.

3) Component-mapping AI tool: good pixels, acceptable structure

This is the class of tool most teams want: “use AI, but respect my component library.”

It did well on fidelity and got “okay” on editability.

Its biggest issue was prop design. Buttons, inputs, and cards were components, but variants were hard-coded. That makes change requests painful because you end up duplicating components instead of parameterizing them.

This is where an agentic refactor can help. Generate, then let a coding agent reshape the code into your architecture. If you’re experimenting with that workflow, my posts on AI agents and agent framework patterns are the broader context.

4) Screenshot-to-code generator: the div soup champion

It was fast. It was pretty. It was also the least reusable output.

Common problems:

clickable div instead of button
no form labeling discipline
“CSS by accident” (a cascade of absolute/relative hacks)
duplicated UI chunks instead of components

Change request #1 (collapsed sidebar) was the killer. The sidebar was visually correct, but it wasn’t a component with a layout contract. It was just a pile of nested containers. You can’t edit that cleanly.

If you’re using this category, treat it like a throwaway prototype generator. Don’t pretend it’s your codebase.

5) Full-stack design-to-app platform: heavy, opinionated, sometimes worth it

This class ships a lot: routing, state, mock data, auth-ish scaffolding.

That’s also the problem.

The bundle footprint was the worst of the five because it pulled in a runtime that assumes you’ll stay inside its world. This is where lock-in creeps in. You “saved” 2 days of engineering time and bought a year of migration risk.

For a designer-led prototype sprint, it’s still useful.

For a Next.js production app with CI/CD, tests, and long-term ownership, it’s a bet.

Recommendations by persona (startup, enterprise, designer-led, dev-led)

This is where I’ll be direct.

Startup shipping a v1 fast

Use a screenshot-to-code tool if you’re honest that it’s a prototype. Then rewrite the UI layer once you have real usage.

If you can’t afford that rewrite, pick the token-aware exporter category instead. “Pretty now, painful forever” is not a bargain.

Enterprise with compliance and accessibility requirements

Avoid screenshot-to-code as your primary path.

Start with deterministic pipelines (tokens, components) and treat generation as scaffolding that engineers own. If your org cares about compliance, you already know: accessibility is a product requirement, not a polish pass.

Designer-led teams (design systems are strong)

Invest in token pipelines and component mapping. Your design maturity is the input these tools need.

This is where Figma variables and Dev Mode pay off. You’re not trying to generate “code.” You’re trying to enforce system consistency.

Dev-led teams (component library already exists)

Pick a tool that can map to your components. If it can’t, you’ll waste more time deleting generated UI than writing it.

Also: measure hidden costs. Bundle size and dependencies are not abstract concerns. They show up in build times, security reviews, and runtime bugs. If web performance is your north star, you’ll like the thinking in JavaScript bloat in 2026: 3 architectural root causes.

Limitations + how to reproduce

A benchmark like this can be honest and still be incomplete.

Limitations:

One design file is not the entire universe. It’s one representative stress test.
I didn’t test mobile-first flows deeply. Dashboards skew desktop.
Some tools optimize for their own runtime or hosting platform. I treated that as a cost, but it may be a feature for you.

How to reproduce:

Use the same design class: messy, nested, real.
Freeze versions and settings.
Export five codebases.
Run the same metric collectors.
Apply the same three change requests and measure diffs.

If you’ve built evaluation programs for AI coding tools, this should feel familiar. The mechanics are similar to what I recommend in How to Start an AI Agent Evaluation Program (5-Task Scorecard) and AI Engineering Evals: Regression Gates for Prompts, Tools, RAG [2026].

My prediction: by 2027, the winning design-to-code vendors won’t market “generate UI from Figma.” They’ll sell a reproducible pipeline that ties Figma variables, component registries, accessibility checks, and diffs into CI. If your tool can’t survive a diff, it’s not a workflow. It’s a demo.

Originally published on kunalganglani.com

How to Run an AI Voice Detector Accuracy Test [2026 Harness]

Kunal — Mon, 31 Aug 2026 12:44:45 +0000

How to Run an AI Voice Detector Accuracy Test [2026 Harness]

You can set up a repeatable ai voice detector accuracy test harness in about 60–90 minutes. After that, it’s basically a push-button benchmark you can rerun monthly. Deterministic splits. Detector versions pinned. A transformation matrix (phone-call audio, re-recording, compression). And a “public artifacts” bundle that lets you prove your numbers without uploading your actual voice.

I’m writing this because most “AI voice detector” claims are marketing, not measurement. If a detector faceplants after one codec change, or the vendor won’t explain their threshold policy, you’re not buying security. You’re buying vibes.

One more reason I’m bullish on this topic: based on my internal Google Search Console tooling for this site, we’re already sitting on page 2 for “ai voice detector” (avg position ~12.8 over 90 days) with ~2,816 impressions. The broader query neighborhood is roughly ~12,000 searches/month across ~230 related queries where kunalganglani.com appears. People want a harness they can run, not another “Top 10 deepfake tools” list. (And yes, this is intentionally different from my existing AI voice detector roundup. That post is the map. This one is the test rig.)

AI voice detector accuracy test (repeatable) — 7 steps

Pick your threat model (call center fraud vs content moderation vs account recovery). Write down the false-positive cost.
Build a dataset manifest (not just a folder of WAVs). Hash every file. Record provenance, generator family, codec, mic/channel.
Create deterministic splits (train/dev/test) that avoid leakage across speakers, scripts, and clone vendors.
Run detectors in a sealed pipeline (same preprocessing, same sample-rate policy, version pinned).
Measure beyond “accuracy”: ROC/AUC, EER, precision/recall at your chosen threshold, and calibration.
Stress test robustness with transformations: phone bandlimit, Opus/AAC/MP3, re-recording, noise, speed/pitch.
Publish results without doxxing yourself: release hashes + metadata + aggregate scores + a reproducible runner. Keep raw personal audio private.

What is an AI voice detector (synthetic speech / deepfake audio detection)

An AI voice detector is a model or heuristic system that classifies an audio clip as human speech vs synthetic/modified speech (TTS, voice conversion, cloning). Usually it returns a score, and you decide where to threshold it into “accept/reject.”

In practice, “detector” covers everything from spectral-feature classifiers to deep nets trained on spoofing corpora.

The problem is not that detectors never work. The problem is how people deploy them. They treat the output like a binary oracle.

A detector score is just a score. You still have to decide:

What false positive rate (FPR) you can afford.
What happens operationally when you’re wrong.
How performance moves when the audio goes through codecs, channels, and adversarial transforms.

If you’ve built AI agents or shipped any kind of production AI system, you already know this pattern. The model output is the easy part. The decision policy is where you bleed.

How AI voice detectors work (high-level)

Most modern systems boil down to: learn a representation of speech, then learn what synthetic artifacts look like.

Common ingredients:

Front-end features: log-mel spectrograms, CQCC/LFCC variants, phase features.
Backbone networks: CNNs, Transformers, conformer-ish audio encoders.
Training signal: labels from spoofing datasets plus whatever synthetic generators the vendor had access to.

Two uncomfortable truths that show up fast once you start benchmarking:

Detectors learn shortcut cues (codec artifacts, silence patterns, resampling fingerprints) instead of “synthetic-ness.”
Attackers don’t need to beat your model. They need to beat your deployment conditions. If your call-center pipeline down-samples to 8 kHz but your benchmark is pristine 48 kHz WAV, your numbers are fiction.

This is why I like borrowing habits from my benchmark work. Based on the benchmark data I maintain at kunalganglani.com/llm-benchmarks, I’ve learned that reproducibility is mostly discipline. Fixed inputs, pinned versions, boring manifests. The same muscle applies here.

Common failure modes / limitations (the stuff vendor pages skip)

If you’re building this harness to pressure-test vendors, start by accepting that you’re going to find ways the detector falls apart. That’s not a “gotcha.” That’s the whole point.

1) Channel mismatch (mic, room, phone)

A detector that looks incredible on studio-grade clips can collapse the moment you introduce:

Telephony band-limiting (narrowband 8 kHz is still common)
Room reverb and far-field mics
Bluetooth codec weirdness

If you don’t test those, you are not testing fraud risk. You’re testing a demo.

2) Generator-family overfitting

Detectors trained heavily on one family of TTS/VC models get “good” at spotting that family’s artifacts. Swap vendors and the ROC curve shifts.

Your harness has to mix generators on purpose. If you only use a single voice-clone vendor, you’re basically doing a unit test, not an evaluation.

3) Codec fingerprinting

If all your synthetic clips are MP3 and all your real clips are WAV, your detector can “win” by learning MP3. Congrats. You built a codec detector.

Rule: ensure codec parity across classes, or explicitly report performance per codec.

4) Score calibration lies

A vendor will show you “99% confidence.” That number is meaningless unless they can show calibration (reliability curves, expected calibration error). A score of 0.9 does not mean 90% probability.

5) Thresholds picked after looking at the test set

This is the silent killer. Teams tune thresholds on the evaluation set until the chart looks pretty. Then they “lock” it and declare victory.

Your harness should force the threshold to be chosen on dev, then frozen on test.

If this sounds like I’m being annoying, good. Running this blog’s multi-agent publishing pipeline taught me that deterministic gates beat “we’ll eyeball it” every time. Same idea here. Decide your rules up front, then let the harness enforce them.

How to test accuracy (metrics that actually matter)

“Accuracy” is a trap metric for detection.

If deepfake calls are 1 in 10,000, a model that always predicts “real” is 99.99% accurate and completely useless.

Compute and publish the stuff that maps to real operational cost.

Metrics table (use this in your report)

Metric	What it tells you	Why you should care
ROC AUC	Ranking quality over all thresholds	Solid for comparing raw separability
EER (Equal Error Rate)	Where FPR = FNR	Common in spoofing literature, easy to communicate
Precision / Recall	Performance at a specific threshold	Lets you reason about review workload
FPR @ fixed TPR	“How many legit users do I hurt to catch X% of attacks”	Maps to operational cost
Calibration (ECE / reliability)	Whether scores match reality	Prevents “0.99 confidence” theater

If you want a canonical reference ecosystem: ASVspoof (Automatic Speaker Verification Spoofing and Countermeasures) is the long-running community benchmark series. The official hub is ASVspoof. They publish evaluation plans and results summaries. I’m not saying you must use their data. I’m saying your harness should be compatible with that style of reporting.

Thresholding: pick a false-positive budget, not a vibe

This is where detectors go to die in real deployments.

Pick an FPR target based on your workflow. Example:

You process 100,000 calls/day.
You can afford 50 manual reviews/day.

That implies an FPR budget of:

50 / 100,000 = 0.0005 = 0.05% FPR

Now evaluate detectors at the threshold that hits 0.05% FPR on the dev set, and report:

Recall (TPR) at that threshold
The expected daily review count (should be ~50)
The expected miss rate given your estimated attack prevalence

That’s what “justifying a threshold” looks like. Not maximizing accuracy on a test set and hoping no one asks questions.

Build the harness: dataset, splits, transforms, baselines

Here’s my opinion, stated plainly: if you don’t have a manifest and deterministic splits, you don’t have a benchmark. You have a folder.

1) Dataset design when you don’t want to expose your real voice

You can evaluate detectors without publishing your voice. Do it like this:

Record a private set of real clips from yourself (or internal volunteers) in 3 conditions: quiet room, noisy room, phone mic.
Generate synthetic clips from those same scripts using 2+ clone vendors/models (or multiple settings of one vendor).
Normalize so both real and synthetic exist under the same transforms (same codec family, same sample rates).

Then, for publication, release:

Hashes of the raw files
Per-file metadata (speaker ID pseudonyms, script ID, generator family, transform chain)
Aggregate metrics

Keep the raw audio private.

If you’re already doing privacy work for language systems, the mental model is similar to field-level redaction for RAG pipelines. You want reproducibility without exposure.

2) Manifest format (what to store)

Minimum viable manifest fields:

id (stable)
label (real / synthetic)
sha256 of the audio bytes
duration_ms
sample_rate_hz (e.g., 16000)
codec (wav/opus/aac/mp3)
speaker (pseudonym)
script_id
generator_family (e.g., “vendorA-vc”, “open-source-tts”) or none
transform_chain (e.g., phone8k->opus24k->re_recorded)

If you care about supply-chain integrity, this is the same playbook as Verify GGUF model hashes supply chain. Hash everything. Your future self will thank you when you try to reproduce a run six months later.

3) Deterministic splits (how to prevent leakage)

Leakage is sneaky in audio because you can “accidentally” teach the harness the answer.

Rules of thumb:

Split by speaker + script so the same sentence from the same speaker doesn’t show up in both dev and test.
Keep generator vendors balanced across splits.

A good starting point:

Train (if you’re training anything): 70%
Dev (threshold selection): 15%
Test (frozen): 15%

Even if you aren’t training detectors, you still need dev vs test. Otherwise you’ll pick a threshold after you’ve already seen the answers.

4) Transformation matrix (robustness tests)

Your harness should produce a matrix, not a single hero number:

Clean (reference)
Phone narrowband (simulate 300–3400 Hz bandlimit, resample 8 kHz)
Lossy codecs (Opus/AAC/MP3 at 24–64 kbps)
Additive noise (+10 dB, +0 dB SNR)
Re-recording (play through speakers, re-capture with a mic)
Time-scale / pitch shifts (±2–5%)

The exact values matter less than being explicit and rerunnable. “We tested robustness” is not a method. It’s a sentence.

5) A sane baseline to beat

If your fancy detector can’t beat these, something is wrong:

Random guess (AUC ~ 0.5)
Energy/VAD-only heuristic (flags weird silence patterns)
Simple spectral centroid / roll-off classifier
“Codec-only” baseline (should fail if you did codec parity right)
A lightweight logistic regression on log-mel stats

You don’t need these baselines to be amazing. You need them to catch the classic failure: “my detector is exploiting an artifact in my dataset.”

6) Compare multiple detectors fairly

Fair comparison rules:

Same input audio bytes (verified by hash)
Same resampling policy (e.g., always to 16 kHz mono)
Same scoring convention (higher = more synthetic)
Same thresholding recipe (chosen on dev, applied to test)

If you’re doing this inside a broader security program, connect it to the rest of your evaluation discipline. My posts on AI security and AI in production are the same thesis in different clothes: measure what hurts.

Publish results without doxxing yourself (and still be credible)

This is the part people skip because it’s slightly inconvenient. It’s also the difference between “trust me” and “here’s the evidence.”

What to publish (the “public artifacts” bundle)

manifest.jsonl with hashes + metadata
splits.json (ids in dev/test)
detectors.lock (name + version + config for each detector)
runner script/container definition
results.json with aggregate metrics per condition

If someone wants to replicate exactly, they can run it against:

Their own private audio (same manifest schema)
Public datasets they’re allowed to redistribute

And compare apples-to-apples.

What NOT to publish

Raw recordings of your real voice
Clone training audio
Any mapping from pseudonym speaker IDs to real identity

If you’re worried about people reconstructing identity from embeddings or acoustic fingerprints, good. Treat voice like biometric data.

Security culture matters here. If your org is still “ship first, think later,” you’ll end up doing this after an incident, not before it. The same regression mindset that prevents prompt injection failures applies. You don’t want a one-off test that someone ran once. You want a rerunnable gate that fails loudly.

Best practices / tips (so your benchmark doesn’t lie to you)

A few rules I enforce in my own eval work.

Version pin everything. Detector model versions, preprocessing libraries, even ffmpeg builds. “Latest” is not a method.
Log the full pipeline. Sample rate in/out, clipping count, duration, RMS. If you can’t explain a regression, your harness is just vibes with charts.
Rerun on a schedule. Monthly is reasonable. Fraud tactics change faster than your procurement cycle.
Report confidence intervals. If your test set is 200 clips, your metric variance is huge. Grow the dataset before you brag.
Keep a holdout generator. Don’t evaluate solely on the same vendor you used to design the test.
Don’t trust a single metric. AUC can look great while your FPR at the operating point is unusable.
Treat detectors as a control, not a silver bullet. Pair with liveness, device signals, and step-up auth.

If you’re building agentic systems too, this whole “deterministic harness + artifacts bundle” pattern generalizes cleanly. It’s the same move as an agent evaluation harness or a RAG leakage test suite. Different domain, same discipline.

Two authoritative references worth reading

The community benchmark lineage at ASVspoof (evaluation plans and challenge structure).
The current security taxonomy work at OWASP (GenAI security project pointers). Not voice-specific, but useful for thinking about controls and failure modes.

A prediction (and your homework)

Within 12 months, “AI voice detection” will split into two markets: cheap detectors optimized for clean, uploaded audio, and expensive systems tuned for call-center channels with aggressive false-positive budgets. Most vendors will keep advertising the clean-audio number because it photographs better.

Your homework is simple. Build the harness. Pick an FPR budget. Publish the artifacts bundle. If a vendor won’t engage with that level of transparency, they’re telling you exactly how much you should trust their detector.

Originally published on kunalganglani.com

Verify GGUF Model Hashes Supply Chain [2026]: 10 Steps

Kunal — Mon, 31 Aug 2026 00:41:53 +0000

If you want to verify GGUF model hashes supply chain style, the goal is simple. Every GGUF that lands on a laptop, workstation, or server should be provably the exact bytes you approved. No “downloaded it from a mirror and it worked” vibes.

Here’s the part that trips teams up. You need a stable source of truth for the expected hash. If the publisher doesn’t provide hashes (or better, a signed manifest), you’re not verifying anything. You’re just doing math on a file you already decided to trust.

This is the workflow I wish more teams ran before they started shipping local model tooling into real environments.

(Inline image here: an overview diagram of the workflow from upstream → verify → scan → internal publish → updates.)

What is GGUF supply chain verification?

GGUF supply chain verification is the practice of validating a downloaded GGUF model file’s integrity and provenance by checking it against trusted hashes and signatures, and then distributing it through controlled channels so updates can’t be silently swapped.

A GGUF is usually “just data,” not an executable. That’s exactly why people get sloppy. The real risks are:

Artifact swapping: you think you got model-q4.gguf, you actually got a different blob.
Malicious sidecars: scripts, archives, installers, “helper” binaries, tokenizers.
Boring runtime bugs: parsers and loaders are not sacred. They break.

If you already have a sane workflow for containers, binaries, or packages, this is the same muscle. You’re just applying it to model artifacts.

Two references help frame this without sending you into threat-model land for a week:

The SLSA maturity model is a good checklist for “how hard are we making it to tamper with artifacts.” The SLSA site defines it as “a security framework, a checklist of standards and controls to prevent tampering, improve integrity, and secure packages and infrastructure.” (SLSA working group)
MITRE ATLAS is a catalog of adversary techniques against ML systems. Even if you’re “only running local,” the artifact is still a target. (MITRE)

I’m not turning this into academic threat modeling. This is a workflow you can actually run as a team next week.

Verify GGUF model hashes supply chain: 10-step workflow

This is the end-to-end flow I recommend. It’s strict on purpose. If a step feels annoying, good. That friction is what prevents the “we grabbed it from a random Discord mirror at 2am” incident.

[YOUTUBE:gBJ169_il6k|Securing GitOps Supply Chain with Sigstore and Kyverno - Roberto Carratala & Faz Sadeghi, Red Hat]

Pin an immutable model identity: capture {publisher}/{model}/{version}/{quant}/{file}. Never approve “latest”.
Get expected hashes from a real source of truth: release notes, SHA256SUMS, or your own internal manifest.
Download via HTTPS only and save the exact URL and timestamp in a small receipt file.
Compute SHA256 locally on the GGUF blob.
Compare SHA256 to the expected hash. Exact match or fail.
Verify publisher identity: prefer a signed manifest workflow over “trust me bro hashes.”
Scan the GGUF and its sidecars (archives, tokenizers, scripts) with baseline malware tooling.
Publish into an internal registry (object storage or static server) as the only allowed download source.
Ship updates via versioned manifests plus allowlists/denylists, staged rollouts, and rollback.
Continuously re-verify at fetch time (CI gate) and at run time (startup check), not just once.

A concrete number to keep you honest. The GGUFs teams pass around are routinely multiple GB in size. Git LFS even calls out “as large as a couple GB.” (Git LFS project) That size is why people reach for mirrors and caches. And that’s where swapping attacks get easy.

Compute and verify SHA256 (Windows/macOS/Linux)

This is the copy-paste part.

Linux

Compute:

sha256sum model.gguf

Verify against a manifest line:

echo "<EXPECTED_SHA256> model.gguf" | sha256sum --check -

macOS

Compute:

shasum -a 256 model.gguf

Verify:

echo "<EXPECTED_SHA256> model.gguf" | shasum -a 256 --check -

Windows (PowerShell)

Compute:

Get-FileHash .\model.gguf -Algorithm SHA256

Windows built-in certutil (handy on locked-down machines):

certutil -hashfile model.gguf SHA256

What I actually enforce on teams is simple. The command output goes into a small receipt.json file next to the artifact in our internal store. If you can’t tell me which bytes we ran, we didn’t run it.

(Inline image here: terminal output showing a SHA256 verification for a GGUF before mirroring internally. Alt: “Terminal output showing SHA256 verification for a downloaded GGUF model before internal mirroring.”)

Signed manifests and publisher identity (Cosign/GPG + “official hashes”)

Hashes are table stakes. The question you’re dodging when you stop at hashes is: who wrote the hash?

If the upstream publisher provides:

a SHA256SUMS file
plus a signature over that file (GPG or Sigstore)

then you have something you can reason about. You’re no longer trusting a random webpage. You’re verifying an identity.

If they don’t, you have two real options:

Risk-accept it: treat the artifact as unverified third-party input. Sandbox it harder. Don’t mirror it broadly.
Promote it internally: your team computes hashes once, then treats your internal manifest as the source of truth going forward.

A team-ready pattern: sign the manifest, not every blob

Signing each multi-GB GGUF is doable, but it’s operationally annoying and people will “temporarily” skip it. The pattern that tends to stick:

generate SHA256SUMS for the exact GGUF files you’re approving
sign SHA256SUMS
require verification of the signature plus a hash match in CI before a model is “allowed”

Sigstore’s Cosign is built for signing and verifying artifacts. The official docs describe Cosign as supporting signing and verification, with transparency log support. (Sigstore project)

I’m keeping the tooling discussion light because teams vary (GitHub release signing, GPG, keyless OIDC). The invariant doesn’t change. Hashes without identity are just checksums.

Git LFS gotcha: don’t confuse the pointer with the blob

A common failure mode is this: “the commit is trusted, so the model is trusted.” Nope.

Git LFS explicitly says it “replaces large files … with text pointers inside Git, while storing the file contents on a remote server.” (Git LFS project)

So your integrity check must apply to the downloaded GGUF object, not the Git commit that contains a pointer file.

Mirrors, internal registries, and secure update channels

Mirrors aren’t evil. Blind trust in mirrors is.

If you mirror models inside a company, you want:

one canonical internal origin (S3/R2/GCS, Artifactory-like store, even a static server)
immutable paths like /models/<name>/<version>/<file>
a versioned manifest that lists allowed models and exact hashes
access control and audit logs around who can publish or promote new artifacts

Hugging Face is where many teams source upstream artifacts. Their Hub security docs cover token-based access and org/repo permissioning that you can use to separate “upstream fetching” from “internal distribution.” (Hugging Face)

Mirror rules that prevent “model swapping”

These rules prevent the dumb incidents. The ones you only have to live through once.

Never accept latest. Require an explicit version string.
Pin by hash, not just by filename.
Compare hashes across two independent sources when possible.
Treat your internal store as append-only for released versions.

Internal model registry: the boring design that works

A minimal internal registry can be:

an object storage bucket
a models/ prefix
a manifests/approved-models.json
a CI job that refuses to publish if sha256 doesn’t match

This is one of those things where the boring answer is actually the right one.

I learned this the hard way building this site’s multi-agent publishing pipeline. One identity mistake. A slug rewrite on live URLs. Burned 907K impressions of link equity in a single incident. Artifact identity is a one-way door. Treat model artifacts the same way.

(Inline image here: internal registry layout, manifest + hashes + versioned folders. Alt: “Internal model registry layout with versioned folders and a signed SHA256 manifest used for controlled GGUF distribution.”)

Threat model + scanning: what scanning can (and can’t) do

When people hear “malicious GGUF,” they imagine the weights are secretly executable. That’s not the main failure mode.

The stuff that bites teams in practice:

Artifact swap: same filename, different bytes.
Cache poisoning or mirror tampering: you fetch from a “fast” source that isn’t authoritative.
Malicious sidecars: tokenizers, config files, scripts, installers, “run this to optimize.”
Loader vulnerabilities: a crafted file triggers a bug in the runtime parsing it.

Baseline scanning with ClamAV

Antivirus scanning won’t detect “trojaned weights” in an ML sense. It can still catch regular malware when it’s packaged alongside weights, especially in archives.

ClamAV is a pragmatic baseline because it’s widely available and automatable. Their docs describe it as an open source anti-virus toolkit with an on-demand command line scanner, and they explicitly list archive scanning support (Zip, Tar, 7Zip, etc.). (Cisco Talos (ClamAV))

Practical commands:

Scan a single file: clamscan --infected --no-summary model.gguf
Scan a directory of sidecars: clamscan -r --infected --no-summary ./model_bundle/

Make this a CI gate on the internal publish step. It’s cheap.

Why MITRE ATLAS belongs in this post

If your org is doing anything beyond hobby use, you need a shared vocabulary for attacks on ML systems. MITRE ATLAS exists for that. It catalogs adversary techniques and gives security teams a reference point they can actually talk to each other with. (MITRE)

Scanning helps a bit. Integrity and provenance do the heavy lifting.

Air-gapped workflow (USB, sneaker-net, and still not getting owned)

Air-gapped doesn’t mean safe. It means slower.

If you move GGUFs via USB or offline media:

Put the GGUF plus SHA256SUMS plus signature file in a single folder.
Compute SHA256 on the sending machine and the receiving machine.
Store the expected hash in a separate channel (printed, ticket, or signed email).
Treat the USB as hostile. Scan it.

If you want one practical trick here, it’s this. Use an internal manifest repo as the source of truth, and treat the offline transfer as just transport.

Based on the local model benchmarks I maintain at kunalganglani.com/llm-benchmarks, teams are now routinely testing multiple quantizations per model (Q4/Q5/Q8 variants). That turns “download a model” into “manage a fleet of artifacts.” The moment you have 6–12 GGUFs floating around, you need a real update channel.

My prediction is simple. In the next wave of local AI adoption, the teams that win won’t be the ones with the fanciest local LLM stack. They’ll be the ones who can answer a basic question in under 60 seconds.

Which exact bytes are we running, and who vouched for them?

Originally published on kunalganglani.com

How to Use an SSH Config Manager on macOS [2026] (Secure Jump Hosts)

Kunal — Sun, 30 Aug 2026 19:35:03 +0000

If you want a sane, secure “SSH config manager macOS” workflow in 2026, you don’t need a new app. You need a hardened ~/.ssh/config that makes the secure path the easy path.

Most people who go shopping for a GUI are really trying to avoid two boring chores: key hygiene and a readable config. If you’re still using one shared key for everything, or you casually ssh -A your agent into random boxes, no macOS SSH manager is going to save you.

This post gives you a copy/paste ~/.ssh/config template built for jump hosts (bastions), clean tunnels, and less yak-shaving. Then I’ll show where macOS GUI “SSH config managers” actually fit without turning your laptop into a credential dumpster.

What is an SSH config manager on macOS?

An SSH config manager on macOS is any workflow or tool that helps you create, organize, and use SSH connection profiles. That can be plain OpenSSH via ~/.ssh/config, or a GUI app that stores hosts, jump chains, identities, and tunnels and then launches SSH sessions for you.

Here’s my bias, upfront. OpenSSH is already the best config manager. It’s audited. It’s everywhere. It’s scriptable. And every other “manager” is basically a UI that eventually runs ssh anyway.

So when someone asks me for a “macOS SSH config manager,” I translate it as: “How do I stop retyping hostnames, stop breaking tunnels, and stop doing sketchy stuff with keys?” Cool. Let’s solve that.

What is the safest way to use a jump host (bastion) with OpenSSH on macOS?

The safest default is boring. That’s a compliment.

Use ProxyJump (-J) through a hardened bastion.
Use per-host identities so compromise of one key doesn’t become “your whole estate is mine.”
Disable agent forwarding by default.
Lock down host key checking so you don’t train yourself to ignore MITM warnings.

In practice, your chain looks like this:

Laptop → bastion (the only host exposed to the internet)
bastion → private target (db-01, k8s-01, whatever)

OpenSSH supports this natively via ProxyJump in config and -J on the command line (OpenBSD Project).

Two numbers I use to keep this from turning into a Rube Goldberg machine:

1 jump host is enough for most small teams. If you’re stacking 2+ hops regularly, you’re usually compensating for missing network segmentation or missing identity tooling.
Multiplexing persistence at 10 minutes (ControlPersist 10m). Long enough that repeated connects don’t feel like punishment. Short enough that you’re not leaving control sockets hanging around all afternoon.

Hardened ~/.ssh/config template for jump hosts (copy/paste)

Most “macos ssh config manager” posts wave their hands here and tell you to “use best practices.” No. You want something you can paste, run, and then tweak.

Set up a simple layout:

~/.ssh/config (tiny, just includes)
~/.ssh/config.d/base.conf
~/.ssh/config.d/jump.conf
~/.ssh/config.d/tunnels.conf

1) ~/.ssh/config

Include ~/.ssh/config.d/*.conf

2) Base defaults (config.d/base.conf)

Host *
  ServerAliveInterval 30
  ServerAliveCountMax 3

  HashKnownHosts yes
  StrictHostKeyChecking ask

  AddKeysToAgent yes

  IdentitiesOnly yes

  ControlMaster auto
  ControlPersist 10m
  ControlPath ~/.ssh/cm/%C

A few quick notes that actually matter:

ServerAliveInterval 30 plus ServerAliveCountMax 3 means you’ll notice dead tunnels in about 90 seconds instead of “why is my command stuck” fifteen minutes later.
IdentitiesOnly yes is the difference between “SSH is calm and predictable” and “Received disconnect: Too many authentication failures” when your agent is full of old keys.
%C gives you a unique control socket name per host/port/user, so you don’t get weird collisions when you bounce between environments.

Important: create the control socket directory with tight permissions:

mkdir -p ~/.ssh/cm
chmod 700 ~/.ssh/cm

If you leave that directory readable to other users on the machine, you’ve turned multiplexing into a local “see what I can poke” situation.

OpenSSH multiplexing lives under ControlMaster, ControlPersist, and ControlPath in the upstream config reference (OpenBSD Project).

3) Jump host + private targets (config.d/jump.conf)

Host bastion-prod
  HostName bastion.prod.example.com
  User ec2-user
  IdentityFile ~/.ssh/keys/prod_bastion_ed25519

Host *.prod.internal
  User ubuntu
  ProxyJump bastion-prod
  IdentityFile ~/.ssh/keys/prod_workload_ed25519

  ForwardAgent no

This is the pattern I want burned into your muscle memory:

Bastion has its own key.
Private workloads have a different key.
Agent forwarding stays off.

ProxyJump is first-class in OpenSSH (-J and ProxyJump). Use it by default instead of dragging ProxyCommand around forever (OpenBSD Project).

How do I set up ProxyJump vs ProxyCommand, and when should I prefer ProxyJump?

Prefer ProxyJump basically always.

ProxyJump is built-in, readable, and supports multiple hops without custom shell quoting.
ProxyCommand is the older escape hatch. It’s still useful when you truly need something odd like traversing a non-SSH transport or wrapping a custom proxy. It’s also easier to screw up.

If your current “ssh jump host proxyjump config” looks like a 200-character ProxyCommand line with nested quotes, you’ve built a footgun. Replace it with ProxyJump.

A concrete smell test: if your jump chain is 1 hop, you should be able to express it as:

config: ProxyJump bastion-prod
CLI: ssh -J bastion-prod db-01.prod.internal

If you can’t, you’re overcomplicating it.

How do I configure ControlMaster/ControlPersist safely, and how do I avoid stale control sockets?

Multiplexing is the highest-ROI quality-of-life feature in SSH. It’s also where people get nervous because “sockets” and “state.” Fair.

The rules that keep it sane:

Keep ControlPersist bounded. I like 10m. If you want to be more aggressive, try 2m.
Put ControlPath in a private directory (chmod 700).
Don’t multiplex across identities. If you reuse the same Host alias but swap User/IdentityFile, you’ll get confusing behavior and you’ll blame SSH for your own config.

Stale control sockets happen when you suspend the laptop, kill Wi‑Fi, or the bastion drops idle TCP. Two ways out:

One-off: ssh -O exit bastion-prod (asks the master connection to shut down)
Nuclear: delete the socket file in ~/.ssh/cm/

The ssh(1) client supports control commands via -O and control paths via -S (OpenBSD Project).

When should you not multiplex?

On shared machines.
When you’re doing high-sensitivity access and you want every session to require a fresh auth boundary.

How do I use per-host identities (IdentityFile + IdentitiesOnly) to prevent failures and key sprawl?

Per-host identities aren’t about being tidy. They’re about blast radius.

The most common failure mode goes like this: somebody has 10–20 keys loaded into their agent, SSH tries them all, the server has a low attempt limit, and the connection gets kicked. Then the “fix” becomes randomly deleting keys until it works.

That’s not engineering. That’s vibes.

The deterministic fix:

Put a specific IdentityFile on each host stanza.
Set IdentitiesOnly yes globally.

Example split:

~/.ssh/keys/prod_bastion_ed25519
~/.ssh/keys/prod_workload_ed25519
~/.ssh/keys/staging_workload_ed25519

Three keys is not overkill. It’s the point.

On macOS, I also like AddKeysToAgent yes so new keys get loaded on first use instead of you babysitting ssh-add all day (OpenBSD Project).

Why is agent forwarding dangerous, and what should I do instead for multi-hop access?

Agent forwarding is dangerous because it extends your authentication capability to the remote machine. If that machine is compromised, your forwarded agent can be abused to authenticate to other hosts.

OpenSSH doesn’t sugarcoat it. ForwardAgent “should be enabled with care” (OpenBSD Project).

My stance: ForwardAgent should be no by default. If a workflow requires ssh -A, treat it like sudo. Explicit, audited, and rare.

What to do instead:

Use ProxyJump for multi-hop access.
Use per-host keys.
If you need a stronger control plane, stop duct-taping SSH and use a managed access system (more on Teleport below).

How do I manage SSH tunnels cleanly with failure detection and keep-alives?

Tunnels are where ad-hoc SSH workflows become actively dangerous.

The classic failure: you think your tunnel is up, you point a tool at localhost:5432, and you’re actually hitting your local machine (or some other service) because the forward never established. Congrats, you’re debugging the wrong system.

Two config patterns prevent this:

1) Make “tunnel-only” stanzas with RequestTTY no and ExitOnForwardFailure yes.
2) Keep them alive with ServerAliveInterval (already in the base defaults).

Example config.d/tunnels.conf:

Host tunnel-prod-db
  HostName db-01.prod.internal
  ProxyJump bastion-prod
  User ubuntu
  IdentityFile ~/.ssh/keys/prod_workload_ed25519

  RequestTTY no
  ExitOnForwardFailure yes

  LocalForward 15432 127.0.0.1:5432

Now you can run:

ssh -N tunnel-prod-db

…and if port 15432 can’t bind, the command fails fast. That’s the whole point of ExitOnForwardFailure (OpenBSD Project).

If you want SOCKS for ad-hoc debugging:

Host socks-prod
  HostName bastion.prod.example.com
  User ec2-user
  IdentityFile ~/.ssh/keys/prod_bastion_ed25519

  RequestTTY no
  ExitOnForwardFailure yes
  DynamicForward 1080

That’s your “ssh tunnel manager mac” without installing anything.

How should I handle known_hosts and host key rotation securely?

Everyone ignores this until a contractor trains the whole team to type “yes” on muscle memory. Then you get a real MITM warning and nobody believes it.

My defaults:

StrictHostKeyChecking ask (not no).
HashKnownHosts yes so your known_hosts file is less useful if someone steals your laptop.

For teams, I like separating known_hosts by environment so you don’t end up with “prod and dev have the same hostname and now everything is broken.” Use per-stanza UserKnownHostsFile if you need to.

Host key rotation is real in 2026. If your servers support it, UpdateHostKeys can help clients learn additional keys (OpenBSD Project).

The directive matters less than the policy. Make it explicit:

Who approves host key changes?
Where do you announce rotations?
What’s the escalation path when someone sees a mismatch?

SSH config manager macOS: GUI tools vs plain OpenSSH config

GUI managers are tempting for the same reason IDEs are. They make the common path easy.

The downside is also the same. They introduce a second truth.

Here’s the pragmatic comparison.

Option	Best for	Where it bites you	Security posture
Plain `~/.ssh/config`	Engineers, CI, reproducible setups	Initial learning curve	Strong. You control keys, host checks, and defaults
SecureCRT	Heavy terminal users who want session management + scripting	Paid, another place to store connection metadata	Good if you keep keys in files/agent and don’t duplicate secrets. Product details: VanDyke Software
Royal TSX	Teams that want shared connection lists without sharing creds	Easy to over-centralize connection docs	Good if you use its credential separation model. Feature overview: Royal Apps

I’m not anti-GUI. I’m anti “GUI as a replacement for fundamentals.”

If you adopt a GUI, demand two things:

It must import/export cleanly to ~/.ssh/config.
It must not push you toward agent forwarding or shared keys just to make setup “simple.”

How can I record SSH sessions for auditing on jump hosts?

If you’re serious about production access, authentication logs aren’t enough. “User X logged in” tells you nothing about what they did after.

Two practical options:

Host-level session recording with tlog on Linux jump hosts. It records terminal I/O so you can replay sessions later (tlog maintainers).
Managed access systems that treat auditing as a first-class feature. Teleport is the obvious example, with an audit log and session recording as part of its SSH proxy model (Teleport docs move around a lot, but the product’s stance is consistent).

A simple operational rule: if you have compliance requirements, or you have more than 5–10 engineers doing prod access, the time you’ll waste arguing about “who ran what” will quickly exceed the cost of doing session recording properly.

How do GUI SSH managers interact with the OpenSSH config file, and what’s a safe hybrid workflow?

A safe hybrid workflow looks like this:

~/.ssh/config is the source of truth.
The GUI reads from it (or you keep them aligned manually, but then be honest that you’re doing double entry).
Keys live as files with correct permissions, or in the OS keychain. Not copied into random app-specific vaults unless you’ve threat-modeled that.
Tunnels are defined as dedicated host stanzas (tunnel-prod-db, socks-prod) so you can run them from Terminal, the GUI, or scripts.

If your team can’t reproduce a connection with ssh -F ~/.ssh/config ..., you don’t have a workflow. You have a collection of personal snowflakes.

What I’d do next if you want to roll this out: rotate a single environment key and see how many people break. If the answer is “most of them,” that’s not a reason to buy a shinier SSH config manager for macOS. It’s your signal to standardize the template, fix the hygiene, and make secure defaults non-optional.

Originally published on kunalganglani.com

How to Secure Docker Rootless Mode in Production [2026]

Kunal — Sun, 30 Aug 2026 18:26:16 +0000

How to Secure Docker Rootless Mode in Production [2026]

I keep seeing the same pattern. A team flips on Docker rootless mode, feels safer, and moves on. Then six months later someone adds a privileged container “just for debugging,” the socket ends up writable by half the org, and rootless becomes a comforting story instead of a control.

The keyword people search for is docker rootless mode security. Here’s the uncomfortable truth: rootless is a good first move, not hardening. Hardening is what you do after the first move.

By the end of this guide you’ll have Docker rootless mode running under a systemd user service, plus a production-ready baseline: userns-remap decisioning, seccomp + AppArmor, safer container flags, and a short verification playbook you can paste into an audit note. Budget 60–90 minutes if you’re starting from a fresh host.

Docker’s docs have been updated through 2026 (rootless: 2026-06-07, userns-remap: 2026-04-14, seccomp: 2026-08-21). Rootless is mature enough to be a default. It’s not a magic forcefield.

If you want the broader “defensible controls” mindset, I’ve been building deterministic gates for this site’s 7-agent publishing pipeline. One lesson keeps repeating: a single control rarely survives contact with reality. Defense-in-depth does.

What is Docker rootless mode and why is it considered more secure?

Docker rootless mode is a Docker Engine configuration where both the Docker daemon and containers run as a non-root user, reducing the impact of a daemon compromise because the daemon no longer has root privileges on the host. Per the official Docker documentation, the core win is shrinking the daemon’s privilege boundary.

Why it’s considered “more secure” is pretty specific:

The classic rootful Docker daemon is a high-value target. If an attacker gets code execution in the daemon, they’re often one step from host root.
Rootless makes “daemon RCE == host root” a much harder chain. The attacker lands as an unprivileged user.

That’s it. That’s the win.

Rootless does not remove the kernel attack surface. It does not make your containerized app immune to RCE. It does not fix secrets hygiene or network egress. If you’re looking for a single switch that makes audits go away, you’re going to have a bad time.

How it works (and where the boundaries really are)

The clean mental model is to separate three things:

The Docker daemon (builds, pulls, mounts, creates namespaces, manages networking).
The container process (your app).
The Linux kernel (shared substrate that you still trust with your life).

In rootful Docker, the daemon runs as root. In rootless Docker, the daemon runs as a normal user and uses user namespaces plus helper components to do the namespace work without elevated privileges.

In practice, rootless setups commonly lean on:

RootlessKit for unprivileged namespaces and port forwarding (RootlessKit).
slirp4netns for user-mode networking (slirp4netns).
fuse-overlayfs for overlay filesystem behavior without privileged kernel overlay usage (fuse-overlayfs).

Those building blocks explain the trade: you’re often swapping kernel fast paths (iptables, kernel overlayfs) for userland components. Security gets cleaner in one dimension. Performance and “weird edge-case behavior” sometimes get worse in others.

Rootless vs userns-remap: same ingredient, different blast radius

People mash these together because both use user namespaces. But they solve different problems.

Rootless changes the daemon’s privilege level. Big deal.
userns-remap keeps the daemon rootful, but remaps container UIDs/GIDs so container root maps to an unprivileged host range.

If your auditor asked “prove container root isn’t host root,” either can be part of that story. If you’re specifically worried about the Docker daemon being a loaded gun, rootless is the more direct move.

This is one of those things where the boring answer is actually the right one: pick based on what you’re afraid of. If your org is worried about daemon exposure and “docker group == root,” rootless is the cleaner story.

Prerequisites (the production stuff, not the happy path)

Before you install anything, check these or you’ll end up with an “it works on my laptop” rootless daemon that flakes out under load.

1) Kernel + cgroups reality check

On a modern distro, you should be thinking cgroup v2. Rootless resource controls are much saner there.
On older hosts with cgroup v1 weirdness, treat rootless as a project, not a checkbox.

Concrete check:

Verify what you’re running: stat -fc %T /sys/fs/cgroup (you want cgroup2fs for v2).

2) A dedicated service user (don’t run rootless as “whoever logged in”)

Create a non-login service user like docker or dockerd and treat it as infrastructure.

I’m opinionated about this because access boundaries that depend on “who happened to install it” don’t survive staff changes. Or weekend debugging.

3) Subordinate UID/GID ranges (you’ll need them for user namespaces)

Even in rootless land, you still care about /etc/subuid and /etc/subgid.

Concrete check:

Confirm entries exist: grep -E '^(docker|dockerd):' /etc/subuid /etc/subgid

If you don’t have these, you don’t have a real user-namespace mapping story. You have vibes.

Install (Rootless Docker) with systemd user services

I’m not going to retype Docker’s installation steps. Use the official rootless docs for your distro: Docker rootless mode.

What I will do is call out the production pattern that’s missing from a lot of writeups: run it as a systemd user service, and treat the socket as the choke point.

Safe production pattern: rootless daemon, explicit socket access

Install and enable the rootless daemon under your dedicated user.
Ensure the daemon uses a user-level socket (usually under the user’s runtime dir).
Control who can talk to that socket.

If a developer can reach your Docker socket, they can usually get “admin-equivalent” power over whatever that daemon can do. Rootless narrows that equivalence to the daemon’s user. That’s still catastrophic if that user owns prod data volumes.

Operational note: if you do centralized logging, make sure journald logs for the user service are shipped. Otherwise you’ll have a “secure” setup with zero usable telemetry when something goes sideways.

Rootless Docker vs userns-remap: what’s the difference, and which should I use in production?

This is the decision table I wish teams put in their internal runbooks.

Dimension	Rootless Docker	userns-remap
Daemon privilege	Non-root	Root
Primary security win	Smaller blast radius if daemon is compromised	Container root maps to unprivileged host IDs
Ops friction	Higher (networking, ports, storage can differ)	Medium (volumes, permissions, some features)
Common gotcha	Port binding, throughput, filesystem backend	File ownership on bind mounts and named volumes
Best for	Hosts where daemon risk is a top concern	Environments where rootless breaks workloads but you still want UID isolation

My stance: if you can run rootless without breaking SLOs, do it. If you can’t, don’t force it. Use userns-remap plus hardening layers and move on with your life.

About remapping and subordinate user and group IDs

Docker’s userns-remap maps container UID 0 to an unprivileged host UID/GID range defined in /etc/subuid and /etc/subgid (Docker userns-remap docs). That mapping is the entire point.

In English: “root in the container” becomes “some random high-numbered UID on the host.”

That’s a real safety improvement for a broad class of container breakouts that rely on host UID 0.

Enable userns-remap on the daemon

Enable it at the daemon level using Docker’s documented configuration flow (userns-remap).

The production gotcha is usually storage + bind mounts. Once IDs are remapped, file ownership semantics change. If your app expects to write to a bind mount owned by real UID 1000, you can end up with permission failures that feel random until you remember you asked the OS to lie about identity.

Disable namespace remapping for a container

Docker supports disabling remapping per container in documented ways. In production, doing that is basically admitting you have a snowflake workload. Use it sparingly. Document why it exists. Put a date on revisiting it. Exceptions have a way of turning into architecture.

User namespace known limitations

The official docs list limitations and edge cases. Treat them as design constraints, not bugs. The real ops reality is that volume permissions and “who owns what on disk” becomes a first-order concern.

Rootless Docker mode limitations (networking, storage, ports, cgroups)

This is where rootless stops being “security blog advice” and turns into a change request you actually have to defend.

Ports: the <1024 issue

Binding privileged ports (below 1024) is traditionally a root-only operation. Rootless setups often require workarounds (reverse proxy, host-level port forwarding, or running the listener behind something that can bind 80/443).

If you’re running a single-node service directly on 80/443, you will feel this immediately.

Networking: slirp4netns tradeoffs

Rootless commonly uses user-mode networking via slirp4netns. That’s a different data plane than “host bridge + iptables.”

In production, that can mean:

Lower throughput
Different MTU behavior
Debugging that looks nothing like your rootful hosts

If your service’s SLO is tightly coupled to throughput or tail latency, benchmark before rollout. I don’t care how many blog posts say it’s “fine.” Measure it on your workload.

Storage: fuse-overlayfs and filesystem semantics

Rootless often leans on fuse-overlayfs for overlay filesystem behavior without privileged kernel overlay configuration.

That’s usually fine for typical web workloads. It can be ugly for IO-heavy jobs.

A concrete rule: if your containers do sustained disk IO and you care about P99, rootless is a performance change. Treat it like one.

cgroups: resource limits are not optional

Rootless without resource limits is just “unprivileged chaos.” Put CPU and memory limits on everything. Even if nobody’s attacking you, a runaway process is operationally indistinguishable from a DoS.

If you run Kubernetes, you already live here. If you run Docker directly on hosts, this is where the weird 2 a.m. incidents come from.

Rootless Docker security checklist (production)

This is the list I’d want to hand an auditor. It’s also the list I’d want oncall to have when a container starts behaving like it’s possessed.

Run the daemon as a dedicated non-root user (not your personal account).
Lock down socket access. Treat “can access the Docker socket” as equivalent to “has admin power over that host.”
Pin and scan base images. Use digests, not mutable tags.
Drop Linux capabilities aggressively. Start from --cap-drop=ALL and add back only what’s required.
Enable no-new-privileges for containers that don’t need privilege escalation.
Use a read-only filesystem where you can (--read-only) and mount explicit writable tmpfs paths.
Set explicit resource limits: CPU, memory, pids limit.
Keep seccomp on. Don’t use --security-opt seccomp=unconfined unless you have a signed exception.
Use AppArmor profiles where available, and log denials.
Treat secrets as hostile: no long-lived secrets in env vars. Prefer short-lived identity and external secret stores.

I’m deliberately mixing security and ops here. In real systems they’re the same thing. The “secure” setup that pages you every night gets disabled, and then you’re worse off than when you started.

For adjacent hardening checklists, see how I structure defense-in-depth for AI in production and the supply chain angle in LLM security. Same principle: reduce blast radius, then reduce reachable surfaces.

Docker seccomp + AppArmor hardening (still matters in rootless)

Rootless changes who the daemon is. seccomp and AppArmor change what the container can do. Different layer.

Pass a profile for a container (seccomp)

Docker supports passing a custom seccomp profile per container (Docker seccomp security profiles). Use this when a workload needs a syscall Docker blocks by default, but you don’t want to go fully unconfined.

Significant syscalls blocked by the default profile

Docker’s default profile blocks a set of “significant syscalls” (their wording) because they show up in breakout chains or are rarely needed by typical apps. The exact list changes over time, but the lesson doesn’t: the default seccomp policy is a baseline. It’s not tuned for your workload.

Run without the default seccomp profile

If you do --security-opt seccomp=unconfined, write down why, set an expiration date, and add monitoring.

I’ve seen this exact failure mode in my own infra work. In the deterministic gates I run for this site’s publishing pipeline, “temporary exceptions” become permanent in about 2 weeks unless someone is explicitly on the hook to remove them.

Understand the policies (AppArmor)

AppArmor is mandatory access control at the kernel level. Docker can apply AppArmor profiles to containers (Docker AppArmor security profiles).

If you’re on Ubuntu, AppArmor is usually already there. That’s the good news.

Load and unload profiles

Docker’s docs cover how profiles are loaded and referenced. The operational point that matters is boring: your deployment system needs a way to ship profiles to hosts, and you need a rollback plan.

Security controls without rollback are just outage generators.

Debug AppArmor

The Docker AppArmor docs call out using dmesg and aa-status to debug denials. In production this matters because AppArmor failures look like “my app randomly can’t open a file.”

You want denials routed to your log pipeline and alert thresholds that don’t spam.

If you’re also building agent systems, the analogy is direct. prompt injection failures look like “the model is being weird.” Without logs, you guess.

What threats rootless does NOT protect you from

This is the part most teams skip. It’s also where security theater is born.

Malicious images / supply chain: Rootless doesn’t stop curl | bash inside your container from exfiltrating data.
Kernel CVEs: You still share the kernel. Rootless doesn’t change that boundary.
App-layer RCE: If your service has RCE, the attacker can still steal data reachable by that service.
SSRF to cloud metadata: Rootless doesn’t prevent your app from hitting 169.254.169.254 if it can route there.
Secrets exposure: Secrets in env vars, baked into images, or sprayed into logs are still your fault.
Lateral movement over the network: Rootless doesn’t segment networks or enforce egress controls.
Writable Docker socket access: If an attacker can access your Docker socket, rootless just changes which host user they become. That can still be enough to own the box.

If this feels repetitive, good. The pattern repeats across domains. In agent systems, people fixate on one control and ignore the kill chain. See AI agents security discussions. Same shape, different nouns.

How to verify protections are actually active

If you can’t verify it, you can’t defend it.

Here’s a practical verification list you can run on a host and capture in a ticket.

Daemon is rootless: confirm the daemon process isn’t running as UID 0.
User namespace mapping: verify /etc/subuid and /etc/subgid entries exist for the daemon user.
Seccomp active: run a container and inspect whether seccomp is in effect. If it’s unconfined, you should know why.
AppArmor active: confirm the profile is loaded and applied, and that denials show up in logs.
Socket permissions: check who can access the Docker socket. Make sure it matches your least-privilege model.

This kind of checklist thinking is how I run my own infra. I maintain benchmark pages at kunalganglani.com/llm-benchmarks, and the recurring lesson is that “defaults” only stay true if you keep verifying them.

Safe production patterns for running the rootless daemon

If you want a setup that won’t get quietly weakened over time:

Run the daemon as a dedicated user.
Make the socket path explicit, and gate access via group membership or systemd socket activation rules.
Prefer immutable infra patterns. Reprovision hosts instead of hand-editing them.
Patch cadence matters more than perfection. Rootless reduces blast radius. Patching reduces the chance you test that blast radius.

If your org is building more automation around deployments, read CI/CD as “security plumbing,” not “developer convenience.” That’s where the real hardening lives.

My prediction: by late 2027, auditors will treat “rootless + seccomp + AppArmor + socket controls” as the baseline the way “TLS everywhere” became boring. The teams that win won’t be the ones with the fanciest container runtime. They’ll be the ones that can prove, continuously, what’s actually enforced on every host.

Originally published on kunalganglani.com

DEV Community: Kunal

Prevent API Key Leaks in Shell History (bash/zsh/fish) [2026]

What is terminal secrets hygiene?

Shell history basics: where history is stored and why secrets leak

How do I stop a command from being saved in shell history (bash/zsh/fish)?

Bash: history controls (HISTCONTROL/HISTIGNORE) and safe patterns

Zsh: history controls (setopt options) and safe patterns

Fish: private mode/history settings and safe patterns

Environment variable scoping: avoid exporting globally; use per-command/env wrappers

1) Prefer per-command environment variables

2) Prefer stdin or files over argv

3) Reduce lifetime: unset aggressively

Dotfiles hygiene: avoid committing secrets; .gitignore patterns; separate local/private files

Use env managers: direnv workflow for per-directory secrets; mise patterns for tasks/env

direnv: local secrets that load on cd and unload on exit

mise: define env and tasks together, but keep secrets out of mise.toml

Add secret scanning: pre-commit + repo scanners; GitHub secret scanning/push protection

Local scanning: gitleaks and trufflehog

Remote scanning: GitHub Secret Protection and push protection

When a key leaks: rotate/revoke, scrub history, scrub git history, and notify/log review

Step 1: Revoke or rotate immediately

Step 2: Figure out where it leaked

Step 3: Scrub shell history locally

Step 4: If it hit git, rewrite history (properly)

Step 5: Prevent reintroduction

Step 6: Audit

The safest way to provide API keys to CLI tools locally (my default)

How to Set Up a Local AI Hub for Home Automation Privacy [2026]

What is a local AI hub?

Step 1: Map what runs local vs what might still hit cloud

What should be able to run locally (and how to test it)

What commonly requires cloud (even on “local-first” products)

The privacy stance I recommend

Step 2: Segment your LAN (so local traffic works, internet doesn’t)

Minimal addressing plan

Why “Hub VLAN” is worth it

Step 3: pfSense rules that keep discovery working (without “allow any”)

The baseline policy

The “things that usually break” allowlist

Practical pfSense pattern

Step 4: Verify outbound traffic (DNS logs → SNI/pcap → firewall logs)

4A. DNS query logs (Pi-hole)

4B. Packet capture + SNI with Wireshark/tcpdump

4C. Firewall logs (pfSense)

Step 5: Offline acceptance test (prove it still works when WAN is down)

Step 6: Firmware updates without turning your privacy model into Swiss cheese

Step 7: Common pitfalls (and how to mitigate them)

Hardcoded DNS (device ignores your resolver)

DoH/DoT (DNS over HTTPS/TLS)

QUIC/HTTP3 hides some hostname cues

Certificate pinning (TLS inspection won’t help)

“Local AI” that still uses cloud LLMs for the hard part

My prediction: “local AI” will become a networking problem, not an AI problem

Xbox Cloud Gaming Pay‑As‑You‑Go Latency: Fix Input Lag Fast [2026]

Xbox Cloud Gaming requirements (bandwidth, devices)

What causes input lag in Xbox Cloud Gaming beyond just low bandwidth?

How much latency is “playable” for different game genres?

Wi‑Fi vs Ethernet for cloud gaming

The Wi‑Fi rule I trust

Practical Wi‑Fi settings that actually matter

QoS/SQM and bufferbloat fixes (the big win)

What to do about it: SQM (CAKE / fq_codel)

Common SQM pitfall

How to test/measure latency (baseline vs under load)

The 15-minute measurement plan

Measure while actually playing

Service caps vs home network problems (what you can’t fix)

Is it your TV/monitor or the network? (isolating display lag)

A quick troubleshooting table (symptom → measurement → fix)

My stance: if you’re paying per hour, treat your network like a budget

Polars 2.0 Upgrade Guide [2026]: Streaming Default + CI Bench

Polars 2.0 Upgrade Guide: Breaking Changes, Streaming Default, and a CI Regression Harness

What is Polars 2.0 (and why does the upgrade feel scary)?

Streaming engine as default: the Lazy API now streams unless you stop it

How do I keep old behavior (in-memory engine) for collect()?

When does streaming change row order, and how do I maintain order?

Why did my join/group_by output order change after upgrading?

Maintain order only where it matters

Breaking changes checklist: what actually breaks in real codebases

CSV reading changes and eager/lazy surprises (read_* now dispatches to scan_*().collect())

direnv: local secrets that load on `cd` and unload on exit

mise: define env and tasks together, but keep secrets out of `mise.toml`

How do I keep old behavior (in-memory engine) for `collect()`?

What should I change if my code uses `pl.read_csv` / `pl.read_ipc`?

`is_in()` strict coercion (lossy casts no longer happen)

`pl.concat()` / `pl.union()` strict behavior

API reshapes, deprecations, and the `LazyFrame.profile()` replacement

`LazyFrame.profile()` is removed

What replaces `LazyFrame.profile()` in 2.0 and how do I profile queries now?