DEV Community: Kunal

How to Reduce Rust Compile Time [2026] (sccache + mold)

Kunal — Sun, 16 Aug 2026 12:44:24 +0000

How to Reduce Rust Compile Time 2026

If you take one thing from this post, let it be this: you can reduce Rust compile time without cargo-culting flags. You need a harness. A few numbers you can trust. Then you change one variable, re-run, and decide if it was worth the churn.

You’ll finish this guide with a repeatable setup that tells you, in minutes, whether you actually reduced compile time. Not “it feels faster”. Real numbers: cold build time, warm build time, incremental rebuild time after a one-line edit, sccache hit rate, and linker time.

If you run a Rust monorepo in 2026, compile time isn’t a “developer experience” nit. It’s a CI bill. It’s also a throughput cap. When builds are slow, everything gets worse: smaller refactors stop happening, reviews batch up, and people start treating the build as an unpredictable weather system.

Here’s my stance: stop chasing random Cargo knobs. Treat compile time like observability. Measure the build graph, then change one thing at a time.

Before we start: the measurement mindset here is the same one I use for cost and performance work on this site. I maintain a live LLM pricing tracker at /llm-prices, and the main lesson from that project is that numbers without assumptions are propaganda. Build time “improvements” are the same. If you don’t separate cold vs warm vs incremental, you’re lying to yourself.

What is Rust compile time optimization?

Rust compile time optimization is the set of techniques that reduce the elapsed time from cargo build to a usable binary by improving build graph parallelism, avoiding unnecessary recompilation via incremental builds and caching, and shrinking expensive compiler and linker work.

In practice, you’re optimizing three things:

The crate dependency graph (parallelism and invalidation boundaries)
Reuse (incremental compilation + local/remote compilation cache)
Linking (often the last big serial step)

A measurable harness to reduce Rust compile time

You can’t optimize something you can’t re-run. The fastest way to waste a week is to “try a bunch of stuff” and then argue in Slack about whether it helped.

So I always start with a tiny benchmark script and a rule: same target, same features, same toolchain, same environment. Every time.

The 5-metric baseline

Run these and record them in your PR description:

Cold build: clean build from scratch.
Warm build: immediate rebuild with no changes.
Incremental rebuild: rebuild after a one-line edit in a leaf crate.
sccache hit rate: hits / requests.
Link time: seconds spent in linking.

Concrete targets that usually matter in real teams:

Cold build down by 20–50% is “worth telling the org”.
Warm build under 5s is where iteration stops feeling sticky.
Incremental rebuild under 1–2s for leaf edits is the happy place.
sccache hit rate above 70% in CI is where it starts paying for itself.

If you’re not sure what to target, start by picking the one that’s actively annoying you.

How do I separate cold build vs warm/incremental build measurements?

Do it explicitly:

Cold: cargo clean then cargo build
Warm: run cargo build again with no edits
Incremental: edit one file in one crate, rebuild once

Keep the environment constant: same toolchain, same target, same RUSTFLAGS, same feature set.

If you’re in a workspace, always build the same thing. “I ran cargo build” is not a stable benchmark if half the time you’re building examples/tests.

Add build graph visibility: cargo build --timings

Cargo timings is the first diagnostic tool I reach for because it forces the conversation into reality. It generates an HTML Gantt chart that shows crate dependencies and how much parallelism you’re actually getting.

Nicholas Nethercote (Mozilla alum) has the best practical write-up I’ve seen on this whole topic. Read it. Then come back. Nicholas Nethercote.

Run:

cargo build --timings

Open the generated HTML. The “aha” you’re looking for is usually boring:

One crate eating 30–60% of the total build.
A long serialized chain where only 1 crate is compiling at a time.

That’s not a “Rust is slow” problem. That’s your crate graph telling you it’s shaped badly.

A snippet-friendly checklist (print this)

Generate a cargo build --timings report and identify the top 3 crates by time.
Split any crate that serializes the graph (or move heavy code behind a feature).
Turn on incremental in dev, and keep release separate.
Install sccache, verify hit rate, then add remote caching for CI.
Swap the linker to mold on Linux and measure link time again.
Trim features and optional dependencies until the timing report changes.
Hunt macro and IR bloat in the top crates (-Zmacro-stats, cargo llvm-lines).

Visualization (cargo build --timings): find the crates that dominate

How do I measure which crates dominate my Rust build time?

Use the timings report, not vibes.

The timings HTML gives you per-crate durations. Take the top offenders and write them down. If one crate is 9s of a 14s build, you already know where to spend your next hour.

Two patterns show up constantly:

The “kitchen sink” crate: one crate owns half the domain model, half the macros, half the dependencies. It blocks the whole build.
The proc-macro chokepoint: everything depends on a proc-macro crate, so compilation waits for it.

Fast builds come from wide graphs. Narrow graphs feel “clean” right up until you have to work in them.

When should I split a workspace/crate to increase parallelism, and what are the tradeoffs?

Split when the timings report shows a crate that:

takes >25% of build time, and
sits early in the dependency chain (lots of crates depend on it).

Tradeoffs (the real ones):

More crates means more Cargo.toml surface area.
You can accidentally increase compile time if you create lots of tiny crates that each pull in the same heavy deps.
API boundaries harden. That’s good for architecture, but it can feel annoying in early-stage products.

Rule I follow: split by invalidation boundaries. If code changes together, keep it together. If code changes independently, make it a crate.

For example, an “api-types” crate that changes every PR is a terrible foundation crate. Push stable things down (core traits, shared error types), and pull unstable things up.

Cargo profile settings (incremental, lto, codegen-units)

Cargo profiles are where you separate “fast local iteration” from “optimized release binary”. Profiles control compiler settings like incremental, LTO, codegen units, debug info, and assertions. The canonical reference is the Rust Project.

One thing I’ll say out loud because teams keep messing this up: if you optimize for release builds at the expense of dev iteration, you pay for it daily. Don’t do that. Make release expensive on purpose. Make dev cheap on purpose.

What Cargo profile knobs most affect compile time for dev vs release?

The knobs that actually move build time:

incremental: big dev win, sometimes noisy in CI.
opt-level: higher is slower. 0 and 1 are usually the sweet spot for dev.
lto: can add minutes on large binaries.
codegen-units: more units can improve parallelism but can change performance and compile time tradeoffs.
debug: full debug info can be expensive. Line-tables-only is often enough.

A sane baseline:

dev: prioritize iteration.
release: prioritize runtime.

If your team is shipping often, add a middle profile like release-fast for staging builds.

Concrete numbers that matter: on a typical service binary, flipping lto = true can turn a 30s release build into 90–180s, depending on code size. That might be fine for nightly. It’s usually not fine for every PR.

What invalidates incremental compilation most often, and how do I reduce dep-graph churn?

Incremental compilation works when edits stay local. It falls apart when your changes force large parts of the dependency graph to be reconsidered.

The churn culprits that tend to be brutal:

Editing a foundational crate that dozens depend on.
Re-export patterns that make “small change” look “global”.
Heavy macro-generated code that changes in big chunks.
Build scripts (build.rs) that embed timestamps or environment-dependent outputs.

Mitigations you can actually ship:

Keep “core” crates stable. Move fast-changing code upward.
Avoid needless pub use fan-out in a root crate.
Make build scripts deterministic. If a script reads the filesystem, lock it down.
Trim features so fewer crates participate in the build in the first place.

sccache installation and usage (local + CI)

sccache is a compiler wrapper that avoids recompilation when possible and can use local or remote cache backends. That’s straight from the project README: Mozilla.

The mistake people make is treating sccache like a magic on-switch. It’s not. It’s a multiplier. If your build is nondeterministic or your graph invalidates constantly, sccache can’t save you.

Install sccache locally

If you’re already in Rust land, the simplest path is usually:

cargo install sccache

Then set the wrapper:

export RUSTC_WRAPPER=sccache

Verify it’s active:

sccache --version
sccache --show-stats

The stat you care about is “Compile requests” vs “Cache hits”. After a second build, you should see hits.

Configure sccache for Rust in CI (remote cache)

Local caching is nice. Remote caching is what moves your CI bill.

The basics:

Pick a remote backend your org already trusts (S3, Redis, etc.).
Ensure the cache key is stable across machines.

The most common reason CI hit rates stay awful is path differences. You check out the repo under /home/runner/work/... in CI and /Users/kunal/... locally, and the compiler bakes paths into artifacts.

sccache supports path normalization via SCCACHE_BASEDIR / base-dir normalization options (see the README section on normalizing paths). If you don’t normalize, expecting >70% hit rate across ephemeral runners is wishful thinking.

How do I verify cache hit rate?

Use:

sccache --show-stats

Track it as a metric.

If your CI does 200 compile requests and gets 20 hits, that’s a 10% hit rate. You don’t have a caching system. You have a warm feeling.

When does sccache not help, and how do I mitigate?

sccache won’t save you when:

Your code changes invalidate everything.
You rely heavily on procedural macros that themselves change often.
build.rs scripts emit different outputs per machine.
Your environment leaks into compilation (paths, env vars, feature flags).

Mitigation playbook:

Normalize paths.
Freeze toolchains (pin Rust version in CI).
Make builds deterministic (lock down build scripts).
Trim features so fewer crates are in the build graph.

If you do all that and still see low hit rates, don’t get religious about caching. Fix the graph and the invalidation boundaries first.

mold installation and usage: cut link time (Linux-first)

mold is designed to be a faster, drop-in replacement for existing Unix linkers. That’s the pitch, and it mostly delivers, per Rui Ueyama.

Linking is often the most annoying part of a Rust build because it’s serial. You can compile crates across 16 cores. Then you hit the link step and watch one core do all the work.

Install mold

On Linux, you typically install mold via your package manager (or build from source).

Quick sanity check:

mold --version

Use mold with Cargo / rustc

There are a few ways to do this; the simplest is setting a linker override via RUSTFLAGS.

For example:

RUSTFLAGS="-C link-arg=-fuse-ld=mold" cargo build

(Exact flags vary by toolchain and platform. The point is: make Cargo use mold as the linker.)

How do I confirm link time improved?

Go back to the timings report.

Run cargo build --timings before.
Switch to mold.
Run cargo build --timings again.

You’re looking for the link step dropping from something like 4–8s to 1–3s on medium-sized binaries. On bigger monorepos, link time can be 10s+. Those are the wins that change how the repo feels.

Platform nuance (macOS + Windows)

Linux: mold is the straightforward win.
macOS: you’re constrained by Apple’s linker ecosystem. You can still reduce link work by trimming debug info and avoiding pathological dependency graphs.
Windows: linker behavior is different again. Don’t assume a Linux linker swap translates.

Measure it. If it didn’t move, undo the change and move on.

Here’s the official demo-style video if you want the quick version before you change a bunch of build plumbing:

[YOUTUBE:NE4Exqt_0z8|Rust Compiles Faster NOW!]

Macros (-Zmacro-stats) and LLVM IR bloat: find compile-time blowups

Once you’ve squeezed the “graph and tools” wins, the remaining pain is usually self-inflicted. Macro explosion, generic explosion, or both.

Nicholas Nethercote recommends using the nightly -Zmacro-stats flag to quantify how much code macros generate. See Nicholas Nethercote.

What code patterns cause compile-time blowups, and how do I find them?

Two common offenders:

Procedural macros generating code on the same order of magnitude as your hand-written code.
Generics/monomorphization creating lots of instantiations, ballooning LLVM IR.

How to find macro bloat:

RUSTFLAGS="-Zmacro-stats" cargo +nightly build

Numbers to watch:

If a proc macro generates 200k lines of expanded code, that’s not “just build tooling”. That’s a build-time tax you chose.
If expanded code is 1x your handwritten code, you should at least question it.

How to find LLVM IR bloat:

cargo llvm-lines

Generic functions are often top offenders because they can be instantiated dozens or hundreds of times in large programs. If a single generic function produces a disproportionate amount of IR, make it smaller or reduce the generic surface area.

This is also where feature trimming pays off. If you can compile 30% fewer crates, you often avoid compiling the most macro-heavy optional stuff entirely.

Feature flag trimming: the most underrated lever

Optional dependencies are not optional to your compile time if you compile with default features everywhere.

A safe workflow I like:

Run timings with your current features.
Disable default features on heavy dependencies.
Add back only what you need, crate-by-crate.

If this sounds like work, it is. But it’s clean work. And it pays you back every single day you touch the repo.

A before/after table (what “good” looks like)

Use a table like this in your repo. Fill it with your own numbers.

Change	Cold build	Warm build	1-line edit rebuild	Link time	sccache hit rate
Baseline	14.2s	6.1s	3.4s	5.2s	0%
+ incremental dev profile	14.2s	5.4s	1.9s	5.2s	0%
+ sccache local	14.2s	2.1s	1.2s	5.2s	68%
+ sccache remote (CI)	14.2s	2.1s	1.2s	5.2s	82%
+ mold	14.2s	2.1s	1.2s	1.9s	82%

Those numbers are illustrative. The important thing is the shape: caching and linker swaps should show up as obvious step changes. If they don’t, your config isn’t working.

One more data anchor, since this blog is obsessed with measurable engineering: based on the LLM pricing tracker I maintain at /llm-prices, small per-unit costs compound brutally at scale. CI minutes are the same. If your org runs 500 builds/day and you shave 60s off each, that’s 8.3 hours of compute time saved every single day.

My opinionated ordering (what to do Monday morning)

If you’re trying to reduce Rust compile time in a real codebase, here’s the order I’d do it:

Generate timings and identify the top 3 crates.
Fix the crate graph (split, move code, isolate proc-macros).
Turn on incremental for dev. Keep release separate.
Install sccache locally. Verify hits.
Add remote sccache in CI and chase hit rate.
Swap to mold on Linux. Measure link time.
Hunt macro/IR bloat only in the crates that show up at the top.

Everything else is busywork.

If you want the broader theme behind this, I wrote a metrics-first piece on observability here: How to Pick LLM Application Observability Metrics [2026]. Same energy. Different bottleneck.

And if compile time pain is showing up as workflow friction and review latency, you’ll probably also like my copy-paste process for Stacked PRs on GitHub and the CI framing in AI Code Review in Your CI/CD Pipeline.

My prediction: by the end of 2026, the teams that feel “fast” won’t be the ones with the fanciest codegen or the most magical build system. They’ll be the ones that put build time on the same dashboard as test duration and deploy frequency, and treat the crate graph like architecture, not an accident.

Originally published on kunalganglani.com

Bun vs Node Performance 2026: Cold Start, Memory, WebSockets

Kunal — Sun, 16 Aug 2026 00:42:20 +0000

You’re not deciding between Bun and Node for “developer happiness.” You’re deciding whether your next service will autoscale cleanly, fit 2× more pods per node, and keep 50,000 WebSocket connections from turning into a permanent CPU tax.

If you’re searching bun vs node performance 2026, you probably already know the hot takes. Bun is “fast.” Node is “mature.” Cool. That doesn’t help when you’re staring at p95 cold starts, tight memory limits, and a real-time feature that suddenly made your fleet stateful.

Here’s what actually matters in 2026:

Cold start is the most honest benchmark for modern infra. It’s what autoscaling and serverless feel like when nobody’s watching.
Memory isn’t heapUsed. RSS decides container density and whether Kubernetes evicts you.
WebSockets are where runtimes get exposed. Churn, backpressure, and broadcast patterns punish sloppy event loops.
Bun can be a clean win, but only in specific service shapes. Greenfield HTTP + WS services benefit the most.
Node still wins by default when your app depends on the ecosystem. LTS cadence, native addons, and tooling depth matter in boring but expensive ways.

What is Bun vs Node performance 2026?

Bun vs Node performance 2026 is the real-world comparison of Bun’s JavaScriptCore-based runtime and Node.js’s V8-based runtime across the metrics that actually hit production: cold start latency, memory footprint (RSS), and WebSocket throughput/connection handling.

When people argue about “performance,” they usually mean throughput on a microbenchmark. In 2026, most teams feel performance as:

Tail latency during scale events (cold start p95)
Binpacking pressure (RSS per replica)
Persistent connection cost (WebSockets: CPU per 10k conns, fan-out capacity)

That’s the shape of this post.

The benchmark harness I’d trust (and what “fair” means)

Most runtime shootouts cheat accidentally.

They benchmark “hello world” with no reverse proxy, no TLS, no connection limits, no warmup phase. Then they act shocked when the results don’t map to production. I don’t care if a runtime wins a toy test by 12%. I care if it lets me run fewer nodes or stop flirting with OOMKills.

Here’s the harness design that stays defensible.

Workloads (three, because production has three kinds of pain)

Cold start (containers/serverless style): start process, bind port, serve first request.
Steady-state memory: idle RSS and RSS under load.
WebSockets:
- Connection churn: connect/disconnect loops (LB reconnections, mobile clients)
- Long-lived connections: 5–60 minutes stable
- Echo vs broadcast fan-out: “send back to caller” is not what chat systems do

Concrete defaults I’d use:

HTTP JSON endpoint: 1 KB JSON request, 2 KB JSON response, basic validation.
WebSocket message size: 256 B, 1 KB, 8 KB (three tiers that match chat, presence, and “oops we shipped JSON over WS”).
Rooms: 100 rooms, 1,000 rooms.
Clients: 1,000 / 10,000 / 50,000 connections (stepwise, because failure modes change).

Fairness controls (the stuff everyone “forgets”)

If you don’t lock these down, you’re not benchmarking runtimes. You’re benchmarking your own sloppiness.

Same machine, same kernel, same ulimits. If one run gets a different ulimit -n, your WebSocket result is a joke.
CPU pinning: keep the server on the same cores (taskset on Linux). Otherwise you’re measuring the scheduler and thermal noise.
Warmup phase: measure p50/p95 after a fixed warmup window. JIT and caches are real.
Same TLS strategy: either terminate TLS at a reverse proxy for both, or do it in-process for both.
Backpressure behavior: if the client can out-send the server, pick a consistent policy (drop, buffer, or slow) and document it.

Node gives you good primitives for this kind of instrumentation via perf_hooks and memory stats via process.memoryUsage() (see the official Node.js contributors). Bun’s runtime and server APIs are built around Bun.serve with integrated WebSocket upgrade handling, as documented by the Bun team.

If you want an external “sanity check” for throughput positioning, TechEmpower is useful. Just don’t pretend it’s your workload. Framework choice, DB access, and JSON behavior dominate. The TechEmpower suite is famous because it’s broad, not because it matches your app.

Here’s a current-year video that shows common benchmark patterns (and some pitfalls) in the wild. Watch it for context, not as a substitute for reproducibility:

[YOUTUBE:_iEaaNIjg7U|Deno vs. Node.js vs. Bun Performance & Comparison (2026)]

A small but important reproducibility detail

Cold start numbers are meaningless unless you say what you counted.

I recommend publishing two cold-start measurements:

Cold start (image already present): excludes image pull time. Measures runtime + init + bind.
Cold start (image pull included): includes pull. Measures your registry/network reality.

In Kubernetes, the first one influences HPA reaction time. The second one matters during cluster recovery, node churn, and “the registry is having a day.”

Results: what typically wins on cold start, memory, and WebSockets

I’m going to be blunt. I don’t trust your absolute numbers unless you ran them on your own infra. But I do trust directional differences when the harness is fair and the workload isn’t contrived.

So here’s the decision table I’d use for 2026, based on how these runtimes are built and what they optimize for.

Metric (real app lens)	Bun (JavaScriptCore + `Bun.serve`)	Node.js (V8 + ecosystem)	What I’d pick
Cold start p50/p95 (container/serverless)	Usually lower startup overhead. Single-binary story helps.	Improved over time, but heavier baseline, more moving parts.	Bun for spiky autoscale services.
Idle memory (RSS)	Often lower RSS for simple services.	Often higher RSS baseline, depends on flags and loaded modules.	Bun if you’re binpacking.
Memory under load (RSS + external)	Can look great until payloads/alloc patterns change. Watch cliffs.	Tooling to analyze memory is excellent (`--inspect`, heap snapshots).	Node if you’re chasing leaks weekly.
WebSocket echo throughput	Strong built-in primitives for upgrade + WS handling.	Depends heavily on library (`ws` vs uWebSockets) and tuning.	Tie. Choose based on your WS library comfort.
WebSocket broadcast fan-out	Can be very good if your server loop is tight and allocations are low.	You can win, but you’ll work for it. Backpressure patterns matter.	Bun for greenfield real-time.
Debugging/profiling maturity	Improving, but fewer battle-tested workflows.	Best-in-class ecosystem and “how to debug this at 3 a.m.” docs.	Node for org-scale services.

Now the practical version.

Cold start: what to measure and what it means

Cold start is p95 user experience wearing an ops hat.

In 2026, scale events happen because:

traffic spikes
deployments roll
a node gets recycled
HPA decides your CPU target is being violated

If your runtime takes 300 ms longer to become ready, you rarely just “lose 300 ms.” You get queuing behind readiness. You get thundering-herd retries. You get that slightly panicky Slack thread where everyone blames the database because that’s the only graph they trust.

A good harness should report at least:

p50 and p95 cold start (in milliseconds)
time to first successful request after bind
time to steady latency (after warmup), because first request is often a liar

And publish your environment:

instance type / CPU model
OS and kernel version
Bun version and Node major

Bun markets itself as reducing startup time and memory via JavaScriptCore and an integrated toolchain. That matches the bet it’s making. Keep the runtime tight, make the default server path fast, ship the thing as a single executable when possible (Bun team). Node is still competitive. It’s just optimized for a different constraint set: stability, compatibility, and an ecosystem that’s basically its own economy.

Memory: RSS vs heapUsed vs external (and why you should care)

Most engineers quote heapUsed because it’s easy to print.

It’s also the wrong number to optimize if your real problem is “why can’t we fit more replicas on this node?”

What matters in containers:

RSS: what the kernel thinks you’re using. This is what triggers OOM and eviction.
heapUsed: what V8 thinks is on-heap.
external: buffers, native allocations, and “stuff not counted in heapUsed.”

Node exposes these quickly with process.memoryUsage() and has genuinely good docs on memory profiling and diagnostics (Node.js contributors).

Bun’s memory story can look fantastic for simple HTTP services. The question I care about is: does it stay well-behaved when the workload is annoying?

Does RSS climb after 10 minutes of WebSocket churn?
Does it flatten after GC cycles?
Does it drop after load stops?

If you run WebSockets at scale, you’re usually holding:

socket buffers
per-connection state
room membership maps
outbound queues

Those allocations are where runtimes show their personality.

WebSockets: echo is easy. broadcast is the benchmark.

Almost every “WebSocket benchmark” online is an echo test.

Echo tests are fine for validating overhead. They’re useless for sizing a chat system, a presence system, or anything where one message fans out to a bunch of sockets.

What you want is a broadcast fan-out test:

N clients connected
clients join rooms
server broadcasts a message to a room
measure messages delivered/sec and p95 delivery latency

Two patterns create very different failure modes:

Connection churn: mobile networks, LB rebalancing, clients that reconnect on deploy.
Long-lived steady connections: dashboards, chat, multiplayer.

Bun’s Bun.serve combines HTTP and WS upgrades in one primitive, which reduces glue code. Glue code is where you accidentally allocate, accidentally buffer, and accidentally never free (Bun team). In Node, you can absolutely build a great WS server. You just have more knobs to get wrong: library choice, upgrade handling, ping/pong strategy, backpressure policy.

If you want a sanity check: uWebSockets.js is often the “fast path” people reach for in Node, while ws is the default ergonomic choice. That gap alone can dwarf the Bun vs Node delta, which is why “runtime X is faster” is such a lazy take.

Deployment implications: turning benchmarks into dollars and incidents

This is where I stop caring about benchmark culture and start caring about consequences.

Benchmarks are only useful if you can turn them into:

pods per node (binpacking)
autoscale reaction time (cold start)
tail latency under connection fan-out

Container density math (the boring answer that saves money)

Let’s say you run a 16 GiB node and reserve 2 GiB for system overhead. You have 14 GiB for workloads.

If your runtime idles at:

200 MiB RSS per replica, you can fit ~70 replicas.
350 MiB RSS per replica, you can fit ~40 replicas.

That’s a 1.75× difference in density.

Then reality shows up. Sidecars. Agents. “Temporary” debug flags that never get removed. Suddenly your cluster upgrade budget is a negotiation.

Autoscaling and cold start: p95 is the tax you pay during deploys

If your p95 cold start is even 500 ms worse, you don’t just lose 500 ms.

You get:

longer readiness
bigger queues
more time in overload
more retries

That’s how you get cascading latency incidents that present as “the database is slow” but started as “pods took too long to come up.”

If you care about this, you’ll probably also care about system-level observability. I’ve written about how to get high-signal data without drowning in telemetry in 7-Step Plan: eBPF Observability Without Sidecars on Kubernetes and how to avoid p99 cliffs in Transparent Huge Pages + Postgres: Stop P99 Latency Cliffs [2026].

WebSocket deployments: load balancers, sticky routing, and failure domains

The operational trap with WebSockets is that the bottleneck moves.

You’ll run into:

LB connection limits (count your max connections per target)
sticky sessions or consistent hashing (rooms need locality)
broadcast topology (single node vs pub/sub)

If you’re doing fan-out, you’re doing distributed systems. Pretending otherwise is how you end up debugging ghost disconnects at 2 a.m.

If you’re running this on Kubernetes, the boring rules still apply:

set realistic readinessProbe and startupProbe
tune net.core.somaxconn
raise ulimit -n
measure connection churn explicitly

When I’d choose Bun vs when I’d stick with Node

Here’s my stance for 2026.

Bun is a safe win when…

You’re building a greenfield HTTP + WebSocket service.
You care about cold start and memory footprint more than obscure ecosystem dependencies.
You want a simpler runtime story (single toolchain, fewer moving parts).
Your WebSocket workload is heavy on concurrent connections and you can keep the app logic lean.

Stick with Node when…

You rely on a mature ecosystem: frameworks, observability agents, native modules.
You need predictable release and support expectations (LTS cadence is a real organizational feature).
Your team’s debugging muscle memory is Node-shaped, and you don’t want to pay the transition tax.
You ship anything involving native addons or complex deployment constraints.

This is one of those cases where the boring answer is actually the right one. Most teams should not migrate a stable Node system just because Bun wins a benchmark. You migrate when the infra math is screaming and you’ve run the harness enough times to trust it.

If you’re already deep in JavaScript/TypeScript architecture decisions, you might also like TypeScript vs JavaScript 2026: Type Safety Finally Worth the Cost? and Hono vs Express in 2026: Which API Framework Actually Wins?.

JavaScriptCore vs V8: why runtime architecture shows up as infra behavior

The engine matters. Just not in the simplistic “JSC fast, V8 slow” way people argue about online.

Bun is powered by JavaScriptCore and is built as an integrated toolkit. Node is powered by V8 and sits at the center of a massive ecosystem.

In practice, the engine difference shows up when:

you care about startup behavior and baseline overhead
your app allocates aggressively (buffers, JSON parsing, WS fan-out)
you need profiling and diagnostics at depth

Node can go very deep on diagnostics and memory profiling. Bun’s bet is that a tighter runtime plus a tight server primitive keeps the hot path simpler and cheaper.

If you want a mental model: engines influence the shape of performance cliffs. Your harness exists to find those cliffs before your users do.

A quick note on microbenchmarks

Microbenchmarks are still useful. I treat them like unit tests for performance regressions, not like decision-makers.

TechEmpower is a good reminder that frameworks and workload mixes dominate outcomes. A runtime that wins “plaintext” can lose “JSON + DB,” and both can lose “WebSockets + fan-out.” Use TechEmpower as context, not as a verdict.

My 2026 prediction: Bun will win more greenfield services, but Node will remain the default

Bun is doing what ambitious runtimes should do. It’s compressing the stack and making “fast by default” real for a class of services.

Node is doing what mature platforms should do. It’s staying dependable while still improving.

My prediction: by the end of 2026, Bun will be a common choice for real-time, connection-heavy services where cold start and RSS are first-order costs. Node will stay the default for everything that leans on the ecosystem and long-term operability.

If you want to pick intelligently, stop arguing about ideology. Build a harness that matches your workload, publish the numbers, and let your infrastructure tell you the truth.

Internal reading trail if you’re in “production mindset” mode:

And if you’re building systems that include AI agents in the backend, you’ll care even more about tail latency and connection stability. Start here: AI agents.

Also, if your team is trying to ship fast with Claude Code or other agentic coding tools, treat runtime changes as part of your risk surface, not just your performance plan: Claude Code.

Finally, don’t forget that “fast runtime” doesn’t save you from app-layer attacks. If you run WebSockets with user-generated content, you’re eventually going to deal with input that tries to control the system. The same mindset behind prompt injection applies to any system that executes actions based on untrusted text: prompt injection.

Originally published on kunalganglani.com

How to Pick LLM Application Observability Metrics [2026]

Kunal — Sat, 15 Aug 2026 12:41:58 +0000

Last year I watched a team burn three days on an “LLM incident” that had nothing to do with the model. p99 doubled, costs spiked, quality dipped. The only thing they had was token counts and a big pile of prompt/response logs. So the debugging session turned into archaeology.

You’ll finish this in about 45 minutes with a working “minimum viable” LLM observability setup: a trace model (LLM + retrieval + tool calls + retries), a short metrics list, and a privacy-safe logging policy you can ship without storing raw prompts.

If you’re only collecting token counts and prompt/response logs, you don’t have LLM application observability metrics. You have an expensive transcript.

This post is about the signals that explain the stuff that actually breaks in production: tool calls that time out, retrieval that quietly returns garbage, caches that “help” by serving wrong answers faster, and safety filters that suddenly start refusing half your traffic.

What is LLM application observability?

LLM application observability is instrumenting an LLM-powered system with traces, metrics, and logs so you can explain latency, cost, failures, and answer-quality changes across the full request path. That means the model call, retrieval, tool/function calls, retries, caching, and safety.

The important part is “full request path”. The model is rarely the product.

I learned this the hard way building the Walmart conversational commerce chatbot at Firework (Zealsight). We handled millions of queries daily with sub-second responses, and the real wins came from tracing the pipeline (retrieval + streaming + orchestration), not staring at token logs. At that scale, “the LLM was slow” is basically never a root cause. It’s a symptom you slap on anything you can’t see.

The minimum signals checklist (copy this into your dashboard)

This is the smallest set of signals I’d accept before calling an LLM feature “production-ready”. Keep it tight. You can always add the fancy stuff later. What you can’t do is debug p99 with vibes.

Signal	Why it matters	How to capture	Suggested metric names
End-to-end request latency (p50/p95/p99)	Users feel p95. Incidents live in p99.	Server span around the whole request.	`llm_app.request.latency_ms`
LLM call latency + TTFT	Separates model slowness from your pipeline slowness.	Span per model call; record TTFT as event/exemplar.	`llm.call.latency_ms`, `llm.call.ttft_ms`
Tool/function call success + latency	Tools fail more than models. Also where most hidden retries happen.	Span per tool call with status + attempts.	`tool.call.success_rate`, `tool.call.latency_ms`, `tool.call.retries_total`
Schema/validation mismatch rate	“It returned JSON” is not correctness.	Validate tool args and structured outputs.	`tool.call.schema_mismatch_rate`
Retrieval latency breakdown	RAG regressions often start as latency regressions.	Separate spans: embed → search → rerank.	`rag.embed.latency_ms`, `rag.search.latency_ms`, `rag.rerank.latency_ms`
Empty retrieval rate	“No docs found” should be loud, not silent.	Count retrievals with 0 usable chunks.	`rag.empty_result_rate`
Top-k actually used	If you ask for k=20 but use 3, you’re paying for nothing.	Log `k_requested`, `k_returned`, `k_used`.	`rag.k_used` (histogram)
Chunk duplication/overlap	High overlap inflates tokens and reduces diversity.	Hash chunk IDs; compute overlap ratio.	`rag.chunk_overlap_ratio`
Cache hit rate (prompt/embedding/semantic)	Cache changes cost and latency by multiples, and can regress correctness.	Count hits/misses by cache tier + version.	`cache.prompt.hit_rate`, `cache.embedding.hit_rate`, `cache.semantic.hit_rate`
Refusal / safe-completion rate	Safety behavior changes over time and by cohort.	Classify outcome category without storing raw prompt.	`safety.refusal_rate`, `safety.safe_completion_rate`
PII redaction rate	Proves your privacy controls are working.	Redact before export; count redactions.	`privacy.redaction_events_total`
Sampling + retention compliance	Prevents “observability becomes a data leak”.	Enforce TTL + sampled logging.	`telemetry.sample_rate`, `telemetry.retention_days`

A note on numbers: graph p50/p95/p99 for anything latency-related. Keep at least 7 days of metrics at full fidelity. Logs can be sampled way harder (often 0.1%–1% for high-volume apps) if you have traces and exemplars.

How to structure LLM telemetry as traces, metrics, and events (so your APM works)

Most teams accidentally create a new observability universe for LLMs. They ship a vendor dashboard, add a giant JSON log blob, sprinkle some notebooks around. Six months later, nobody trusts any of it, and on-call falls back to… reading prompts.

I’m blunt about this: force LLM telemetry into standard primitives.

Traces answer: “Where did the time go?”
Metrics answer: “Is it getting worse over time, and for whom?”
Logs/events answer: “What exactly happened on this one weird request?”

If you already run OpenTelemetry (OTel), you’re most of the way there. I wrote a deeper implementation guide in OpenTelemetry Instrumentation for AI Agents [2026]: Ship It. I’m not going to duplicate the schema here. What you need from this post is a trace shape you can standardize on and the minimum set of attributes that won’t get you hauled into a privacy review.

The trace shape I use for LLM apps

You want one trace per user request, with spans that map to real pipeline boundaries. Not “AI stuff happened here”. Real boundaries.

http.request (root span)
llm.plan (optional, if you do multi-step / agentic planning)
rag.retrieve (optional, if you do RAG)
- rag.embed_query
- rag.vector_search (or hybrid search)
- rag.rerank
tool.call.<name> (0..N)
llm.generate (the final response)
response.stream (optional; track streaming UX separately)

The rule I won’t compromise on: every external dependency gets its own span.

Payments API. Span.

Postgres lookup. Span.

Vector DB call. Span.

If you don’t do this, your p99 chart is just a mystery novel where every suspect has an alibi.

Treat the LLM like a microservice. Because that’s what it is in your architecture.

Minimum span attributes that don’t leak data

I try to avoid raw prompt logging by default. Instead, I put join keys on spans so I can correlate behavior across versions and cohorts.

request_id (server-generated)
user_cohort (coarse: free, pro, internal)
model.name + model.version (or provider + deployment)
prompt_template_id + prompt_version
rag.corpus_id + rag.index_version
tool.name + tool.version
cache.tier + cache.key_version
safety.outcome (enum)

Those version fields matter more than people think. Half of the “LLM randomness” incidents I see are actually “we changed an index / prompt / tool schema and forgot to correlate it”. Then everyone argues about temperature settings like that’s the problem.

If you want a concrete example of versioning and pipeline identity, my AI Engineering Evals: Regression Gates for Prompts, Tools, RAG [2026] post shows how I tie observability back to eval gates.

How to trace and measure tool/function calls (latency, errors, retries, schema mismatch)

Tool calls are where demos go to die.

Your LLM can be perfect and your app still fails because:

the tool timed out at 2,000 ms
the LLM produced invalid args 3 times and you retried
the upstream API returned 429 and you backoff-looped
the tool “succeeded” but returned an empty payload (a silent failure)

What I measure for each tool

For each tool/function, I want five things:

Success rate: success / attempts
Latency: p50/p95/p99 of the tool span
Retry count distribution (histogram, not a single average)
Error categories (timeouts vs 4xx vs 5xx vs validation)
Schema mismatch rate (LLM produced args that didn’t validate)

If you do only one thing: break retries out as their own spans.

Retries are the sneakiest source of “why did p99 double?” because the user only sees the final response. You quietly did 2–4 extra round trips and congratulated yourself for “recovering”.

How to attribute downstream failures to the right span

I see a lot of traces where everything is marked “OK” except the root span. That’s useless.

A simple rule:

If the tool request failed (timeout / non-2xx), mark the tool span as error.
If the tool returned data but you rejected it (schema/validation), mark the tool span as error.
If the tool succeeded but the LLM refused to use it, do not mark the tool span as error. Mark a decision event on the llm.plan or llm.generate span.

This distinction is how you stop blaming infra for product behavior.

If you’re building AI agents or doing agent orchestration, tool-call observability is your real “agent reliability” layer. The model is the easy part.

Cache metrics that actually matter (and how to detect cache-caused regressions)

Caches are a first-class dependency in LLM apps. Pretending they’re just an implementation detail is how you get blindsided.

You usually have at least two of these. Most teams end up with all three:

Prompt cache (exact match, or templated)
Embedding cache (query embedding reuse)
Semantic cache (approximate match, “close enough”)

If you don’t instrument caches, you’ll misread both cost and quality.

A semantic cache can cut cost by 30%–60% in high-repeat workloads. It can also destroy correctness when the query is similar but not identical. Faster wrong answers are still wrong.

I go deeper on cost levers in Reduce LLM API Costs 60%: 6 Techniques [2026] and per-task accounting in Agent Per-Task Cost Calculation [2026]: Retries, Tools, Caching. Here I’m focusing on what to observe so you can catch regressions before support tickets do.

Minimum cache metrics per tier

cache.<tier>.hit_rate
cache.<tier>.lookup_latency_ms (p95 matters)
cache.<tier>.stale_served_rate
cache.<tier>.bypass_rate (how often your app skips cache)
cache.<tier>.key_version_mismatch_rate

That last one is your regression smoke alarm.

If you change prompt templates or tool schemas and don’t bump cache key versions, your cache becomes a correctness bug factory. You’ll see “great latency, bad answers” and the Slack thread will immediately blame the model.

Detecting cache-caused quality regressions without storing prompts

You can detect most cache regressions with three join keys:

prompt_version
cache.key_version
eval_outcome_bucket (or a proxy)

If you run continuous evals, correlate “bad outcome” with cache hit/miss.

If you don’t have full evals yet, use proxies:

user rephrase rate within 30 seconds
user click-through on citations (for RAG)
“thumbs down” rate
session abandonment after answer

They’re imperfect. They beat guessing.

RAG observability metrics for production (and how to debug answer quality drops)

Retrieval-Augmented Generation (RAG) is where most “LLM apps” become real products. It’s also where people ship the least observability.

RAG failures are usually boring:

retrieval returned nothing
retrieval returned duplicates
retrieval returned irrelevant chunks
reranker got slower and you silently lowered k
index got rebuilt and semantics shifted

On the Walmart chatbot, one lesson kept repeating: retrieval quality dominated answer quality at scale. Model upgrades were incremental. Retrieval regressions were catastrophic.

Minimum RAG observability metrics

Latency by stage: embedding, search, rerank
Empty-result rate (k_returned = 0 or usable_chunks = 0)
Top-k distribution: requested vs returned vs used
Chunk overlap ratio: duplicates / near-duplicates in the final context
Citation coverage: how often your answer cites at least 1 retrieved chunk
Context token budget: tokens spent on retrieved context vs generation

If you want my take on why “just increase the context window” is a trap, see RAG Context Window Limits: Why Bigger Is Not Better [2026].

Debugging playbook: “answer quality dropped”

When someone tells me “quality dropped”, I don’t start by debating model upgrades. I run this sequence:

Check rag.empty_result_rate for a spike. If it goes from 0.5% to 5%, that’s your incident.
Check rag.search.latency_ms p99. If p99 jumped, some callers will time out and fall back.
Compare rag.index_version. If the index rebuilt, correlate the quality change with the version boundary.
Look at rag.chunk_overlap_ratio. If overlap jumped, your chunking pipeline probably changed.
Check rag.k_used. If you silently started using fewer chunks, you probably introduced a budget clamp.

This is exactly why you track index identity in spans. Without it, you can’t correlate, so you can’t fix.

If you’re doing retrieval-augmented generation seriously, you’ll eventually want offline evals and replay harnesses. My Agent Evaluation Harness [2026]: Replay, Rubrics, CI Gates post is the next step.

Refusal rate and safety outcomes (without storing sensitive prompts)

If your refusal rate is “unknown”, you don’t have a production system. You have a liability.

Track these as metrics:

safety.refusal_rate
safety.safe_completion_rate
safety.policy_trigger_rate (by category)

Then correlate them with:

model.version (providers change behavior)
prompt_version (you change behavior)
tool.name (tools can trip policy boundaries)
user cohort

A refusal spike can come from:

an upstream model update
an overly aggressive content filter
a new tool that surfaces sensitive data
a prompt injection chain that tries to jailbreak the system

If you’re not threat-modeling this, start here: AI security. I also keep a checklist in AI security that matches how I think about “safe by default” telemetry.

How to classify outcomes without logging prompts

Use an enum outcome plus a coarse reason code.

Example outcomes:

ok
refused
partial
safe_completion
tool_blocked

Reason codes can be your taxonomy. Keep it stable and versioned. If you change your categories every month, you’re destroying your own trend lines.

The point is to measure behavior without hoarding content.

Privacy-safe LLM logging: what to store, what to hash, sampling, retention

“Just log the prompts” is the GenAI equivalent of “just run it as root”. It works right up until the day it ruins your week.

I’m not saying “never store prompts”. I’m saying: make prompt storage an explicit, reviewed exception.

I wrote a dedicated privacy playbook in Data Privacy in RAG Redaction and Retention [2026 Playbook] and the security angle in Prevent Sensitive Data Leakage in RAG: The 2026 Playbook. Here’s the minimum policy I’d ship.

Default: store structure, not content

Store:

prompt template ID + version
tool names + versions
retrieval corpus ID + index version
token counts and cost estimates
safety outcomes
hashes of prompts/responses (salted)

Don’t store by default:

raw user prompts
raw model outputs
raw retrieved chunks

If you need debugging content, use short-lived secure capture:

gated behind a feature flag
enabled for a single request_id
TTL of 24 hours (or less)
access logged and limited to on-call

Redaction and hashing that’s actually useful

Redact obvious PII before telemetry export.
Hash the canonicalized prompt (lowercased, whitespace-normalized) with a rotating salt.

Canonicalize because otherwise you can’t correlate duplicates.

Rotate salts because otherwise your logs become a tracking vector.

Sampling strategy for high-volume LLM apps

A sane default I’ve used:

100% metrics (cheap)
10% traces for baseline, plus 100% traces on errors
0.1% logs (structured events), plus “burst” logging for a single incident window

If you’re doing millions of requests/day, logging every prompt is not just risky. It’s expensive.

If you need to reason about cost tradeoffs, I keep my own datasets on pricing and performance. Based on the benchmark and cost data I maintain at kunalganglani.com/llm-prices and kunalganglani.com/llm-benchmarks, the gap between “fast cheap model” and “frontier model” can be multiple times in $/request. Observability that can attribute cost to spans is how you stop arguing in Slack and start making decisions.

Implementation notes: don’t create an observability silo

The goal isn’t “an LLM dashboard”. The goal is: when p99 spikes, your on-call uses the same workflows they use for every other service.

A few practical moves:

Emit tool spans as normal client spans. Your APM already knows how to graph dependency latency.
Put LLM metadata on spans as attributes, not log blobs.
Use exemplars to attach “this trace is a p99 outlier” to your histograms.
Keep a strict boundary between telemetry and product analytics. They can share IDs, not raw content.

If you’re running production AI systems, treat observability changes like API changes. Version them. Review them. Test them.

The uncomfortable prediction

Within 12 months, “token logs” will be viewed the way we now view console.log in a backend service. Fine for a demo. Embarrassing in production.

If you’re building LLM apps, pick 10–12 signals, wire them into traces/metrics/events, and ship a privacy policy with the same seriousness you ship auth.

Then do the hard part. When your dashboards light up, don’t blame the model. Fix your pipeline.

Originally published on kunalganglani.com

7-Step Plan: eBPF Observability Without Sidecars on Kubernetes

Kunal — Sat, 15 Aug 2026 00:41:39 +0000

eBPF observability without sidecars on Kubernetes is a node-level approach where a DaemonSet attaches eBPF programs to the Linux kernel to infer network and syscall-level behavior, instead of injecting an Envoy (or similar) proxy into every pod. I care about this right now for one boring reason: clusters keep getting denser, and the “small” per-pod tax of sidecars turns into real money and real incidents. Also, P99 latency is what users feel. Not your averages. Not your dashboards that look green.

Key takeaways

Sidecarless eBPF gives you cheaper, simpler baseline network visibility, but it does not replace in-process tracing for business-level spans and custom attributes.
If you remove sidecars, you must re-validate P99 with a canary plan and explicit rollback criteria, or you will ship latency regressions with false confidence.
Kernel capability is the real platform dependency for eBPF. Managed Kubernetes makes this a procurement and node-image problem, not an “app team” problem.
Ambient and hybrid patterns (gateway-only proxies, partial mesh, node agents) are usually the right end state, not an ideological “no proxies ever.”
Cardinality is a budget, not a surprise. Treat labels like an API and enforce limits before the first rollout.

What you gain and lose vs service mesh sidecars (Envoy)

Let’s stop pretending this is a moral debate. It’s a trade.

A service mesh sidecar (usually Envoy) buys you semantic visibility and control at Layer 7 because it literally terminates and re-initiates traffic. That’s why meshes can do mTLS, retries, timeouts, protocol-aware metrics, and consistent trace context propagation without touching application code. They also give you a very predictable place to emit telemetry because every request gets forced through the same choke point.

eBPF-based, node-level observability is the opposite philosophy. It says: the kernel already sees what actually happened. Observe it once per node and stop stapling a proxy onto every workload.

If you’ve ever had a cluster get wobbly because thousands of pods restarted and each one had to pull, init, and warm a sidecar, you already understand the appeal.

What you typically gain moving from sidecars to eBPF node agents:

Fewer moving pieces per workload. Less time debugging injection webhooks. Less “why is this namespace configured differently?” nonsense.
Lower per-pod resource tax. You stop paying the “every pod has a proxy” overhead in CPU, memory, and scheduling pressure.
Cleaner failure domains. A bad proxy config doesn’t brick an entire deployment.
Faster scale-out. Fewer images to pull. Fewer containers to start. Less churn during HPA events.

What you lose (or need to rebuild somewhere else):

mTLS as a default. The sidecar is the easy enforcement point. Without it, you need another enforcement layer (CNI policy, node-level enforcement, or an ambient data plane).
L7 semantics. Retries, consistent request/response metrics, protocol-aware routing. Proxies are good at this because they’re literally in the path.
Application context. User IDs, order IDs, feature flags, and any “what this request means” metadata does not exist in the kernel.

My stance: if you’re removing sidecars primarily for simplicity and cost, you’re probably right. If you’re removing sidecars because you think eBPF gives you “full service mesh observability,” you’re about to have a bad quarter.

The mental model I use is simple:

eBPF is great at: what happened (flows, connection attempts, latency at TCP/HTTP boundaries when parsable, DNS queries, drops).
Sidecars are great at: what it meant and what we did about it (retries, route decisions, mTLS identity, request attributes).

Which signals are reliable with eBPF, and what still needs in-process instrumentation

The reliable eBPF signals are the ones the kernel can see without guessing.

[YOUTUBE:8WCbGSCyDSo|Hubble - eBPF Based Observability for Kubernetes - Sebastian Wicki, Isovalent]

Network flows (L3/L4): who talked to whom, over which port/protocol, how much data moved.
DNS visibility: queries and responses at the node boundary.
Connection health: SYN retries, resets, timeouts.
Some HTTP visibility: depending on the tool and config, you can often get method, path, status code, latency. But it’s inherently “best effort” because you’re reconstructing application semantics from kernel events.

The quickest way to internalize this is to listen to how the Cilium ecosystem talks about it. In Sebastian Wicki’s CNCF talk on Hubble, the center of gravity is flows and network-level visibility first. That’s the honest core of eBPF observability.

What still requires in-process instrumentation (and always will):

Business spans: “Checkout”, “FraudCheck”, “PlanUpgrade”. The kernel does not know your domain.
Custom attributes: tenant_id, user_id, cart_value, feature flags.
Queue semantics: you can see network and syscalls, but “spent 40 seconds in topic X” is application knowledge.
DB query tags: unless your client library emits spans/metrics, eBPF won’t tell you “this was query class Y from code path Z.”

This is where teams faceplant: they treat “sidecarless” as “instrumentationless.” It’s not. You’re just shifting the baseline from per-pod proxy telemetry to per-node kernel telemetry.

If you want a clean split of responsibilities, the pattern that keeps working is:

eBPF node agent for baseline networking + golden signals.
OpenTelemetry (OTel) in-process for business traces and any attribute you’d ever use in a postmortem.

If you’re already on OTel, connect it to your eBPF signals instead of trying to pick a winner. My post on OpenTelemetry instrumentation is aimed at AI agents, but the discipline is the same for microservices: instrument what you’ll page on.

Kernel and managed-Kubernetes constraints (BTF, CO-RE, cgroup v2)

This is the part people hand-wave in architecture reviews because it feels “platform-y.” Then the rollout hits reality.

Sidecars are portable because they’re just containers. eBPF agents are only portable if the underlying kernel features are there.

One important constraint here: the research tooling I use to pull validated kernel requirements and managed-Kubernetes matrices failed while generating this draft (the web_search tool returned empty results). So I’m not going to invent a “kernel >= X.Y” claim and dress it up as certainty. That’s how bad infra decisions get justified.

Here’s the checklist I actually use to evaluate eBPF viability on a real cluster:

Kernel feature availability: confirm the specific eBPF hooks your agent needs are enabled.
BTF availability: does the node kernel expose BTF data required for modern eBPF workflows?
CO-RE compatibility: can your eBPF programs run across kernel versions without per-version rebuilds?
cgroup v2: increasingly common and it changes enforcement/visibility models.
LSM / security posture: some orgs lock down kernel capabilities hard. Know what your security team will actually allow.

In managed Kubernetes, you don’t “just upgrade the kernel.” You upgrade node pools, AMIs, or managed OS images. That means your migration plan has to include infra owners from day one.

If your org treats node images as snowflakes, fix that first. Otherwise you end up with a split-brain cluster where half the nodes are observable and the other half are dark.

Deployment patterns: DaemonSets, gateway-only proxies, and ambient mesh

“Sidecarless” isn’t one architecture. It’s a spectrum.

Here are the four patterns I see work in practice, from least disruptive to most.

1) Observe-only eBPF DaemonSet (no mesh change)

This is the first move I’d make for almost any Kubernetes team. Run an eBPF agent as a DaemonSet and don’t touch the datapath.

Goal: baseline your service graph and latency without changing traffic.
Risk: low. The agent can still burn CPU if you’re sloppy, but you’re not intercepting requests.

2) Partial mesh: keep sidecars for “hard requirements”

Keep sidecars only where they’re doing something you can’t replace easily.

Examples:

workloads requiring strict mTLS identity
workloads doing L7 routing / canarying via the mesh
legacy services where you need a proxy to normalize telemetry

Everything else gets node-level observability plus in-process tracing.

3) Gateway-only proxies (edge + choke points)

A lot of teams eventually admit they only needed proxy semantics at the boundary.

Put Envoy/NGINX at ingress and maybe egress.
Use eBPF for east-west visibility.

This is one of those cases where the boring answer is actually the right one.

4) Ambient mesh (sidecarless service mesh)

Ambient mesh is the attempt to keep mesh-level security and policy while dropping per-pod sidecars. If you’re already deep into Istio (or similar), this can be a clean evolutionary path.

I’m not going to embed vendor-specific claims here because, again, sources didn’t load. But the principle matters: split the mesh data plane into node-level components.

Rule of thumb: don’t jump straight to ambient mesh if your real pain is “too many sidecars.” Start with observe-only and gateway-only. You’ll learn faster with less blast radius.

A phased migration plan (observe-only → partial mesh → gateway-only → sidecarless)

This is the plan I’d run with a real team, not a conference talk plan.

Phase 0: Decide what you’re actually trying to fix

Write down the pain in numbers, not vibes.

“We have N sidecars per node and they consume X% of allocatable memory.”
“Our P99 at ingress is Y ms, but P99 between services is Z ms and we can’t explain the gap.”
“Deployments are slowed by sidecar init and image pull.”

If you can’t quantify the pain, you can’t call the migration a win.

Phase 1: Add eBPF observe-only, build your service graph

You want two outputs:

a service dependency graph that matches reality
a baseline latency distribution per edge (P50/P95/P99)

This phase usually produces the first uncomfortable truth. A bunch of “critical” calls aren’t critical. A bunch of “internal” calls are on the user path.

Phase 2: Keep sidecars only where they provide a must-have control

Most clusters end up with a minority of workloads that truly need proxy semantics. I usually see people land around 10–30%.

Be ruthless. If a sidecar exists only because “that’s our template,” delete it.

Phase 3: Move policy to the network layer, not the pod

You can’t remove sidecars and then shrug about security.

In practice, that means:

enforce identity and policy with your CNI / network policy model
use gateway proxies for boundary controls
keep OTel in-process for business tracing

Phase 4: Remove sidecars for the rest, with rollback criteria

Rollback criteria is not “things feel weird.” It’s measurable:

P99 regression > X% for Y minutes on key endpoints
increase in 5xx rate > Z
missing trace coverage over a threshold
telemetry cost increase beyond budget

This is where I’ll borrow a lesson from my own work building this site’s publishing pipeline. I’ve learned the hard way that deterministic gates before LLM review catch more than doubling the review model’s size. The same mindset applies to infra migrations. Put numbers in front of humans before humans debate.

If you want a CI/CD angle on this, my post on CI/CD is AI-themed, but the gating logic is the point.

How to validate P99 latency after removing sidecars

Most teams do “remove sidecars, eyeball dashboards, ship it.” That’s how you end up reintroducing sidecars six weeks later, quietly, after an incident.

Sidecars impact latency in two competing ways:

They add hops and queues (serialization, filter chains, connection pools).
They can reduce tail latency by enforcing timeouts/retries consistently, or smoothing out client behavior.

So don’t assume “removing sidecars makes it faster.” Measure it.

Benchmark design that doesn’t lie

Pick 3–5 critical endpoints that dominate user experience.
Capture real traffic shape. If you can’t, replay production logs into a staging environment.
Run at two load points: normal peak (say 60% of known max) and stress (say 90%).
Measure end-to-end at ingress and hop-by-hop between services.

Golden signals to track during the canary

Track at least:

latency (P50/P95/P99)
error rate (4xx/5xx split)
saturation (CPU throttling, run queue, network drops)
retries/timeouts if applicable

Canary approach:

start with 1 node pool (or 5% of traffic)
run for a full business cycle (often 24–72 hours)
expand only when your rollback triggers stayed quiet

If you’re interested in tail latency mechanics more broadly, my Postgres write-up on P99 latency cliffs is a good reminder that “average is fine” is how incidents begin.

Cardinality and telemetry cost: control it like a budget

Cardinality is where “more observability” turns into “why is our bill 3x.” eBPF makes it easier to collect lots of dimensions because the kernel can see a ton of distinct tuples (src/dst, ports, pod IDs, DNS names). That’s useful. It’s also how you accidentally light money on fire.

Treat telemetry like an API:

Define allowed labels for metrics. Anything else gets rejected.
Cap high-cardinality fields. Don’t emit pod_uid, full URL query strings, or user IDs as metric labels.
Sample traces intentionally. Head-based sampling for baseline volume. Tail-based sampling for error-heavy endpoints.
Aggregate at the edge. Per-node aggregation can prevent hot-spotting your backend.

A concrete budget model that works:

pick a top-line monthly telemetry budget (say $X)
allocate 50% to metrics, 30% to logs, 20% to traces (tune for your org)
decide what gets cut first when you exceed budget (usually high-cardinality metrics)

This is also where I’ve become allergic to “we’ll figure it out later.” In the incident log for this blog’s publishing pipeline, one slug rewrite burned 907K impressions of link equity in one shot. One “small” operational decision. Massive compounding cost. Observability cardinality works the same way.

For teams building AI in production, telemetry cost discipline is already table stakes. Kubernetes observability should be held to the same bar.

My take: sidecarless is the baseline, not the finish line

The industry is drifting toward a layered model, because it matches reality:

Node-level eBPF for “truth from the kernel” and cheap, ubiquitous visibility.
In-process OpenTelemetry for business meaning.
Proxies, but only where they’re genuinely doing policy work (ingress, egress, a few sensitive workloads).

The hype pitch is “eBPF replaces meshes.” The real story is “eBPF makes meshes optional.” That’s a huge win if you’re tired of paying per-pod tax.

My prediction: once Kubernetes teams treat sidecars as an exception rather than the default, the new operational bottleneck won’t be proxies. It’ll be governance. Who gets to emit which labels, at what cardinality, with what budget. The teams that treat telemetry like a product surface will move faster. Everyone else will keep rediscovering the same billing surprise, one incident at a time.

Originally published on kunalganglani.com

Stacked PRs on GitHub [2026]: A Copy-Paste Workflow

Kunal — Fri, 14 Aug 2026 12:45:10 +0000

Stacked PRs on GitHub are the fastest way I know to stop shipping-by-mega-PR without turning your team into a process cult. If you’re here for how to do stacked PRs on GitHub, you probably have the same two problems every team hits: review fatigue (nobody wants your 1,800-line diff) and developer blocking (you’re “done” but can’t start the next slice until main merges).

Key takeaways

Stacked pull requests work when every PR is reviewable alone, even if it can’t ship alone.
A simple naming + title convention is 80% of the battle for dependent pull requests GitHub workflows.
Changing a PR’s base branch is powerful, but it can make comments outdated and even drop commits from the timeline.
CI for stacked PRs should be tiered (smoke per PR, full suite once per stack), or you’ll pay N× for the same signal.
Merge the stack bottom-up, and have a restack routine ready for conflicts and base-branch drift.

The Problem with Large Pull Requests

A big PR is a tax you pay in three currencies: reviewer attention, CI time, and your own momentum.

I don’t care how good your team is. Once a diff gets huge, the odds of a serious review drop hard. People start skimming. They defer it. They ask for “quick calls” because reading it properly is too much. Then you end up shipping vibes, not software.

Tomas Reimers (Co-founder/CEO at Graphite) uses a deliberately painful example: a single PR with ~2,000 lines added. The point isn’t that 2,000 lines is always wrong. The point is that it’s predictably less likely to be reviewed well, or reviewed at all. That “I’ll look later” PR becomes the load-bearing wall for your whole feature.

Then you do one of three things, none of them good:

You ping reviewers until you feel annoying (because you are).
You context-switch to something else and lose the thread.
Or you keep piling commits onto the same branch until the diff turns into a horror movie.

Stacked PRs exist because the classic “branch off main only” mental model is mostly habit. Git doesn’t require it. GitHub doesn’t require it. Teams just got trained into it.

GitHub making stacked PRs a first-class concept in docs (create/manage/review/CI/troubleshoot/merge) matters because it stops being tribal knowledge. You can point to a canonical workflow and say, “No, this isn’t some weird personal preference. It’s supported.”

Here’s the contrarian part: the mechanics are easy. What’s hard is hygiene. Without hygiene, stacked PRs just give you five smaller messes instead of one big mess.

(Inline illustration suggestion: “One huge PR vs. 4 stacked PRs” diagram.)

Stacked Pull Requests (and what they actually are)

Stacked pull requests are multiple pull requests where each PR’s base branch is the PR below it, creating a dependency chain like:

main <- PR1 <- PR2 <- PR3

Instead of one branch containing every commit for a feature, you split the feature into layers. Each layer is small enough to review and reason about. The stack still represents one coherent feature, but reviewers can approve incrementally.

What are stacked pull requests?

They’re dependent PRs where the “compare against” branch is not main, but another feature branch. GitHub shows each layer’s diff as “what changed since the base layer”. That’s the whole point.

This is why people who’ve used Gerrit/Phabricator feel at home here. You’re reviewing a sequence of changes, not a monolith.

The simplest branch naming + PR title convention

I’m opinionated here because ambiguity kills stacks. If your naming is sloppy, you force everyone to keep the whole stack in their head. That defeats the point.

Branch naming (copy-paste)

feat/<ticket>-<feature>/01-foundation
feat/<ticket>-<feature>/02-domain
feat/<ticket>-<feature>/03-api
feat/<ticket>-<feature>/04-ui

Example:

feat/123-comments/01-foundation
feat/123-comments/02-storage
feat/123-comments/03-endpoints
feat/123-comments/04-ui

PR title format (copy-paste)

[1/4] Comments: foundation
[2/4] Comments: storage
[3/4] Comments: endpoints
[4/4] Comments: UI

Why numbers? Because humans scan lists. Also, Slack links become readable. If you’ve ever tried to track “Comments PR” vs “Comments PR (follow-up)” vs “Comments PR final-final”, you know the pain.

PR description template (copy-paste)

Stack: # → # → # → #
Depends on: #
Unblocks: #
Scope: What this PR includes (2–4 bullets)
Non-goals: What this PR explicitly does not include
Review notes: “Start with Files changed. Ignore generated files. Suggested order: X then Y.”

Add a label like stacked and optionally stack:<ticket> so it’s filterable.

Internal links you might want while you’re here:

If your team is already fighting review overload, my policy write-up on code-review is the same problem from the other side.
If your CI is already expensive, you’ll recognize the same pattern as eval gating in CI/CD.

GitHub PR mechanics that matter for stacks

This is where GitHub-native stacks differ from the muscle memory you might have from Gerrit or Phabricator.

Draft pull requests

Use Draft PRs aggressively for the top of the stack.

Rule of thumb:

Bottom PR ([1/4]) should be Ready for review as soon as it has tests and a clean story.
Everything above it starts life as a Draft pull request until its base is stable.

Drafts matter because they set expectations. They tell reviewers: “you can start looking, but don’t waste time trying to reason about CI failures that are just upstream churn.” GitHub supports drafts as a first-class state, and it cuts down on the pointless back-and-forth.

Pull request refs and merge branches

GitHub doesn’t just run checks on your head branch. It can create a synthetic merge ref (often described as a “merge branch”) representing “what would happen if we merged this PR into its base right now”. That’s the check you actually care about.

For stacked PRs, this is non-negotiable because the base branch is itself changing.

Your checks should answer:

Does PR3 work when merged into PR2’s branch as it exists today?
Or are we green only on the head branch while the merge result is broken?

If you use GitHub Actions required checks, you’re effectively requiring checks on that merge result.

Differences between commits on compare and pull request pages

GitHub shows commits differently depending on whether you’re looking at a compare view or the PR view. In stacks, this confuses people because commits can “move” after rebases/restacks.

The fix is cultural, not technical: make review about diffs and intent, not commit archaeology.

Practical guidance:

Optimize for Files changed and a tight PR description.
Treat commit order as a convenience, not a contract.

Collaborative development models

Stacked PRs fit best with a “feature branch + PR” model, but they also work in trunk-based development if you treat each layer as a short-lived branch.

The constraint is social, not technical: you’re asking reviewers to approve incremental work. If your team insists that every PR must be independently shippable and perfectly polished, stacks will feel “wrong”. If your team can handle “reviewable and correct, but not deployable alone”, stacks are great.

GitHub’s foundational PR docs are worth reading once (not every week): About pull requests.

(Inline illustration suggestion: “PR tabs that matter: Conversation vs Checks vs Files changed”.)

How to create dependent pull requests on GitHub (UI + CLI)

This section is intentionally “do this now.”

UI workflow (GitHub-native)

Create bottom branch from main: feat/123-comments/01-foundation.
Push and open PR1 targeting main.
Create PR2 branch from PR1 branch, not from main: feat/123-comments/02-storage.
Push and open PR2 targeting PR1 branch as the base.
Repeat for PR3/PR4.

If you do only one thing right: always branch from the previous layer.

If you branch PR3 from main “because it’s easier”, you didn’t build a stack. You built parallel PRs that will fight each other.

Can you stack pull requests with GitHub CLI?

Yes. GitHub CLI (gh) doesn’t magically “manage stacks” for you, but it supports the mechanics just fine.

A minimal flow:

Use git locally to create branches off branches.
Use gh pr create for each branch.
Ensure the --base points at the branch below.

I’m not putting a giant command wall here because teams have different defaults and hooks. But the only two commands you need to remember are:

gh pr create --base <base-branch> --head <head-branch>
gh pr edit --base <new-base-branch>

If you want a repeatable CLI environment for this kind of work, build muscle memory around clean tooling like direnv + mise so your git hooks, linters, and CI parity stay consistent.

How do you change the base branch of a pull request?

GitHub makes this a UI action and a CLI action, but it comes with sharp edges.

GitHub explicitly warns:

“When you change the base branch of your pull request, some commits may be removed from the timeline.”
“Review comments may also become outdated because the line of code that the comment referenced may no longer be part of the changes.”

Source: Changing the base branch of a pull request.

Also: when you open a PR, GitHub pins the base to the commit that branch references at that moment. If the base branch gets new commits later, GitHub does not automatically update the base commit for your PR. That’s why stacks drift and suddenly your “green” PR isn’t really green anymore.

Practical playbook:

Only change base branches when you’re restacking after a merge or conflict.
When you change a base, leave a comment: “Restacked onto <branch>; some threads may be outdated.”
Expect reviewers to lose some context. Plan for it.

Rebase vs merge: what to do with stacked PR branches

Should stacked PR branches be rebased or merged?

My stance: rebase locally, merge on GitHub. Keep the PR diff clean, keep main history sane, and don’t rewrite history other people are depending on.

The canonical warning comes from Scott Chacon and Ben Straub in Pro Git: rebasing rewrites history and is dangerous on branches other people base work on. In stacks, that’s literally the point. Other branches are based on your branch.

Source: Scott Chacon and Ben Straub.

So the real rule is:

If your stack is only you, rebasing to keep diffs tight is fine.
If a teammate is building on top of your stack branch, treat that branch as public. Avoid rebasing it. Use merges or coordinate like adults.

Concrete guidance:

Bottom PR (PR1): avoid force-pushing after review starts unless absolutely necessary.
Top PRs (PR3/PR4): rebase is more tolerable early, because fewer people reviewed and fewer branches depend on them.

What happens to reviews when you rebase a stacked PR?

Threads go stale. Approvals can be dismissed depending on branch protection settings. And even when GitHub keeps the “Approved” state, the human reality is that reviewers feel like they’re re-reviewing the same thing.

The mitigation isn’t “never rebase.” The mitigation is discipline:

Keep each layer small (think 100–300 lines net change, not a law but a useful target).
Rebase early, not late.
When you must rewrite history, do it once, do it cleanly, and tell people.

CI for stacked PRs with GitHub Actions (without paying N×)

How do you keep CI green with stacked PRs?

You need to decide what “green” means at each layer.

If you run the full suite on every PR in a 6-PR stack, you’ve built a CI cost multiplier. It’s not theoretical. It’s your cloud bill and your queue time.

GitHub Actions gives you the primitives: the pull_request event and filters (branches, paths) to decide when workflows run.

Source: Events that trigger workflows – pull_request.

Here’s a strategy that actually holds up in practice.

The tiered CI playbook (copy-paste policy)

Per-PR required checks (fast, deterministic)

Lint + typecheck
Unit tests for changed packages
A “smoke” integration test that runs in <10 minutes

These should be required on every PR in the stack, because they’re your “merge safety belt.”

Per-stack full checks (slow, expensive)

Full integration suite
E2E tests
Performance or load tests

Run these:

On PR1 only (the bottom), or
Nightly on the stack label, or
On merges to main

If you’re doing modern merge discipline, consider also running full checks on merge queue / merge group. Just don’t make every single PR pay for the entire test matrix.

Preventing CI duplication in practice

You have three levers:

Path filters: don’t run backend integration tests when only docs changed.
Concurrency: cancel in-progress runs when a new commit is pushed to the same PR.
Required checks policy: only require the smoke suite on stacked PRs; require full suite on merge-to-main.

This is the same “regression gates” mindset I use in AI eval pipelines: cheap checks on every change, expensive checks on fewer choke points. If that resonates, AI engineering eval gates is the same idea applied to prompts and tooling.

(Inline illustration suggestion: “CI tiers per PR vs per stack” flowchart.)

Review hygiene: how to keep reviewers sane

Stacked PRs fail when reviewers don’t know what to do with them.

And no, “just read them in order” isn’t enough. If you don’t set expectations, people will review PR4 like it’s PR1 and you’ll get philosophical debates in the UI layer. Every time.

Here’s the reviewer contract that works.

How should reviewers review a stack without re-reviewing the same diff multiple times?

Rule 1: Review each PR as a layer, not as a feature.

PR1: “Is the foundation right?”
PR2: “Is the domain/model correct?”
PR3: “Is the API behavior correct?”
PR4: “Is the UI correct?”

If someone comments “Why are we doing this?” in PR4, that’s a process smell. That belongs in PR1.

Rule 2: Don’t bikeshed naming in the middle of the stack.

If you rename a core type in PR3, you just created churn across PR4 and PR5. Either:

Fix it in PR1/PR2, or
Defer it to a follow-up.

Mid-stack churn is how stacks become slower than mega-PRs.

Rule 3: Approve with explicit dependency language.

Use a standard comment:

“Approved assuming PR1 merges as-is.”

This gives the author permission to keep moving without pretending everything is independently shippable.

Rule 4: When a base changes, stop arguing with outdated threads.

If a thread is outdated, resolve it with a single note: “Restacked; this moved to PR2.” Then move on.

If your team is also adopting AI-assisted coding and getting PR floods, you’re going to want a policy anyway. Start here: AI coding team workflow policy.

Failure modes: restacking, conflicts, and merging safely

Stacks aren’t fragile. They’re just honest about dependency.

You’re going to hit conflicts. You’re going to have to restack. That’s normal. The failure mode is pretending you don’t need a playbook and then reinventing one in a panic every sprint.

How do you restack after the bottom PR merges?

When PR1 merges into main, every PR above it is now based on the wrong branch.

The clean GitHub-native approach:

Update your local main.
Update your local PR2 branch by rebasing/merging onto main.
Push PR2.
Change PR2 base branch on GitHub from PR1 branch → main.
Repeat upwards: PR3 base becomes PR2 branch, PR4 base becomes PR3 branch.

This is where GitHub’s warning matters: changing base can make comments outdated and drop commits from the timeline. That’s not a reason to avoid it. It’s a reason to:

Restack quickly after merges.
Avoid restacking daily as a hobby.

How do you handle mid-stack changes that require modifying lower layers?

This is the classic: reviewer of PR4 finds an issue that should be fixed in PR2.

Do not patch it in PR4.

Do this instead:

Make the fix on PR2 branch.
Merge/rebase PR3 and PR4 on top of PR2.
Leave a short comment in PR4: “Fix moved to PR2; restacked.”

Yes, it’s extra work. It’s still less work than shipping a tangled diff where the real behavior change is hidden in the wrong layer.

How do you merge stacked PRs in order safely?

Bottom-up. Always.

Merge PR1 → main.
Restack PR2 base to main (or update it so it targets main).
Merge PR2.
Continue.

If you use GitHub’s merge queue, stacks can work, but you need to be strict about required checks on each PR’s merge result.

Tooling: plain Git/GitHub vs GitHub CLI vs Graphite/Git Town

You don’t need paid tooling to do stacked PRs. You need discipline.

Here’s the decision table I’d use.

Option	Best for	What you gain	What you pay	My take
GitHub UI only	Small teams, low churn	Zero setup	Manual base changes	Fine for stacks of 2–4 PRs
GitHub CLI (`gh`)	Engineers who live in terminal	Faster create/edit, scripting	Still manual restacking logic	Great default for most teams
Graphite / GitKraken	High churn, lots of stacks	Purpose-built stack ops, navigation	Cost + workflow lock-in	Worth it if stacks are daily life
Git Town	Teams that want structured branching	Consistent local branch ops	Learning curve	Solid if you commit to it

If you want a good primer on why stacks exist and the “what if you didn’t have to branch off main?” framing, Tomas Reimers’ talk write-up is still useful context: Tomas Reimers.

Here’s the official talk video (worth 20 minutes):

[YOUTUBE:U8wJoxxmskM|Stacked Pull Requests | GitKon 2022 | Tomas Reimers, Graphite]

The 8-step stacked PR recipe (print this)

This is the part you can paste into your team wiki.

Slice the feature into 3–6 layers (foundation → behavior → API → UI).
Create branches with numbered names (01-…, 02-…).
Open PR1 targeting main. Make it Ready for review.
Open PR2..PRn targeting the branch below. Keep them Draft until stable.
CI tiering: require fast checks on every PR; run full suite once per stack or on merge.
Reviewer rules: review the layer only; approve “assuming base merges”; avoid mid-stack churn.
Merge bottom-up only.
Restack after merges by updating bases (expect some outdated threads; communicate clearly).

If you do this consistently, the surprising outcome isn’t just faster reviews. It’s better design. When you’re forced to name and isolate layers, bad abstractions show up early.

A quick note on metrics

If you want to prove this to your org, track two numbers for a month:

Median PR “time to first review”
Median PR “time to merge”

If stacked PRs don’t improve those, your bottleneck isn’t PR size. It’s staffing or ownership.

Closing: stacked PRs are a culture change disguised as branching

GitHub making stacked PRs first-class is a hint: this workflow is no longer just a niche for teams that miss Phabricator.

The teams that win in 2026 won’t be the ones with the fanciest tooling. They’ll be the ones who can keep review quality high while shipping in thin slices.

My prediction: within a year, “stack hygiene” will become as normal as “write tests.” Not because it’s trendy. Because it’s the only way to scale review when output keeps going up.

If you try this workflow, be ruthless about one thing: don’t let stacks become an excuse to ship half-thought-through layers. Small PRs are not inherently good. Reviewable PRs are good.

Originally published on kunalganglani.com

Local LLM Break-Even Math [2026]: Power, Idle, Depreciation

Kunal — Fri, 14 Aug 2026 00:41:49 +0000

Local LLM break even cost model power depreciation is the phrase I wish existed back when I first started seeing people justify $2–5k hardware purchases with “GPU price ÷ tokens.” That math is comforting. It’s also usually wrong.

The uncomfortable truth is that local LLM economics are dominated by what your GPU does when it’s not generating tokens. Idle power. Low utilization. Depreciation. Random failures. The fact that you could have left the money in something boring and liquid instead.

If you want a real answer to “Should I buy a GPU or just pay an API?”, you need a total cost of ownership model. One you can copy into a spreadsheet and tweak with your own kWh price, duty cycle, and resale assumptions.

Key takeaways

A local LLM cost model is mostly a utilization problem, not a “tokens per second” problem.
Idle power can be a first-order cost if your rig sits on for 16–24 hours/day.
Depreciation and resale value matter more than people admit, especially if you upgrade every 12–24 months.
Break-even vs cloud depends on your workload shape (bursty vs steady), not your peak throughput.
Opportunity cost is the adult version of “but I already bought the GPU.”

What this post is (and isn’t)

This is not another simplistic calculator.

It’s a parametric framework for a local LLM total cost of ownership model that includes:

Fixed costs (hardware, depreciation, capital)
Variable costs (electricity, maintenance)
Real-world penalties (idle time, downtime, failure risk)
A clean way to compare against cloud/API pricing

I’ll also give you a ready-to-copy spreadsheet structure and three worked scenarios (hobbyist, indie, small team).

I’m going to keep the numbers illustrative because this is meant to be 2026-ready. You swap in current pricing and rerun. Don’t hardcode 2024-era prices and pretend it’s “analysis.”

For related baseline break-even thinking, I already have a simpler calculator-style post at Local LLM Cost vs Cloud API: 2026 Break-Even Math [Calculator]. This post is the grown-up version.

The naive model everyone uses (and why it fails)

The naive model is basically:

Buy GPU for $X
GPU generates Y tokens/sec
Cloud costs $Z per 1M tokens
Therefore break-even occurs after N tokens

The failure modes are obvious once you say them out loud:

It assumes 100% utilization. Your GPU is not generating tokens 24/7 unless you’re actually running a service.
It ignores idle power. Many rigs sit powered on all day because it’s your workstation, not a headless inference box.
It ignores depreciation/resale. GPUs are not servers you keep for 5 years. In local LLM land, people flip hardware quickly.
It ignores failure risk. A dead GPU fan or PSU turns your “cheap inference” into downtime and emergency spend.
It ignores opportunity cost. Capital isn’t free, even if you paid cash.

If you’re building local AI seriously, you need to think like FinOps, not like a Reddit comment.

The TCO model: separate fixed vs variable costs

Here’s the structure I use.

Step 1: Define your time horizon

Pick a horizon that matches reality:

12 months if you upgrade often (common for local LLM hobbyists)
24–36 months if you’re a small team buying “workhorse” machines

Let’s call it H months.

Step 2: Fixed costs (hardware + capital)

Fixed costs are things you pay regardless of utilization.

Hardware capital

GPU cost
CPU/mobo/RAM/SSD cost (if this is a dedicated box)
Networking/UPS (often ignored, sometimes critical)

Call total upfront cost CapEx.

Depreciation (with resale)

This is the key: the cost is not what you paid. It’s:

DepreciationCost = CapEx - ResaleValue

Model ResaleValue as a percentage of CapEx after H months.

Best case: 70% resale after 12 months (hot market)
Typical: 50%
Worst: 20% (new generation drops, or your card is beat up)

Opportunity cost

If you want to be honest, add a cost of capital. Even 5% annual matters on a $4,000 build.

OpportunityCost ≈ CapEx × AnnualRate × (H/12)

This is the “I could have done literally anything else with that money” line item.

Step 3: Variable costs (electricity + maintenance)

Electricity is two numbers: idle draw and load draw.

Define:

P_idle watts (system on, not generating)
P_load watts (system generating tokens)
kWh_price ($/kWh)

Then:

EnergyCost = kWh_price × (P_idle×IdleHours + P_load×LoadHours) / 1000

This is where utilization finally shows up.

Maintenance / incidentals

Fans
Thermal paste
A replacement SSD
That one power supply you regret buying

Call it MaintCost over the horizon. For a single rig, even $100–$300/year is realistic.

Step 4: Downtime and failure risk

For hobbyist rigs, downtime is “annoying.” For a small team, downtime is “you’re blocked.”

A simple expected-value model works:

FailureRiskCost = FailureProbability × ReplacementCost

You can add a downtime penalty if this is production:

DowntimeCost = DowntimeHours × CostPerHour

If you’re doing AI in production style workloads, you should price downtime honestly.

Step 5: Total local cost over horizon

Put it together:

LocalTCO = DepreciationCost + OpportunityCost + EnergyCost + MaintCost + FailureRiskCost

Now you need an output metric.

How do you calculate local LLM cost per 1M tokens when GPU sits idle most of the day?

You calculate it by dividing TCO by the tokens you actually generate, not your theoretical maximum.

Define:

Throughput = tokens/sec when generating
DutyCycle = fraction of time you’re generating tokens (0 to 1)
HoursOnPerDay = how long the machine is powered on

Then:

LoadHours = HoursOnPerDay × 30 × H × DutyCycle

IdleHours = HoursOnPerDay × 30 × H × (1 - DutyCycle)

TotalTokens = Throughput × LoadHours × 3600

Finally:

LocalCostPer1M = LocalTCO / (TotalTokens / 1,000,000)

This is the whole point: if DutyCycle is 5%, your GPU is basically a space heater with a side hustle.

If you want to sanity-check throughput assumptions, I publish reproducible local inference runs at LLM benchmarks. One thing I’ve learned maintaining that database is that quantization quality cliffs are model-family-specific. A blanket “just run Q4” recommendation is wrong, and it will change your real throughput and thus your cost per token.

Utilization is the break-even lever (not peak performance)

Here’s a simple intuition table. Assume:

Horizon: 24 months
Machine on: 16 hours/day
Idle draw: 80W
Load draw: 450W
Electricity: $0.15/kWh
CapEx: $3,500
Resale: 50% after 24 months
Opportunity cost: 5% annual
Maintenance + failure EV: $400 over horizon
Throughput: 150 tok/s average

You can swap every number. The point is the shape.

Duty cycle (GPU generating)	Tokens over 24 mo (approx)	Energy cost (approx)	Local cost per 1M tokens (approx)
5%	31B	$300	$80–$120
20%	126B	$700	$20–$35
50%	315B	$1,300	$8–$15
80%	504B	$1,900	$5–$10

Why the huge swing?

Depreciation is mostly fixed.
Opportunity cost is fixed.
Idle energy is fixed-ish.

So your denominator (tokens) is what moves.

This is also why “my GPU does 2x tok/s” is often less important than “my GPU is actually used.”

What utilization rate makes buying a GPU cheaper than renting cloud GPUs?

Break-even is when:

LocalCostPer1M ≤ CloudCostPer1M

Cloud cost depends on whether you’re comparing to:

LLM APIs (pay per token)
Cloud GPUs (pay per hour)

APIs are easier to compare. Cloud GPUs require you to build your own serving stack.

API comparison (token pricing)

You plug in your current provider’s $/1M input and output tokens. For a concrete anchor, Anthropic publicly listed Claude 3.5 Sonnet at $3 per million input tokens and $15 per million output tokens in their announcement (Anthropic).

But your blended $/1M depends on your ratio of input:output. A coding agent might generate a lot of output. A RAG Q&A flow might be input-heavy.

If you want help thinking about token shapes and retries, see AI Agent Cost Per Task [2026]: Token Budgets & Break-Even Math and Agent Per-Task Cost Calculation [2026]: Retries, Tools, Caching. In production, retries dominate bills. I’ve seen that first-hand building the Firework short-video generation platform where the AI feature’s bill was dominated by retries and regeneration, not first-pass tokens.

Cloud GPU comparison (hourly pricing)

Cloud GPUs can win if your workload is bursty. If you only need heavy inference 2 hours/day, renting is rational.

For instance-type discovery and current on-demand rates, AWS’s instance family pages are the right canonical source, even if prices shift by region. Start with AWS EC2 G5 instances.

Your cloud TCO will include:

Instance-hours × $/hour
Storage
Egress (often ignored)
Ops time (absolutely ignored, but real)

If you’re a small team, ops time is the silent killer. Every hour you spend babysitting drivers is an hour you’re not shipping.

How should you account for depreciation and resale value of GPUs used for LLM inference?

You should account for it explicitly, and you should do sensitivity analysis.

I use three scenarios:

Best: You resell quickly, market stays hot. 60–70% resale.
Typical: You keep it 2 years, sell when the next gen is out. 40–55% resale.
Worst: You keep it 3 years or the card is effectively unsellable. 10–25% resale.

Two practical notes:

Depreciation is not linear. GPUs often cliff when a new generation drops.
Local LLM usage can hurt resale. If you ran it hot 24/7, buyers can tell.

If you’re building a spreadsheet, make Resale% a single input cell so you can slide it from 20% to 70% and watch your break-even move.

How to model downtime, failures, and replacement costs for hobbyist rigs?

Don’t overcomplicate it. Use expected value.

For a hobbyist rig:

Failure probability per year: 5–10% for “something annoying” (fan, PSU, SSD)
Replacement cost: whatever you’d actually pay, not MSRP fantasies

So:

FailureRiskCost = p_fail × cost_replace × years

For a small team, add a downtime penalty:

If a broken rig blocks 2 engineers for half a day, that’s 8 engineer-hours.

This is where local “cheap tokens” gets exposed. Tokens aren’t the expensive part. People are.

If you’re serious about reliability, treat your local inference service like production AI. You need observability, limits, and safe deployment patterns. (And yes, this is why I write so much about that side of the house.)

Opportunity cost: the line item everyone pretends doesn’t exist

Opportunity cost is the difference between:

Buying a $4k box today
Keeping that $4k liquid

You can model it as:

A conservative annual rate (4–6%)
Or your company’s cost of capital

Even at 5% annual, $4,000 over 2 years is about $400. That’s not nothing. It’s also not the biggest cost. But it’s the difference between “local wins” and “local barely ties” in a lot of real spreadsheets.

If you hate the idea of opportunity cost, fine. Put the rate at 0%. Just don’t pretend you did math.

A spreadsheet template you can copy in 10 minutes

Create these input cells:

Horizon months H
CapEx
Resale %
Annual opportunity rate
kWh price
Hours on per day
P_idle (W)
P_load (W)
Duty cycle
Throughput (tok/s)
Maintenance cost (over horizon)
Failure probability/year
Replacement cost

Then compute:

DepreciationCost
OpportunityCost
LoadHours, IdleHours
EnergyCost
FailureRiskCost
LocalTCO
TotalTokens
LocalCostPer1M

That’s it. No magic.

If you want a simpler version with fewer knobs, start with LLM cost style API optimization first. If you can cut your API bill by 60% with caching and routing, that changes the whole break-even picture.

Worked scenarios: hobbyist, indie, small team

These are not “the numbers.” They’re examples to show how the model behaves.

Scenario A: Hobbyist workstation (low utilization)

Machine on: 10 hours/day
Duty cycle: 5–10%
Throughput: 80–150 tok/s
Horizon: 12 months

Outcome: local cost per 1M tokens often looks worse than mid-tier APIs, unless you have heavy daily usage.

This is where most local rigs sit. People buy hardware for privacy and fun. That’s valid. Just don’t call it “cheaper.”

Scenario B: Indie builder shipping an agent (moderate utilization)

Machine on: 16 hours/day
Duty cycle: 20–40%
Horizon: 24 months

Outcome: break-even becomes plausible. Your tokens amortize depreciation fast enough that local can beat “premium” APIs, especially if you’re generating lots of output.

If you’re building AI agents that run continuously, you’re closer to a service profile. Utilization climbs.

Scenario C: Small team with a shared inference box (high utilization)

Machine on: 24/7
Duty cycle: 50–80%
Horizon: 24–36 months

Outcome: local inference can get very cheap per 1M tokens. At this point, your main risks are reliability and ops overhead, not raw cost.

This is also where you start caring about serving stacks (vLLM, batching, queueing) and governance. A shared box is a shared blast radius.

For serving tradeoffs, see vLLM vs Ollama 2026: Production Power or Developer Ease?.

Local vs cloud isn’t just cost: the hidden “soft” variables

A purely numeric model still misses two real-world things:

Privacy and data control. If you can’t send data to a third party, cost doesn’t matter. Local wins by constraint.
Latency and UX. Local can be faster for tight loops. Cloud can be more consistent at scale.

If you want to reason about latency properly, don’t use averages. Use streaming metrics. I wrote up how I think about that in LLM Latency Benchmark Methodology: Streaming UX Metrics [2026].

And if you’re doing retrieval work, remember this: model choice is rarely the dominant factor. When I built the Walmart conversational commerce chatbot at Firework, retrieval quality dominated answer quality at scale, and we were handling millions of queries daily with sub-second responses. That kind of system has a very different cost profile than a single local box.

Conclusion: the break-even question you should actually ask

Most people ask: “When does my GPU pay for itself?”

The better question is: “What workload do I need to justify keeping this GPU busy?”

If your answer is “I’ll use it sometimes,” you’re buying it because you want it. Own that. It’s a hobby. That’s fine.

If your answer is “This will run daily, for real users, with real uptime expectations,” then do the spreadsheet, include depreciation and idle power, and make the call like an adult.

My prediction for 2026: local LLM adoption will keep climbing, but the winners won’t be the people with the biggest GPUs. They’ll be the people who can keep them utilized, reliable, and boring. That’s where the actual cost advantage lives.

Originally published on kunalganglani.com

OpenAI Codex Desktop Linux Install Guide [2026]: Sandbox + Data Egress

Kunal — Thu, 13 Aug 2026 12:44:59 +0000

OpenAI Codex Desktop Linux Install Guide [2026]: Sandbox + Data Egress

Codex Desktop for Linux is the newest “AI coding agent on your machine” entry to hit the mainstream developer feed, and the search query I keep seeing is basically: openai codex desktop linux install guide. Most posts stop at “it exists.” That’s useless. On Linux, the only parts that matter are packaging trust, sandboxing, and what leaves your network.

Key takeaways

Codex Desktop is a desktop agent running under your user account. Treat it like a powerful tool runner, not a cute chat UI.
On Linux, the safest default is boring: verify the download, sandbox it, and keep filesystem access scoped to a single workspace directory.
Assume anything you paste, any file you grant access to, and any command output you share can be uploaded. Don’t debate it. Measure it.
Put a proxy in front of it if you care about enterprise controls. You want one choke point for egress and audit.
If you can’t answer “what did it execute, what files did it read, and where did it connect,” you’re not operating it safely.

(Inline illustration suggestion: a diagram showing “Codex Desktop → sandbox → workspace dir only → proxy → OpenAI/OpenRouter” with logs captured at each step.)

Quickstart (the safe version)

Linux desktop distribution still isn’t “solved.” Pretending otherwise is how people end up running unsigned binaries out of ~/Downloads with their entire home directory mounted. Your job here is simple: reduce blast radius.

Install in 7 steps (works regardless of distro/package)

Download from an official source (vendor site or official GitHub release). Avoid random repackaged builds.
Verify integrity: if checksums/signatures are provided, verify them before you run anything.
Create a dedicated workspace directory (example: ~/codex-work/) and keep it boring.
Create a dedicated OS user if you’re serious (example: codex) so the agent doesn’t inherit your whole home directory.
Start with a sandbox: Flatpak permissions, Firejail, or a VM. Pick the lightest thing that gives you control.
Start with network visibility: run with a proxy or at least capture traffic during first run.
Only then sign in and connect it to a repo with non-sensitive code first.

The HN thread about “ChatGPT Desktop (Codex Desktop) for Linux” immediately devolved into packaging + permissions anxiety. Good. That’s the right instinct. Here’s the thread: HN item id=49281916.

What is Codex Desktop (and why Linux changes the risk)

Codex Desktop is a local desktop application that can chat about code, read files you allow, and run developer tools/commands on your machine. That combination is the whole story. It’s not “just generating text.” It’s interacting with your filesystem and toolchain.

Linux changes the risk profile for two reasons:

Distribution is fragmented. You might get an AppImage, a .deb, an .rpm, Flatpak, Snap, or the cursed “curl pipe bash” approach (don’t). Each format has different update semantics and a different sandbox story.
Linux gives you real knobs. If you’re willing to do a bit of work, you can lock an app down hard. Most people just… don’t.

Also, the ecosystem around “Codex” isn’t tiny. The official openai/codex repo shows six figures of GitHub stars and five figures of forks (high drift, but still a real adoption signal). Source: openai/codex.

Installing and running Codex CLI (official baseline)

Even if you’re here for the Desktop app, you should know what OpenAI ships “in the open.”

The openai/codex repository is OpenAI-maintained and includes a Quickstart plus install/run instructions for a terminal-based Codex CLI. See: openai/codex.

Why do I care? Because the CLI repo is where you can sanity-check the security policy and how OpenAI frames safe operation.

Security policy (read it, don’t skim)

OpenAI’s openai/codex repo includes a SECURITY.md with explicit guidance and a pointer to safe operation boundaries. The raw file is here: OpenAI SECURITY.md.

One line Linux operators should pay attention to: it points to “Agent approvals & security” documentation and explicitly calls out sandboxing, approvals, and network controls as the intended boundary controls.

Using Codex with your ChatGPT plan

The CLI docs include a section titled “Using Codex with your ChatGPT plan” (in the repo navigation/headings). Entitlements change. Plans change. Blog posts rot. I’m not publishing a brittle matrix that’s wrong in 60 days.

My stance is operational: auth is an event. If you authenticate the desktop agent with your main account, you just tied a local tool runner to your primary identity. Use the minimum privileges that still lets you do your job.

(Inline illustration suggestion: a “two identities” diagram: personal account vs work account, each with separate SSH keys + separate proxy policy.)

How do I install OpenAI Codex Desktop on Linux? (package formats + verification)

I can’t assume which packaging OpenAI is shipping the day you read this. The HN thread has people mad about AppImage, and other people mad about “system integration daemons.” Welcome to Linux.

So here’s the install guidance that stays true even if the packaging changes.

1) Prefer a sandboxed distribution format when possible

If there’s a Flatpak build, it’s usually the best default for desktop agents because you get:

Portal-mediated filesystem access
Easy permission inspection
Straightforward revocation

If you must use AppImage or a native package, add your own sandboxing layer (Firejail/AppArmor/VM). Otherwise you’re trusting whatever it does in your user context. And yes, that includes reading your dotfiles.

2) Verify what you can actually verify

Verification comes in tiers:

Best: vendor provides a signed release and you verify the signature chain.
Good: vendor provides checksums and you verify the hash matches.
Bare minimum: download over TLS from a first-party domain and keep the artifact.

What not to do: treat “someone posted a mirror on a forum” as a supply chain.

3) Update strategy (don’t let auto-update surprise you)

Auto-update is fine. Silent auto-update is not.

If the app updates itself outside your package manager, you need at least two things:

A way to observe version changes (log it, even if it’s a local text file)
A way to roll back if it breaks your sandbox assumptions

At Rise People, the big lesson from shipping SOC 2-oriented scaffolding tooling was that compliance baked into the workflow beats “we’ll review it later.” Treat desktop agent updates the same way. If you don’t know what version is running, you don’t have a system. You have vibes.

Does Codex Desktop run in a sandbox on Linux? (and how to enforce one)

People casually assume desktop apps are sandboxed now. On Linux, that’s only true if you installed it in a sandboxed container format (Flatpak/Snap) or you built a sandbox yourself.

Here’s how I think about the options:

Flatpak: best balance for most engineers. Use portals + overrides.
Firejail: great for AppImage and random binaries. Fast to apply, decent defaults.
AppArmor/SELinux: powerful, but you need patience and you’ll fight policy.
VM (or a disposable dev VM): if you’re dealing with sensitive repos, this is the boring right answer.

I wrote a deeper variant of the VM approach here: AI agents and specifically AI Agent Sandbox Linux VM [2026]: Safe Tool Use, No K8s.

Flatpak permissions: the portal model you should embrace

If you’re on Flatpak, inspect and minimize:

Filesystem mounts (home, host, specific paths)
Device access
Network access (usually on)

The key control is scope. Your goal is: the agent sees ~/codex-work/ and basically nothing else.

Firejail profile: blunt instrument, still effective

Firejail isn’t magic. But it’s a solid “make it annoying to escape” layer for AppImage-style installs.

Your initial success criteria:

No access to ~/.ssh unless you explicitly allow it
No access to ~/.gnupg
No access to ~/.aws, ~/.kube, etc.
Read/write only within a workspace dir

If you’re writing a team policy, Firejail is the quick win.

What files can Codex Desktop read on my machine?

The honest answer: whatever your OS permissions and the app sandbox allow.

So stop debating what Codex “should” do and define your boundaries:

Workspace-only: the agent can read/write within a single project directory.
Repo-only: it can read the repo but not your home directory.
Full home: don’t do this unless you genuinely accept the risk.

Credential hygiene (SSH, GPG, cloud creds)

If the agent can read ~/.ssh, it can often:

Enumerate keys
Use ssh-agent sockets if accessible
Perform git operations as you

Hardening moves that actually work:

Use separate SSH keys per repo or per trust domain
Use a separate SSH_AUTH_SOCK for the agent environment
Put work keys on a hardware token if you can

If you’re doing “vibe coding” with an agent that can see your production AWS creds in ~/.aws/credentials, you’re not moving fast. You’re gambling.

Does Codex Desktop upload my code to OpenAI servers? (and how to reason about data egress)

If the desktop agent is using a hosted model, some data leaves your machine. The interesting questions are:

exactly what leaves (full files vs diffs vs snippets)
when it leaves (on open vs on request)
whether telemetry/crash logs include code

I prefer a test-driven posture: assume worst, then verify with network capture.

“What data leaves your machine” table (practical threat model)

Data type	When it can be sent	How to limit it	How to verify
Prompts you type	Every request	Don’t paste secrets. Build redaction habits.	Capture HTTPS destinations + request sizes via proxy.
Code snippets/diffs	When you ask it to edit/understand files	Restrict filesystem scope to workspace.	Compare file reads to outbound bursts.
Terminal output	When you share command results	Don’t run commands that print secrets, then hand the output to the agent.	Log tool execution + capture outbound.
Repo metadata (paths, filenames)	When agent indexes context	Limit directory access. Use a clean workspace dir.	Watch for directory listing activity + traffic spikes.
Crash reports/telemetry	On crash or opt-in telemetry	Disable telemetry if possible; block endpoints.	Block domains; verify no egress to telemetry hosts.

This isn’t fear-mongering. It’s basic LLM security hygiene.

How do I restrict Codex Desktop’s filesystem access?

There are only three strategies that reliably work:

Run it as a different Linux user with a minimal home directory.
Use a sandbox (Flatpak portals or Firejail) and only mount the workspace.
Use a VM and treat it like a disposable coding environment.

The “different user” approach is underrated because it’s boring and it works across distros.

A policy I like for internal tools is default-deny, then grant access one directory at a time. At Rise People, our SOC 2 scaffolding CLI succeeded because the defaults nudged engineers into compliant paths instead of relying on PR reviews and good intentions.

How do I set a proxy for Codex Desktop (HTTPS proxy)?

If you want enterprise-style controls (audit, DLP, allowlists), you want a proxy choke point.

Two patterns:

System proxy: set environment variables like HTTPS_PROXY/HTTP_PROXY for the app process (method varies by launch mechanism).
Network-level enforcement: firewall rules or an outbound tool like OpenSnitch.

If you’re routing model traffic through a gateway, OpenRouter is a good mental model for “OpenAI-compatible” APIs. Their docs describe it as “hundreds of AI models through a single API endpoint,” and they show OpenAI-SDK integration as one path. See OpenRouter docs.

I’m intentionally not claiming “all SDKs work by only swapping base URLs.” The tighter claim is: many OpenAI SDKs can be configured to point at a different base_url, and OpenRouter documents OpenAI-SDK usage as one integration path.

How can I observe/confirm what network requests Codex Desktop makes?

You have three practical options on Linux:

Packet capture (tcpdump/Wireshark): good for destinations, timing, volume.
Explicit proxy (mitmproxy or corporate proxy): best for audit and allowlists.
Outbound firewall prompts (OpenSnitch): best for interactive control.

The goal isn’t to decrypt everything. It’s to answer three operator questions:

Which domains/IPs does it talk to?
How often?
How big are requests when it’s “reading” a repo?

If your org is serious about production AI, this is non-negotiable.

(Inline illustration suggestion: screenshot-style checklist “traffic capture on first run” with domains, timestamps, request sizes.)

Where are Codex Desktop logs stored on Linux? (and what you should log yourself)

Exact paths vary by packaging and build, so I’m not going to pretend there’s one canonical list. On Linux, desktop apps commonly write under:

~/.config/…
~/.local/share/…
~/.cache/…

What matters more: keep your own minimal “paper trail,” regardless of where the app logs.

A useful local audit trail for a desktop agent has three components:

Process execution log (what commands/tools did it run)
Filesystem access boundaries (what directories were even mounted/visible)
Network egress record (where did it connect)

Incident writeups are basically endless proof that this matters. Tailscale’s August 12, 2026 post, “How we tracked down a 16-year-old SQLite bug,” is a love letter to forensics and traceability. They describe “months of intense forensics” to isolate a deep bug. Source: Tailscale blog post.

If you can’t reconstruct what happened, you don’t have a secure posture. You have a hope posture.

For a deeper agent logging schema (OpenTelemetry + redaction), see: AI Agent Observability Logging Schema [2026]: OTel + Redaction and OpenTelemetry Instrumentation for AI Agents [2026]: Ship It.

How do I disable telemetry/crash reporting in Codex Desktop?

Treat telemetry control as a three-layer problem:

In-app settings (if offered)
OS-level blocks (hosts/firewall allowlists)
Proxy policy (best for teams)

And even if you “disable telemetry,” validate it with your own traffic capture. Settings UIs lie all the time. Sometimes accidentally. Sometimes because nobody bothered to test the edge cases.

Can I use separate SSH keys with Codex Desktop for Git operations?

Yes. And you should.

Two practical approaches:

Per-repo SSH config: map hostnames or repo paths to specific IdentityFile entries.
Separate agent environment: run Codex Desktop in an environment where SSH_AUTH_SOCK points to a restricted agent (or no agent at all).

The point is simple: the agent shouldn’t inherit your keys to the kingdom. If it needs push access, give it a deploy key scoped to one repo. Not your personal key that can push everywhere.

This ties directly to broader AI agents hygiene and AI Agent Threat Model: 7 Attack Vectors [2026].

Team policy checklist (the minimum I’d ship)

If you’re rolling Codex Desktop out to a team, here’s the baseline policy I’d want written down.

Packaging: only install from approved sources. Keep a record of version and artifact hash.
Sandbox: Flatpak or Firejail required. Workspace-only mounts.
Credentials: separate SSH keys. No shared tokens in env vars. No production AWS creds on dev boxes.
Network: proxy required for company repos. All egress logged.
Observability: log tool executions and keep logs for at least 30 days.
Human-in-the-loop: anything that writes to git needs explicit approval, especially force pushes.

If your agent can push to main, you’re one sloppy approval away from an incident.

Closing: the prediction

Linux is going to become the most controllable environment for agentic desktop tooling. Not because it’s “more secure by default,” but because it gives you knobs that actually work.

My prediction for the next 12 months: the teams that win with Codex Desktop (or any desktop agent) won’t be the ones with the best prompts. They’ll be the ones who can answer, on demand, what ran, what it touched, and what left the machine.

Originally published on kunalganglani.com

AI Agent Sandbox Linux VM [2026]: Safe Tool Use, No K8s

Kunal — Thu, 13 Aug 2026 00:42:35 +0000

AI agent sandbox linux vm setups are the fastest way to turn “LLM tool use” from a cool demo into something you can run without sweating every command. The moment your agent can execute a shell, install packages, or curl the internet, you’ve created a tiny production-incident generator. My stance is simple: one disposable Linux VM per agent run, with default-deny network egress, snapshot rollback, and auditable logs.

Key takeaways

Treat an agent run like running untrusted code. Because functionally, that’s what it is.
Default-deny outbound egress is the only policy that survives prompt injection and supply-chain surprises.
Use a base image plus copy-on-write overlays so every run resets in seconds.
Inject secrets as short-lived, least-privilege credentials. Don’t bake them into images. Don’t leave them on disk.
Persist only what you can defend in a post-incident review: artifacts and audit bundles. Throw the rest away.

Inline illustration suggestion: Diagram showing “Agent Orchestrator” launching a disposable VM, with arrows for allowlisted egress, artifact export, and log bundle export.

What is an AI agent sandbox?

An AI agent sandbox is an isolated execution environment that lets an agent use real tools (shell, Git, package managers, browsers) while limiting blast radius: file access, network access, and credentials. In practice, it’s a box you’re comfortable letting get owned.

When people say “sandbox,” they often mean “a Docker container with vibes.” That’s fine for toy demos. For serious tool use, I want something that actually holds up when the agent goes off-script.

At minimum, I’m looking for:

A hard boundary (VM or VM-like isolation)
A repeatable, disposable filesystem
Strict outbound network controls
A clean secrets story
Forensics: who ran what, changed what, and talked to what

If you’re building AI agents that touch the outside world, sandboxing is not a “later” problem. It’s the entry price.

The 8-step checklist I use as the mental model

This is the minimal loop that keeps you out of trouble:

Create a new VM instance (per run).
Attach a fresh copy-on-write root overlay on top of a read-only base image.
Configure networking on a dedicated interface (tap/bridge).
Apply default-deny egress rules on that interface.
Inject short-lived secrets at boot (scoped to this run).
Run the agent’s tool loop through a single tool-runner entrypoint.
Export artifacts and a log bundle.
Destroy the VM and wipe overlays.

If you do only one thing from this post, do #4. Default-deny egress changes the whole risk profile.

Why letting an agent run tools is uniquely risky

“Untrusted code execution” used to be a special event. With agents, you’ve productized it.

The failure modes in real agentic workflows are not subtle. They’re the obvious stuff we all learned to fear, except now it’s automated and fast.

Prompt injection → tool misuse. The model gets talked into running commands it shouldn’t. If you haven’t internalized this yet, read my prompt injection post and my broader AI security.
Supply-chain installs. The agent pip installs or npm installs something sketchy because it “fixed the build.” If you’re thinking “we pin versions,” congrats. That’s step 0, not the solution.
Credential theft via environment/process. Agents and tools love environment variables. Malware loves them more. /proc visibility and sloppy secret injection are how you lose.
Data exfiltration via outbound HTTP. If the agent can talk to the internet, it can leak. The easiest exfil path is the one you already gave it.
Accidental destruction. It’s not always malicious. rm -rf, recursive edits, or “clean up this directory” on the wrong mount happens.

If your agent has a shell, it’s a junior engineer with root and zero judgment. Sandbox accordingly.

A concrete number to make this feel less hand-wavy: Firecracker’s whole pitch is density and speed because this pattern is meant to run at scale. The project site says it can start user space in as little as 125 ms, create up to 150 microVMs per second per host, and add <5 MiB memory overhead per microVM. That’s the runtime telling you “disposable per-task VMs are not crazy.”

(Those numbers are from the official Firecracker site.)

Isolation runtimes: pick your poison

There’s no perfect isolation. There are only tradeoffs you understand and can operate.

For solo devs and small teams, I keep the shortlist simple:

Linux containers (runc)
gVisor
Kata Containers
Firecracker microVMs
Full-fat VMs (QEMU, VMware, etc.)

And yes, Kubernetes can orchestrate some of this. But you asked for “no Kubernetes required,” and I agree with the premise. K8s is great at scheduling. It does not magically solve default-deny egress, secrets lifecycle, or audit bundles. You still have to do the hard parts.

The Comparison Table

Here’s the table I wish existed back when everyone first told me “just run the agent in Docker”:

Runtime	Isolation boundary	Cold start	Egress control ergonomics	Snapshot/rollback ergonomics	Best fit for agent sandboxes
Containers (runc)	Shared kernel	Very fast	Easy-ish (netns/iptables), but mistakes leak	Layered FS, but state leaks through mounts	Lowest friction, highest foot-guns
gVisor	User-space kernel layer	Fast	Similar to containers, extra guardrails	Similar to containers	Better than runc when you can accept compat gaps
Kata Containers	VM-backed containers	Slower than runc	VM networking patterns	VM disk patterns	When you want “container UX, VM boundary”
Firecracker	MicroVM (KVM)	Fast for VMs (125 ms claim)	Clean per-VM interface	Great with overlays/snapshots	Strong default for server-side “one run, one VM”
Full VM (QEMU)	VM (KVM optional)	Typically slower	Fine, but heavier	Fine, but heavier	When you need maximum compatibility

If your threat model includes “agent might run arbitrary code from the internet,” VM-backed isolation is the boring answer that’s actually right.

Also: don’t over-rotate on “most secure.” The real question is “most secure that you can operate consistently.” A flaky security control is just a future incident with better marketing.

Security: default deny is the only way

Allowlisting outbound traffic feels annoying until you’re the person explaining to your cofounder why a package install beaconed to a random domain.

Default-deny egress works because it doesn’t care why the agent is misbehaving:

prompt injection
malicious dependency
accidental command
model bug

If it can’t phone home, it can’t exfiltrate.

A practical egress model for small teams

You don’t need a service mesh or a policy engine to get 80% of the value.

On a single host running disposable VMs, the pattern is:

Give each VM a dedicated network interface (a tap device, typically attached to a bridge).
Apply firewall rules on that interface (nftables on Linux, pf on macOS, iptables if you’re stuck in the past).
Start with deny all, then add an allowlist.
Log denies. Deny logs are your best “what did the agent try to do?” signal.

A concrete implementation hint: I’ve seen small teams bind VM traffic to a dedicated bridge (e.g., br-agent) and apply egress rules only on that bridge. That way you’re not playing whack-a-mole with the host’s global networking.

What to allowlist (and how not to get tricked)

Most coding agents need less network than people assume. Typical allowlist buckets:

Git hosting: github.com (and your internal Git host)
Package registries: pypi.org, files.pythonhosted.org, registry.npmjs.org, distro mirrors
Container registries: your specific registry domains if you build images
Time + identity: an NTP source, your IdP endpoints if you’re doing OIDC

The trap: DNS is an exfil channel. If you allow arbitrary DNS to arbitrary resolvers, you’re letting the agent encode secrets in queries. Treat DNS as part of egress.

The safer approach for a small setup:

Use a single resolver you control (even a local caching resolver).
Pin or restrict DNS egress to that resolver.
Allowlist by domain + resolved IP ranges, not “anything on 443.”

If you want the short version: allowlist destinations, not ports.

Disposable filesystems: snapshot, run, roll back

Agents are messy. They create files. They install packages. They “just try something.” That’s the whole value prop.

Your filesystem design should assume:

Every run will leave garbage
Some runs will try malware-style persistence
You’ll eventually need to answer “what changed?”

The base image + overlay pattern

The simplest durable design is:

A read-only base image you patch and update deliberately (weekly is fine).
A copy-on-write overlay (per run) that captures all changes.
A scratch/work volume (optional) for larger temp files.

On the VM side, the concept maps cleanly to qcow2 backing files or overlayfs-like semantics depending on your stack. The important part is operational: every run starts from the same base, and the overlay dies with the run.

A number to keep you honest: if you run 20 agent tasks a day and each leaves behind 2 GB of junk, you’re burning 40 GB/day. Disposable overlays make cleanup deterministic.

What should be persisted vs discarded after each agent run?

Persisting the wrong stuff is how sandboxes quietly become “semi-trusted pet environments.” That’s when weird, unreproducible issues show up. Then people blame the model. It was the state.

My rule:

Persist:
- Build artifacts you intentionally export (binaries, patches, generated docs)
- A structured audit bundle (more on that below)
- A minimal “run manifest” (inputs, tool permissions, allowlist config, VM image hash)
Discard:
- The VM disk overlay
- Package caches (pip, npm, apt) unless you can isolate them safely
- Shell history inside the VM (you already have external transcripts)
- Any copied workspace that contains secrets

Caching is the one everyone tries to sneak back in for speed. If you want warm performance without persistent state, use a warm pool of pre-booted VMs with empty overlays, not long-lived disks.

Secrets injection without leaving landmines

If your agent can access production credentials, you’ve built a very expensive secret-leaking machine.

The goal is not “the agent can deploy.” The goal is “the agent can deploy in a narrow, revocable way.”

The least-bad secrets lifecycle

For small teams, this pattern holds up:

Mint a short-lived token per run (minutes, not days).
Scope it to the minimum set of actions (read-only if you can).
Inject it at boot using a user-data style mechanism (cloud-init-like), or a one-shot secrets file mounted in memory.
Redact secrets in logs at the boundary (tool runner) before anything gets shipped.

Concrete examples of “scoped to minimum”:

A Git token that can only read a single repo.
A package registry token that can only download, not publish.
A cloud token that can only write to one bucket prefix for artifacts.

And please stop putting long-lived secrets in environment variables if you can avoid it. Processes leak env. Debug logs leak env. People paste env into tickets.

If you’re doing AI in production work, secrets hygiene is where “prototype” turns into “adult supervision.”

Auditability: make every run reviewable after the fact

A sandbox that can’t be audited is security theater.

Assume you will eventually need to answer these four questions:

What commands did the agent run?
What network destinations did it try to reach?
What files did it change?
What artifacts did it produce?

If you can’t answer those quickly, you don’t have control. You have vibes.

“Wrap tool entrypoints” means one choke point

Instead of letting the agent call bash, git, pip, and curl directly, route everything through a single “tool runner” entrypoint. This is where you:

log argv + working directory
capture stdout/stderr
record exit code + runtime duration
attach a permission context (“read-only repo”, “network allowlist v3”, “no write outside /workspace”)

You can structure logs as OpenTelemetry spans if you want to get fancy. I wrote a full schema for this in AI agents.

What to log (minimum viable forensics)

Per run, I want a bundle that contains:

Run manifest: timestamp, VM image hash, agent version, tool policy version, egress allowlist version
Command transcript: every tool call, args, cwd, exit code
Filesystem diff summary: list of files created/modified/deleted under /workspace
Network flow log: destination IP:port, SNI/hostname if available, bytes sent/received, allow/deny decision
Artifacts: patch files, build outputs, test reports

That’s enough to reconstruct intent without saving the entire VM disk.

A concrete retention guideline that won’t bankrupt you: keep audit bundles for 30 days by default, and keep “suspicious runs” for 180 days. If you don’t have a security team, your future self is the security team.

Getting started: a no-Kubernetes architecture that actually works

Here’s the prescriptive design I’d ship for a solo dev or a small team on a single dev server.

Architecture: one host, one orchestrator, many disposable VMs

Components:

Agent orchestrator (a small service or even a CLI) that:
- creates a VM per run
- attaches overlay disks
- configures networking
- injects secrets
- starts the agent tool loop
- exports artifacts + logs
- destroys the VM
MicroVM runtime: Firecracker if you’re on Linux and want density; otherwise a standard VM stack.
Developer VM wrapper (laptop ergonomics): Lima is a pragmatic choice on macOS/Linux because it launches Linux VMs with automatic file sharing and port forwarding (similar to WSL2).

Firecracker’s own description is clear: it’s purpose-built for “secure, multi-tenant container and function-based services,” implemented as a KVM-based VMM with a minimal device model to reduce attack surface. That’s exactly the shape we want for “agent runs arbitrary tool code.”

Warm pools without Kubernetes

Competitor posts love warm pools implemented with CRDs. You don’t need that.

A warm pool for small teams is:

Keep N pre-booted VMs paused/idle (N is usually 2–10).
Each VM is sitting on the same base image but with an empty overlay.
When a run starts, you assign it a warm VM, attach a fresh overlay, apply policy, and go.

You should also set:

a hard concurrency limit (start with 2 if you’re on a laptop)
CPU/memory caps per VM (e.g., 2 vCPU, 4–8 GB RAM per run)
a wall-clock timeout per run (e.g., 10–20 minutes)

This is less about cost and more about blast radius. Unlimited concurrency is how an agent turns a small bug into a host meltdown.

When you actually should use Kubernetes

If you’re already operating Kubernetes well, it can help with scheduling, packaging, and lifecycle. The industry trend is real. The Kubernetes SIGs project agent-sandbox literally describes itself as enabling management of “isolated, stateful, singleton workloads” for “AI agent runtimes.”

But K8s doesn’t remove the need for:

thoughtful default-deny egress
secrets scoping
auditable tool boundaries
snapshot rollback patterns

If you don’t have those, you just have a compromised agent… scheduled nicely.

Inline illustration suggestion: “Single-host” architecture diagram with: base image store, overlay store, egress firewall, secrets broker, artifact store, log store.

A reality check (because nothing is perfect)

This approach isn’t free. It’s just the best trade I’ve found for the “agents with real tools” era.

Here are the honest limitations:

You’re still trusting the host. VM isolation reduces guest-to-host breakout risk, but it doesn’t eliminate it. Patch your kernel. Use hardware virtualization. Reduce host attack surface.
Egress allowlists are operational work. Registries change IPs. CDNs are annoying. If you allowlist too broadly, you lose the point. If you allowlist too narrowly, your agent can’t do its job.
Audit logs can leak secrets. If you don’t redact at the boundary, you’ll end up storing credentials in logs. That’s worse than not logging.
Performance and UX tradeoffs are real. Starting a VM, attaching disks, applying firewall rules. It’s extra latency. Firecracker’s design exists because people wanted VM boundaries without VM pain, but there’s still overhead.

One more: if your agent needs to interact with a user’s real browser session or OS GUI, a Linux VM sandbox helps, but it doesn’t solve the “human session is the crown jewels” problem. That’s a different architecture.

A pragmatic posture for 2026

My bias is that more teams will ship agents with tool access before they ship proper security controls. The market rewards speed. Incidents punish you later.

Running this blog’s 7-agent publishing pipeline (261+ posts), I’ve learned that deterministic gates catch an entire class of failures that “just use a smarter model” will never reliably catch. Sandboxing is the same kind of boring engineering. It’s not about smarter agents. It’s about guardrails that don’t get confused.

If you want adjacent reading on operationalizing agent systems, start with agent orchestration, AI security, and AI in production.

The point nobody wants to say out loud

Most “agent safety” conversations are still stuck on model behavior. That’s the wrong layer.

Tool-using agents are systems. Systems fail. Systems get attacked. And when they do, the only thing that matters is blast radius.

My prediction: by the time we hit 2027, “agent runs tools on the host” will be viewed the same way we now view “production app runs as root.” It’ll still exist, but it’ll be a red flag.

If you’re building agents today, you have a chance to make disposable Linux VM sandboxes the default. Not because it’s trendy. Because it’s the first design that lets you sleep.

Originally published on kunalganglani.com

Deepfake Voice Detection: 7-Step Detector Eval Guide [2026]

Kunal — Wed, 12 Aug 2026 12:48:49 +0000

Deepfake voice detection is the practice of determining whether an audio clip or live call contains human speech or AI-generated / voice-cloned speech, using a mix of machine learning signals, provenance metadata, and operational controls.

Key takeaways:

Deepfake voice detection is an evaluation problem before it’s a tooling problem. If you can’t describe your threat model and test set, you can’t buy your way out.
Your detector bake-off should include telephony (8 kHz), re-recording, codec damage, noise, and overlapped speech. These break “99% accurate” demos fast.
Report more than accuracy. At minimum, publish AUC, EER, calibration (ECE or Brier), and latency at your target throughput.
Choose thresholds based on fraud cost, not vibes. False positives burn call center capacity. False negatives burn money.
In production, detection is defense-in-depth: combine detectors with out-of-band verification and provenance standards like C2PA.

I’m writing this as a “part 2” to my tools-focused post on deepfake voice detection. That post is doing fine, but Google is pretty clearly asking for something else on the main query: a repeatable, defensible evaluation methodology that a security or fraud team can actually run.

Also, this is not theoretical. Per my own Google Search Console snapshot for this site, the exact query “deepfake voice detection” has been hovering around average position ~11 over the last ~90 days. That is page-2 purgatory. The way out is not “more tools”. It’s a scorecard.

What Is an AI Voice Detector and Why Does It Matter in 2026?

An AI voice detector is a system that analyzes audio and outputs a likelihood score (or label) that the speech was generated or converted by a model rather than spoken by a human.

Why it matters in 2026 is simple: voice is now an interface for money. Call centers, banks, crypto exchanges, and even internal IT helpdesks treat “a human voice on the phone” as a weak-but-useful authentication factor. Voice cloning made that assumption dangerous.

Here’s the stance I’ll defend through the rest of this post:

Detection alone will not “solve” voice fraud. But evaluated detection can move you from blind to measurable. And measurable is how you get budget, reduce loss, and avoid vendor theater.

Two practical consequences you should internalize:

Your accuracy number is meaningless without your audio pipeline. A detector that looks great on clean WAV files can fall apart after one Opus transcode and a noisy office.
Your success metric isn’t “catch deepfakes”. It’s something like “reduce successful impersonation losses by 30% without increasing handle time by 20 seconds.” That forces you to think thresholds, escalation, and fallbacks.

If you’re building fraud defenses, think of this like spam detection circa 2004. The model is part of the system. The system is the product.

(Visual break: architecture diagram of a call flow with scoring + escalation.)

The Rising Threat of AI Voice Fraud

The “AI voice fraud” story is already boring in the worst way. Not because it’s solved. Because it’s becoming routine.

The Federal Bureau of Investigation has a public warning page specifically about criminals using AI for scams, including AI-generated audio for impersonation. Their mitigations aren’t “use a better detector”. They’re operational: callback numbers, code words, and verification steps.

That’s a huge tell.

Law enforcement is implicitly acknowledging what most detector vendors don’t like to say out loud: deepfake voice detection is probabilistic, and fraud is adaptive. So you need layered defenses.

In practice, I see three threat patterns show up repeatedly in incident postmortems across the industry:

Family emergency / executive urgency scripts. The audio doesn’t need to be perfect. It just needs to create panic.
Call center account takeover. Fraudsters target the lowest-friction lane: “reset my password”, “change my payout details”, “update my phone number.”
Internal helpdesk / IT social engineering. If your org has SSO, the helpdesk is a crown-jewel path. A good clone plus one leaked employee detail is a bad day.

If you run a CX org, a bank, or even a SaaS with high-value accounts, the right mindset is:

Assume you will hear AI-generated speech in inbound calls.
Assume attackers will iterate after the first time you block them.

Which brings us to the part most content skips: how detection works enough to evaluate it.

How AI Voice Detection Technology Works

Most “how it works” explanations stop at “spectrograms” and call it a day. That’s not enough for evaluation.

A useful mental model is that detectors typically combine a few families of signals:

Spectral / feature-space artifacts: classic audio features (MFCC-like representations, spectral bands) plus learned features that pick up inconsistencies in how synthetic audio distributes energy.
Neural codec fingerprints: a lot of modern TTS/voice conversion pipelines pass through neural codecs. That can leave subtle, model-family-specific traces.
Prosody and temporal cues: timing, stress, rhythm, breath patterns, turn-taking. Humans are messy. Models are getting better, but they’re still “too consistent” in weird ways.
Model-specific classifiers: vendor detectors that identify audio generated by that vendor (or their model families). These can be strong for narrow provenance detection and weak for general “is this fake?” detection.

Here’s the practical implication for evaluation:

If a detector is primarily trained on a specific generation stack, it may be brittle against unseen stacks.
If it relies heavily on clean spectral cues, it may fail under compression or re-recording.
If it claims “general deepfake detection,” you should force it to prove generalization across datasets and perturbations.

This is also why I’m skeptical of one-number claims. A detector that’s “99% accurate” on dataset A might be “coin flip” on dataset B. And you’ll only discover that after you ship it, unless you build a real eval.

Best AI Voice Detector Tools in 2026 (Free and Paid)

You can find long lists of tools. I already wrote one with actual comparisons in AI voice detector: tools tested.

For this post, the key is: tools are only “best” relative to your threat model and operating constraints. Still, you should understand the common categories you’ll be evaluating:

Vendor provenance classifiers: for example, the ElevenLabs AI Speech Classifier returns a probability that audio was generated with ElevenLabs tech. On their own page they explicitly warn: it “Does not reliably classify audio generated with the Eleven v3 model.” That’s not a knock. It’s an honest statement that should shape your expectations.
General-purpose deepfake detectors: typically research-derived models or commercial services trained across multiple TTS/VC methods.
Call-center / telephony-integrated solutions: detectors plus VoIP integration, streaming, dashboards, and analyst workflows. See my deployment-focused write-up on deepfake voice detection.
DIY / open source research models: flexible, but you own the eval, the latency, and the failure modes.

Free vs paid in practice often comes down to:

Latency and throughput guarantees (paid vendors usually win)
API and integrations (paid wins)
Transparency into training data and failure cases (often open models win)
Cost predictability at scale (varies; do the math)

If you’re running a bake-off, shortlist 2–4 detectors from different categories. If all your candidates are the same kind of model, you’re not doing evaluation. You’re doing brand comparison.

(Visual break: “detector types” illustration.)

How Accurate Are AI Voice Detectors?

This question is always asked and almost always answered badly.

“How accurate are deepfake voice detectors?” is like asking “how accurate are spam filters?” Accurate on what emails? For what cost of false positives? With what adversary?

So I’ll define what “accurate” should mean in a vendor eval:

Discrimination: can the model separate real vs fake across a range of thresholds? (AUC)
Operating point performance: at your chosen threshold, what are false positives and false negatives? (confusion matrix)
Cost-weighted performance: what happens when a false negative costs $10,000 and a false positive costs $15 in handle time? (minDCF / custom cost metric)
Calibration: does “0.9 probability” actually mean “~90% of these are fake” over time? (ECE or Brier)
Robustness: does it still work after transformations your audio pipeline will apply?
Latency: can you run it fast enough to be useful in-call?

Here’s a compact table you can drop into an internal scorecard.

Metric	What it tells you	Why you should care in production	Common way it gets gamed
AUC (ROC-AUC)	Overall separability across thresholds	Good for comparing models before choosing a threshold	Evaluated on clean lab audio only
EER	Threshold where false accept = false reject	Useful sanity check for “balanced” performance	Hides asymmetry of real fraud costs
Calibration (ECE/Brier)	Whether scores map to real probabilities	Critical if you do risk-based routing	Vendors output “confidence” that isn’t calibrated
Cost-weighted metric (minDCF-like)	Performance under your loss function	Aligns model choice to business impact	Vendors pick a loss function that flatters them
Latency (ms)	Time to score per chunk / call	Determines streaming feasibility	Measured on tiny batches, not real throughput
Robustness suite pass rate (%)	How performance degrades under transforms	Predicts in-the-wild failure	Not reported at all

A concrete number you can use as a forcing function: if your call center can tolerate ≤ 250 ms added per 2-second audio chunk for streaming risk scoring, that’s your hard budget. Any detector that can’t hit it is not a “real-time detector.” It’s a post-call analytics tool.

If you want a model for how to build evals in general (not just audio), I’ve written extensively about regression gating in AI engineering evals and production monitoring in evaluate AI agents in production.

Can AI Voice Detection Be Fooled?

Yes. And the only interesting question is: which attacks matter in your production environment?

Most detector demos are evaluated on pristine inputs. Fraud doesn’t happen on pristine inputs. It happens through:

phone mics
speakerphones
call recording systems
conferencing apps
VoIP codecs
background noise
angry people talking over each other

So here’s the adversarial test matrix I actually care about. If your vendor can’t run this, you should run it yourself.

The “real call flow” transformation suite

At minimum, test these transformations on both real and fake audio:

Telephony bandlimiting: downsample / band-limit to 8 kHz (PSTN-like). This is the most common “silent killer.”
Codec transcoding: run through Opus and AAC at a few bitrates. Real systems re-encode.
Re-recording: play audio through a cheap speaker and capture it with a phone mic. This destroys many fingerprint-style cues.
Additive noise: office noise, street noise, café noise at multiple SNRs.
Packet loss / jitter simulation: if you do VoIP streaming, simulate dropped frames and jitter buffers.
Time-stretch / pitch shift (mild): not sci-fi. These happen accidentally in some pipelines, and adversaries can do them intentionally.
Overlapped speech: agent interrupts customer, customer talks over agent. Many detectors implicitly assume one speaker.

For each transformation, don’t just record “accuracy”. Record delta versus clean audio. A robust detector has graceful degradation. A brittle detector cliff-dives.

Hard negatives: the stuff that triggers false positives

If you deploy detection, your biggest operational pain is false positives. So you need “hard negative” audio:

strong accents
emotional speech (crying, yelling)
whispered speech
low-quality mics
people with speech impairments
background music (retail stores are a nightmare)

I’m being blunt here: many teams only test on “nice narrator voice in a quiet room.” That’s how you ship a detector that flags half of your customers in Toronto because they’re code-switching mid-sentence.

And yes, attackers can actively try to fool detectors. They can re-record, add noise, or choose generation tools that are less detectable. That’s why you treat this like any other adversarial classification system: you don’t ship it once. You monitor it.

If you’ve done any AI security work, you’ll recognize the pattern: threat model, red team, telemetry, iteration.

The Evaluation Protocol: A 7-Step Detector Scorecard You Can Reuse

This is the core of the post. It’s the part I wanted to exist when I started evaluating audio detectors for real systems.

I’ll lay it out as a protocol you can run in a week or two.

Step 1: Write the threat model in one page

If you skip this, you will measure the wrong thing.

Your one-pager should include:

Channel: inbound call center (PSTN/VoIP), voice notes, conferencing, etc.
Adversary capability: commodity voice clone tools vs targeted high-effort clones
Goal: social engineering for payout change, password reset, or reputation harm
Constraints: latency budget, privacy constraints, storage limits

Make it painfully specific. “AI fraud” is not a threat model.

Step 2: Build the dataset plan (lab + in-the-wild)

Use a mix:

Open benchmark datasets for reproducibility
Your own in-the-wild samples for reality

For open benchmarks, the industry anchor is the ASVspoof Challenge. It’s the de-facto hub for spoofing countermeasures for automatic speaker verification, and it provides shared evaluation plans and datasets across multiple editions.

ASVspoof has multiple task flavors over the years (logical access vs physical access vs deepfake). The exact split you use matters less than the discipline: consistent protocols, held-out evaluation sets, and reproducible reporting.

Your internal dataset should include at least 200–500 real calls (properly consented and handled) across your key customer segments. If you can’t collect that, you’re not ready to operationalize. You’ll be flying blind on false positives.

Step 3: Define the transformation suite and generate variants

Take every clip in your evaluation set and generate transformed variants using the matrix above.

Rule of thumb: aim for 10–20 variants per source clip. That sounds expensive until you realize it’s mostly automation.

Now you have an evaluation set that matches your pipeline.

Step 4: Choose metrics you will publish internally

Pick a minimal set and stick to it across vendors:

ROC-AUC
EER
Calibration: ECE or Brier
Latency: p50/p95 per chunk (or per file)
Cost-weighted score: your loss function

If you want a template for how to treat evals as a regression gate, borrow the approach from AI engineering evals and apply it to audio.

Step 5: Calibrate scores and pick thresholds based on cost

This is where most teams get lazy and then blame the model.

You should pick an operating threshold using:

estimated fraud loss per successful event (e.g., $5,000)
estimated analyst / handle-time cost per false positive (e.g., $10–$50)
your acceptable friction budget

If your fraud loss is 100x higher than your false positive cost, you should bias toward catching more even at the cost of more reviews. If it’s the opposite (high-volume, low-loss), you bias toward fewer false positives.

Write the numbers down. Make someone sign off.

Step 6: Run a red-team day

Treat this like a practical exercise, not a research paper.

Have a small group generate attacks:

re-recorded audio
multilingual samples
code-switching
overlapped speech
different TTS/VC providers

Log which ones slip through and why.

This is where you learn if your detector is robust or if it’s memorizing one family of artifacts.

Step 7: Produce a vendor scorecard (one page + appendix)

Your final output should be something your procurement and security leadership can read.

Include:

dataset card (what’s in/out)
transformation suite definition
metrics table
failure modes and examples
recommended threshold(s) by lane (high-risk vs low-risk)
operational plan (where it runs, what happens on “uncertain”)

If you want to formalize the “operational plan” piece, it maps well to the observability + governance work I’ve done for AI agents and AI in production. Different domain, same discipline.

(Visual break: scorecard template mockup.)

Datasets, Multilingual Edge Cases, and Drift Monitoring (The Stuff That Actually Breaks You)

Most posts never mention multilingual performance. That’s a mistake.

Detectors are trained on distributions. Languages shift distributions. Accents shift distributions. Code-switching shifts distributions mid-utterance.

If your customer base is multilingual (Canada says hi), you need to treat this as table stakes.

Multilingual evaluation plan

At minimum:

Include 3+ languages relevant to your users.
Include accented speech within the same language (not just “US English”).
Include code-switching samples (two languages in one call).
Measure false positive rate per segment.

A concrete operational target I like: your false positive rate should not increase by more than 2x for your top accent groups compared to your baseline. If it does, you’ll create an “AI detector” that is functionally an accent detector. That’s a reputational and legal hazard.

Drift monitoring in production

Deepfake voice detection is not set-and-forget.

You need:

a rolling holdout set of recent calls (consented, redacted)
periodic re-scoring (weekly or monthly)
alerting on distribution shifts in scores
incident review when your false positive queue spikes

If you’ve built monitoring for probabilistic systems before, you’ll recognize this as the same problem as LLM output drift. My mental model comes from building deterministic gates and feedback loops in this site’s publishing pipeline (the boring, repeatable checks catch more than “bigger model review” ever did). The lesson carries: measure first, then automate.

For logging and privacy patterns, borrow ideas from AI agent observability and adapt them for audio: store derived signals where possible, redact aggressively, keep retention short.

Audio Watermarking and the C2PA Provenance Standard

Detection answers: “does this look fake?”

Provenance answers: “where did this come from, and what happened to it?”

You want both.

The Coalition for Content Provenance and Authenticity (C2PA) publishes a technical specification for signing and verifying content provenance metadata. In plain English: content can carry cryptographic “content credentials” that tell you the origin and edit history, assuming the ecosystem adopts it and the metadata survives the journey.

Watermarking research (like Meta’s AudioSeal project, referenced in the research brief) is the other side: embed a robust signal into generated audio so you can detect origin even after transformations.

The hard reality:

Provenance won’t be universal. Attackers won’t cooperate.
Metadata can be stripped. Many platforms re-encode and drop it.
Watermarks are an arms race too. They can degrade under heavy transforms.

So the right production posture is defense-in-depth:

If provenance exists and verifies, treat it as a strong signal.
If provenance is missing, fall back to detection.
If detection is uncertain, fall back to operational verification.

This is the same pattern we use in LLM security and broader AI security: never bet the company on one probabilistic classifier.

What Should You Do If You Receive a Suspicious AI Voice Call?

This is the part people actually need when they’re in the moment.

If you receive a suspicious call (family emergency, bank request, executive request), here’s the playbook I’d want my own family to follow:

Assume the voice can be faked. Don’t argue about whether it “sounds real.”
Switch channels. Hang up and call back using a number you already trust (from contacts or official website).
Use a shared secret. Families and teams should have a simple code word for emergencies.
Slow it down. Scams rely on urgency. Create time.
Escalate inside the org. If it’s work-related, route to your fraud/security team with the recording if policy allows.

Notice how none of this requires a detector. That’s intentional. Detectors are great for systems. Humans need habits.

And yes, this aligns with the FBI’s emphasis on out-of-band verification for AI-enabled scams.

Use Cases: Who Needs an AI Voice Detector?

Not everyone needs deepfake voice detection. But if you match any of these, you should at least evaluate it:

Banks / fintech / crypto exchanges: account takeover and payout changes
Call centers: password resets, address changes, high-value support lanes
Enterprises with IT helpdesks: social engineering into SSO resets
Media and journalism: verifying leaked audio, preventing reputational hits
Marketplaces: seller/buyer disputes where voice evidence is used

A practical heuristic: if a successful impersonation event can cost you $10k+ (or a headline), detector evaluation is worth the effort. If your worst-case loss is $50, spend your time on better authentication and agent training.

Legal Requirements: AI-Generated Audio Disclosure in 2026

I’m not a lawyer, and you shouldn’t treat this as legal advice.

But you should assume the regulatory direction is clear: disclosure and provenance expectations will increase for AI-generated media, especially in advertising, political content, and consumer communications.

Even if your jurisdiction doesn’t mandate disclosure for every AI-generated audio use case, your risk team should care about:

consent for call recording and analysis
biometric / voiceprint handling policies
retention limits
explainability for adverse actions (e.g., blocking a customer)

This is another reason to invest in calibration, scorecards, and “what happens when uncertain.” Regulators don’t love black boxes that can’t justify decisions.

The bottom line: ship an evaluation harness, not a detector

Deepfake voice detection is going to follow the same trajectory as every other security classifier: the “model” becomes a commodity, and the competitive advantage shifts to evaluation, operations, and incident response.

If you’re a buyer, my challenge is simple: stop asking vendors for a demo. Ask them for a failure-mode report on 8 kHz telephony, re-recording, and multilingual audio.

If you’re a builder, my prediction is even simpler: within 12–18 months, the teams that win will treat voice deepfake detection like they treat RAG and other probabilistic systems. Continuous evals. Drift monitoring. Escalation lanes. No hero metrics.

The page-1 content for “deepfake voice detection” won’t be another listicle. It’ll be the first guide that gives security teams a protocol they can run, defend, and iterate. Build that, and the ranking will follow.

Originally published on kunalganglani.com

Debug HTTP/3 QUIC in Production: 8-Step Playbook [2026]

Kunal — Wed, 12 Aug 2026 00:43:02 +0000

HTTP/3 debugging in production is the art of proving what protocol real users negotiated (H3 vs H2 vs H1), why QUIC handshakes fail, and whether enabling H3 actually improved latency.

Key takeaways:

You haven’t “enabled HTTP/3” until you can show negotiated h3 on real requests, not just an Alt-Svc header.
The fastest Wireshark-free triage is: DevTools protocol column → Alt-Svc presence/caching → UDP/443 reachability → curl transcript → CDN/origin QUIC logs.
Silent fallback to HTTP/2 is normal behavior, not a bug. Treat it like a decision tree with observable checkpoints.
Measure impact by forcing H3 vs forcing H2 in the same client, with fresh connections, and by separating handshake time from TTFB.
If you can’t attach artifacts (NetLog JSON + curl -v output + edge log fields) to the incident ticket, you’re not debugging. You’re guessing.

On this blog I run a deterministic multi-step publishing pipeline with an incident log, and the lesson transfers cleanly to networking. Deterministic gates and reproducible artifacts beat vibes every time. The same mindset that saved me from shipping broken SEO templates at scale is the mindset that will save your on-call when QUIC is doing something weird at 2 a.m.

The 8-step HTTP/3 debugging checklist (copy/paste)

This section is intentionally procedural. When you’re on-call, you don’t want lore. You want a checklist.

In Chrome DevTools, confirm the negotiated protocol is actually h3.
Confirm Alt-Svc is being advertised on the right responses (and with a sane ma value).
Eliminate “connection reuse lies” (fresh profile/incognito, disable cache, new connection).
Validate UDP/443 reachability from the failing network (hotel Wi‑Fi, corp VPN, mobile carrier).
Force HTTP/3 using curl and capture a verbose transcript (--http3, -v, plus DNS/IPv6 notes).
Force HTTP/2 using curl and capture the same transcript (so you can compare apples-to-apples).
Pull CDN edge logs that include protocol + QUIC handshake failure reasons (or the closest available fields).
If you own the origin QUIC stack, enable QUIC logs / qlog long enough to capture a failing session.

You’ll notice what’s missing: packet capture. Most devs can’t do it on the device/network where the bug happens anyway.

Confirm HTTP/3 is being used in Chrome (not just Alt-Svc)

The most common failure mode I see is people stopping at “I see Alt-Svc: h3=\":443\" so we’re on HTTP/3.” That’s not confirmation. That’s advertising.

Here’s what I actually trust in Chrome:

1) Add the Protocol column in DevTools

Open DevTools → Network.
Right-click the header row (Name/Status/Type/…)
Enable Protocol.
Reload the page.

For each request, you’ll see something like h3, h2, or http/1.1.

2) Filter to the requests that matter
The page might load some third-party assets over h3 while your HTML document is still on h2. Filter for:

The HTML document request
Your critical JS bundle
Your main API call

3) Don’t get tricked by connection reuse / coalescing
HTTP/3 is multiplexed, and browsers reuse connections aggressively.

To avoid false confidence:

Use an Incognito window (fresh cache and connection pool).
In DevTools Network, check Disable cache.
Hard reload.

If you’re trying to debug “first visit” behavior, this matters. Alt-Svc can be cached, and the first navigation might be H2 even if subsequent navigations go H3.

4) Look at response headers anyway
You still want to check Alt-Svc, but interpret it correctly:

Is Alt-Svc present on the HTML response (not just static assets)?
Is it advertising the correct authority (hostname + port)?
Is ma long enough to matter? (ma=0 is effectively “don’t cache this.”)

A concrete number to keep in your head: ma is in seconds. So ma=86400 is one day.

Visual breathing room: this is a great place to insert an image showing DevTools with the Protocol column.

Verify HTTP/3 with curl (and read the verbose output)

Browsers are great, but they’re not reproducible enough for incident tickets. Curl is the “attachable artifact” tool.

What you want from curl is:

A transcript that shows which protocol was negotiated.
A transcript you can re-run from a CI runner, a jump box, or a teammate’s laptop.

The minimum curl commands I use

Force HTTP/3: curl -v --http3 https://example.com/
Force HTTP/2: curl -v --http2 https://example.com/

If you’re testing an API endpoint, hit a specific URL, not /.

How to interpret curl output

In verbose output, look for:

The resolved IP and whether it picked IPv4 or IPv6. (This matters for UDP reachability and MTU issues.)
ALPN negotiation. For TLS over TCP you’ll often see h2/http/1.1. For QUIC, you want h3.
Alt-Svc behavior. Curl may print Alt-Svc headers and it may cache them depending on build/options.

A practical workflow:

Run --http3 first.
If it fails, run --http2 immediately after from the same machine/network.
If H2 succeeds and H3 fails, you’ve narrowed the failure domain to UDP/QUIC/H3, not “site is down.”

If your curl doesn’t support HTTP/3, that’s also a signal. Not every environment has an ngtcp2/quiche-enabled build.

Visual breathing room: this is a good place for an image of terminal output with the key lines highlighted.

Why HTTP/3 silently falls back to HTTP/2 (the real reasons)

Silent fallback is a feature. HTTP/3 clients are supposed to recover.

In production, fallback usually happens for boring reasons:

1) UDP/443 is blocked

This is the big one. Enterprise firewalls, captive portals, and some VPNs still treat UDP like it’s suspicious.

Symptoms:

Browser shows h2 even though Alt-Svc is present.
Curl --http3 hangs or errors quickly, while --http2 works.

Fast test:

Try the same URL on a different network: home Wi‑Fi vs phone hotspot.
If hotspot works and corp Wi‑Fi doesn’t, stop blaming your CDN config.

2) Alt-Svc isn’t being sent where it needs to be

Common misconfig:

Alt-Svc only on static assets, not the main HTML document.
Alt-Svc stripped by an intermediate proxy.
Wrong port or authority advertised.

Remember: clients learn about H3 via Alt-Svc. If you never advertise it on the “entry” response, you’ll never graduate clients to H3.

3) QUIC version / ALPN mismatch

Even if UDP is open, protocol negotiation can fail.

Causes:

Client and server don’t share a QUIC version.
Server advertises h3 but the stack is misconfigured for the host/SNI.

You don’t need Wireshark to suspect this. You need server logs that say “handshake failed because …” (more on that below).

4) IPv6 weirdness and Happy Eyeballs outcomes

Sometimes H3 works over IPv4 but not IPv6 (or vice versa). Browsers race connections. That can make your problem look “flaky”.

Concrete example:

If IPv6 path has a smaller effective MTU and drops fragments, QUIC’s initial handshake packets can get blackholed.

5) MTU / fragmentation issues

QUIC handshake packets can be larger than you expect.

If a path drops IP fragments or ICMP “Packet Too Big” messages, you’ll see:

QUIC handshake failures on specific networks.
Retries or timeouts.

This is where packet capture helps, but you can still debug it Wireshark-free by correlating:

Which networks fail
Whether IPv6 fails more than IPv4
Whether failure correlates with specific client OS versions

Debug QUIC handshake failed (decision tree, fastest checks first)

When someone pings you with “debug quic handshake failed”, this is the order I check things.

Step 1: Is the client even attempting QUIC?

If DevTools shows h2, the browser might not be trying QUIC at all.

Check:

Was Alt-Svc present on the first response?
Is the user on a fresh profile vs a long-lived session?

Step 2: Did QUIC get attempted but abandoned?

This is where Chrome NetLog is your best weapon.

Chrome NetLog gives you a JSON file that includes QUIC session events. It’s noisy, but it’s the closest thing to a “flight recorder” you can get without packet capture.

Workflow:

Reproduce the issue in Chrome.
Export a NetLog.
Attach it to the incident.

Even if you never personally parse it, it gives your networking/SRE folks something concrete.

Step 3: Check UDP reachability from that network

If UDP is blocked, stop. There’s no magical server config fix.

Practical actions:

Confirm the environment: corporate VPN? hotel Wi‑Fi? mobile carrier?
Try a control test: same URL from another network.

Step 4: Compare forced H3 vs forced H2 using curl

This is the fastest way to separate “QUIC is broken” from “origin is slow.”

Capture both transcripts. Post them in the ticket.

Step 5: Pull edge logs for protocol and handshake outcome

If you’re on a CDN, you often can get log fields like:

negotiated protocol (h3 vs h2)
QUIC handshake status
edge colo
client IP / ASN

Even if the exact fields vary by vendor, the goal is consistent: turn “it fell back” into “it fell back for this cohort, on these networks.”

Step 6: Enable origin QUIC logs / qlog (if applicable)

If you control the origin (or you’re using something like nginx QUIC / Envoy QUIC / a QUIC-enabled load balancer), you want structured logs.

qlog is a common format used by QUIC implementations to record events like:

handshake phases
packet loss
transport errors

If you enable it, do it intentionally:

Sample it (1% or only for a specific path).
Time-box it (15–60 minutes).
Treat it like sensitive telemetry.

Measure HTTP/3 performance impact (TTFB, handshake time, reuse)

Enabling HTTP/3 can make things faster, but it’s not automatic. If your latency is dominated by origin compute, QUIC won’t save you.

Here’s how I measure without lying to myself.

What to measure

At minimum:

DNS (did we change resolution behavior?)
Connect / handshake time (this is where QUIC can shine)
TTFB (where origin and caching show up)
Content Download

Concrete numbers to anchor your analysis:

QUIC runs over UDP, typically port 443.
Alt-Svc caching is in seconds (ma=86400 is one day).
A “good” difference is not 5 ms on localhost. It’s a consistent delta across percentiles on real networks.

How to compare H3 vs H2 fairly

Rules:

Compare from the same client machine and network.
Force protocols with curl, or isolate sessions with incognito.
Control caching. You want to test both cold cache and warm cache.

Pitfalls:

Connection coalescing can mask differences.
CDN caching can dominate TTFB, making protocol differences vanish.
Some measurements include service worker behavior.

Browser tooling that actually helps

Chrome DevTools Network timings: good for quick “is connect time different?” checks.
Performance panel: useful for separating network from main-thread work.

If you’re already doing structured performance work, you might appreciate the mindset in my post on LLM latency where I argue for decomposing latency into components you can actually act on. Same principle here.

What to log on your CDN and origin to diagnose QUIC

This is the part most teams skip, because it’s “ops work.” Then they act surprised when debugging is impossible.

CDN/edge logs: the minimum fields I care about

Even if vendor names differ, I want:

Negotiated protocol (h3/h2/http/1.1)
Edge colo / region
Client ASN (or at least country)
Upstream connect time and response time
Error reason when QUIC fails (handshake failure, version mismatch, timeout)

If you can’t get the QUIC failure reason, at least log the negotiated protocol. Otherwise your incident ends in “seems fine on my machine.”

Origin logs: prove what reached you

If you terminate TLS/QUIC at the origin:

Log whether the request arrived over QUIC or TCP.
Log connection IDs if your stack exposes them.
Log handshake failures separately from request failures.

If you terminate at the CDN and talk H2 to origin, then origin won’t see QUIC at all. Don’t waste time looking for it in origin logs.

My bias: structured logs over ad-hoc grep

Running this blog’s multi-agent publishing pipeline taught me a painful but useful rule: deterministic gates and structured outputs catch issues earlier than “smart” review ever will. Apply that here. If your QUIC debugging depends on one person remembering a magic grep pattern, you don’t have a system.

QUIC retries and 0-RTT: what breaks and how to notice

QUIC has two concepts that matter in production:

Retry

A server can force a client to prove address ownership before committing resources. This can add an extra round trip.

If you suddenly see worse connect times on H3, retries might be happening.

How to detect without packet capture:

In NetLog/qlog, look for retry-related events.
Look for increased “connect” or “handshake” time in DevTools, but stable TTFB once connected.

0-RTT

0-RTT lets a client send application data immediately on a resumed connection. It’s not free.

What breaks 0-RTT in practice:

TLS session tickets not being reused (rotation too aggressive, wrong cache scope).
Load balancers that don’t preserve session resumption state.
Configuration changes that invalidate tickets.

Security note: 0-RTT data can be replayed. Don’t allow it for non-idempotent requests.

Middleboxes, firewalls, and MTU: network causes you can test

QUIC’s biggest enemy is not your app. It’s the network.

Common culprits

Corporate firewalls that block UDP or rate-limit it aggressively.
VPNs that tunnel TCP well but treat UDP as optional.
Captive portals that intercept traffic before the session is established.
Path MTU issues that blackhole larger QUIC packets.

A practical test matrix

When you’re trying to reproduce “HTTP/3 fallback to HTTP/2 troubleshooting” issues, don’t test on one network.

Test across at least:

Home Wi‑Fi
Phone hotspot
Corporate VPN

That’s 3 environments. If it fails in only 1 of 3, you’ve learned something actionable.

Concrete numbers again, because they matter: QUIC is typically UDP 443. If your firewall rules allow TCP 443 but block UDP 443, you will get fallback. Every time.

A production-first workflow: ship artifacts, not opinions

Here’s the workflow I recommend teams standardize. Not as a tribal “do this sometimes.” As a runbook.

The “attachable artifact” bundle

For every HTTP/3 incident, attach:

A screenshot (or HAR) showing DevTools protocol negotiation for the failing request.
A Chrome NetLog JSON captured during repro.
Two curl transcripts from the same machine/network: forced H3 and forced H2.
Edge log query results showing protocol distribution and failures for the affected window.

That bundle is enough for:

SREs to reason about network vs app
CDN support to escalate
Developers to reproduce and verify a fix

This is exactly how I think about AI in production too. Debugging is about turning “I think” into “I can show.”

Conclusion: HTTP/3 is easy to enable and hard to prove

HTTP/3 is now “on by default” in a lot of stacks, which is precisely why production debugging is painful. When it fails, it fails quietly, falls back, and your dashboards keep looking mostly green.

My prediction: within 12–18 months, teams will treat “protocol negotiation observability” as a first-class requirement the same way they treat TLS cipher visibility today. If you’re building a performance-sensitive frontend, you should get ahead of that.

The challenge is simple: next time you claim “we’re on HTTP/3,” attach one artifact that proves it. If you can’t, your system isn’t ready for the next incident.

Originally published on kunalganglani.com

Transparent Huge Pages + Postgres: Stop P99 Latency Cliffs [2026]

Kunal — Tue, 11 Aug 2026 12:44:28 +0000

Transparent Huge Pages (THP) Postgres performance problems almost never show up as “the database is slow.” They show up as a system that looks fine on averages and then occasionally falls off a cliff at p99. Suddenly you’ve got “random” timeouts, queue backups, and a pager that hates you.

I’m going to be blunt. If you run self-managed Postgres on Linux and you haven’t made an explicit decision about THP, you’re accepting tail latency risk for no good reason. The kernel is doing work you didn’t ask for, at the exact moment you can least afford it.

This post is the runbook I wish more teams had: detect → measure → change → validate → enforce. Especially if you’re running Postgres on Kubernetes nodes where config drift is basically a law of nature.

Key takeaways

THP is not the same thing as explicit Huge Pages (hugetlbfs). Confusing them is how teams “tune huge pages” and still keep the latency cliff.
For OLTP Postgres, the safest default is THP=never plus defrag=never. That combo stops background compaction from stalling your backends.
If you choose madvise, you still need to control defrag and verify whether Postgres is actually using THP. Otherwise you’ve just moved the cliff somewhere else.
Validate with p99/p99.9 latency and pg_stat_statements, not averages. Averages will lie to you.
In Kubernetes, this is a node guardrail. If you don’t enforce it, it will drift. And then you’ll “mysteriously” rediscover this post during an incident.

What Transparent Huge Pages (THP) is, and why Postgres teams keep getting burned

Transparent Huge Pages (THP) is a Linux kernel feature that automatically backs virtual memory with huge pages (commonly 2 MB) instead of base pages (commonly 4 KB). It can also promote and demote page sizes over time.

The kernel docs are pretty explicit about scope: THP mainly applies to anonymous memory mappings and tmpfs/shmem, and the intent is “performance without app changes” by handling page promotion/demotion automatically. (Linux kernel docs)

On paper, that’s a nice deal. Fewer TLB misses. Fewer page faults. Free speed.

In production with Postgres, it’s usually not a nice deal.

Postgres is latency-sensitive and memory-active under concurrent load. The thing that bites you isn’t the existence of huge pages. It’s the kernel doing automatic promotion plus defrag/compaction work at runtime, on your schedule, not its own.

The kernel docs call out the tradeoff clearly: huge pages reduce TLB misses and reduce page-fault frequency. One page fault per 2 MB region is 512x fewer faults than 4 KB pages. But page faults can become more expensive, and the kernel may need background work to create and maintain huge pages. (Linux kernel docs)

When that background work lands at the wrong moment, you get the classic “latency cliff.” Your median stays boring. Your p99 gets obliterated.

THP vs explicit Huge Pages (hugetlbfs) in Postgres

This is where a lot of “performance tuning” content goes off the rails.

Postgres docs talk about huge pages in the context of kernel resources. In the PostgreSQL 18 docs, 18.4. Managing Kernel Resources is the umbrella section, and 18.4.5. Linux Huge Pages is the part that’s actually about explicit huge pages, typically pre-allocated and managed intentionally. (PostgreSQL docs: 18.4 Managing Kernel Resources)

Explicit huge pages (hugetlbfs) are predictable. You provision them. You monitor them. You know what you’re getting.

Transparent huge pages are the opposite. They’re opportunistic and adaptive. That’s fine for some workloads. For OLTP Postgres, it’s where tail latency goes to die.

Why THP causes Postgres p99 spikes even when the average looks fine

Here’s the failure mode that keeps repeating in real systems:

Everything looks stable. Memory pressure is moderate but not catastrophic.
THP is enabled in always, so the kernel tries to allocate and maintain huge pages.
The kernel background thread (khugepaged) scans memory and tries to collapse 4 KB pages into 2 MB huge pages.
When it can’t find a contiguous 2 MB region easily, it triggers compaction/defrag work.
Compaction can stall processes. Those stalls show up as tail latency spikes in Postgres queries.

This is not some spooky “kernel magic.” It’s exactly what the THP design implies: promotion/demotion costs real CPU and can involve memory compaction. The sysfs knobs under /sys/kernel/mm/transparent_hugepage/ exist because the kernel authors know these tradeoffs matter. (Linux kernel docs: THP sysfs knobs and modes)

So why do averages look fine?

Most queries never hit the stall.
When it happens, it’s intermittent.
Your dashboards default to rolled-up means because observability vendors love pretty lines.

And why does p99 explode?

A few unlucky backends get paused during compaction.
Under load, one stalled backend turns into queues, lock waits, and cascading slowdowns.

Tail latency is where “the DB is fine” turns into a user-facing outage.

Exact THP modes and the recommended settings for Postgres OLTP

Linux exposes two knobs you actually care about for this topic:

/sys/kernel/mm/transparent_hugepage/enabled
/sys/kernel/mm/transparent_hugepage/defrag

The modes you’ll see are:

always: the kernel aggressively uses THP
madvise: the kernel uses THP only when applications request it via madvise()
never: the kernel doesn’t use THP

Those names aren’t blog-invented. They’re straight from the kernel docs. (Linux kernel docs)

My opinionated defaults (the ones that stop latency cliffs)

If you’re running OLTP Postgres (user-facing requests, spiky concurrency, real SLOs), the default that prevents most THP-induced cliffs is:

enabled = never
defrag = never

If you have a specific reason to keep THP around (some analytics-heavy mixed workloads, some VM setups), the least dangerous option is:

enabled = madvise
defrag = never

If you’re currently on always, you’re basically telling the kernel: “Feel free to do background memory work whenever you want.” That’s not a production posture. That’s a dare.

Safe defaults matrix (what you can actually change)

Environment	What you control	Recommended THP setting for Postgres OLTP	Notes
Bare metal, self-managed Postgres	Full kernel params	`never` + `defrag=never`	Most deterministic.
VM (cloud or on-prem)	Usually full kernel params	Start with `never` + `defrag=never`	Huge pages can help some virtualization overhead, but OLTP tail latency matters more.
Kubernetes (self-managed nodes)	Node kernel params	`never` + `defrag=never`	Enforce at node boot. Don’t try to “fix” inside the container.
Managed Postgres (RDS/Cloud SQL/Aurora/etc.)	Almost nothing	You can’t change THP	Don’t waste time hunting sysfs knobs you don’t own. Focus on query and schema tuning.

Step 1: Check system-wide THP usage (and don’t guess)

You want two facts, not vibes:

Current mode (enabled)
Current defrag behavior (defrag)

On most distros:

cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

You’ll see something like always madvise [never] where the brackets mark the active mode.

That’s the kernel’s sysfs interface doing exactly what it says on the tin. (Linux kernel docs: THP sysfs knobs and modes)

Also check whether khugepaged is active and burning CPU during incidents:

ps -eo pid,comm,pcpu,pmem,args | grep khugepaged
top / htop during a spike and look for khugepaged

If khugepaged CPU jumps at the same time your p99 jumps, that’s not “proof,” but it’s a pretty loud hint.

Step 2: Check THP usage per process (is Postgres actually using it?)

System-wide mode is necessary, but it’s not sufficient. You also want to know what’s happening inside the Postgres process.

At minimum:

Identify the Postgres PID (or the postmaster PID)
Inspect /proc/<pid>/smaps or /proc/<pid>/smaps_rollup and look for huge page indicators

On RHEL-family systems, Red Hat’s runbook includes practical steps for checking system-wide and per-process THP usage. This is one of those rare cases where the boring enterprise doc is the right reference because it’s written for operators, not for Twitter. (Red Hat KB 46111)

Why you care:

If you set madvise but nothing in your stack requests THP, you might be chasing a ghost.
If you set never but a tuned profile flips it back on, you’ll want receipts.

Step 3: Benchmark before you change anything (pgbench + p99)

If you change kernel memory behavior in prod without a baseline, you’re doing performance cosplay.

Here’s the loop I like:

Capture p50/p95/p99/p99.9 latencies from your app or from pgbench.
Capture a 5–15 minute window of pg_stat_statements.
Make the THP change.
Repeat the same measurement window.

You’re looking for a very specific shape:

Median might not move much.
p99/p99.9 should smooth out.
Query max times in pg_stat_statements should drop.

Use an actual number for the run. For example: run pgbench for 10 minutes at a concurrency of 64 and capture latency percentiles.

If your app has a known “spiky hour” (batch jobs, traffic peak, cron chaos), measure during that. Don’t benchmark at 2 p.m. and declare victory.

Step 4: Apply the fix — disable THP at run time (immediate mitigation)

When you’re in an incident, you want the change now, even if it won’t persist.

Runtime mitigation:

Write never to /sys/kernel/mm/transparent_hugepage/enabled
Write never to /sys/kernel/mm/transparent_hugepage/defrag

This is the operational equivalent of pulling the emergency brake. It won’t survive a reboot. That’s fine. The goal is to stop the cliff and stabilize.

Red Hat’s procedure covers the runtime disable path and it’s worth following if you’re on RHEL. (Red Hat KB 46111)

Step 5: Make it persistent — disable THP at boot time (the real fix)

If you don’t persist it, it will come back. And yes, it will come back at 3 a.m.

You’ve got a few durable options. Which one you use depends on what your fleet standards look like.

Option A: GRUB kernel parameters (works broadly)

Set the kernel command line so THP is disabled at boot.

Common pattern:

Add transparent_hugepage=never to the kernel cmdline

Then regenerate GRUB config and reboot.

This is the most reliable “set it and forget it” method across distros.

Option B: systemd-tmpfiles (cleaner than rc.local)

On systemd hosts, you can write the sysfs knobs at boot via a tmpfiles rule.

This avoids resurrecting rc.local hacks and keeps the change declarative.

Option C: tuned profiles (RHEL ecosystems)

If you use tuned, be careful. Some profiles override THP behavior. The worst case is thinking you disabled THP and then a tuned profile quietly flips it back.

Again, Red Hat is explicit about monitoring and disabling THP in an enterprise environment. Follow it if that’s your world. (Red Hat KB 46111)

Step 6: Validate with pg_stat_statements (Postgres 18/19-era workflow)

If you’re not using pg_stat_statements, you’re debugging Postgres blind.

The official docs are clear: pg_stat_statements must be loaded via shared_preload_libraries (so yes, you need a restart). It tracks planning and execution stats, and it relies on query identifier calculation, which is enabled when compute_query_id is auto or on. (PostgreSQL docs: pg_stat_statements)

That’s the 2026 angle that actually matters here: newer Postgres makes statement-level instrumentation less annoying, so you have fewer excuses.

F.32. pg_stat_statements — track statistics of SQL planning and execution

That’s literally the section title in the docs, and it’s exactly what you want. You’re not chasing a slightly faster mean. You’re trying to stop unpredictable stalls.

F.32.1. The pg_stat_statements View

The view has the columns you need to validate tail improvements: min/mean/max times, total time, and call counts. Rows are keyed by (dbid, userid, queryid, toplevel). (PostgreSQL docs: pg_stat_statements)

What I pull in practice:

Top queries by total execution time (to see if anything regressed)
Top queries by max execution time (to catch tail cliffs)
A before/after comparison window around the change

Also: reset stats when you do controlled experiments. Otherwise you’re mixing pre-change and post-change behavior and pretending it’s science.

Step 7: Validate with p99/p99.9 latency, not just statement stats

pg_stat_statements is necessary, but it’s not the same thing as end-to-end latency.

What I like to validate after the change:

App-level request p99 and p99.9 over at least 24 hours
DB pool wait time (if you have it)
Postgres wait events distribution (newer Postgres versions make this more useful)
OS signals: khugepaged CPU drops, fewer compaction stalls

If p99 got better but throughput dropped, you may have traded one bottleneck for another. That’s not the common outcome with THP changes, but don’t assume you’re immune.

Step 8: Kubernetes guardrails — enforce THP settings on nodes and prevent drift

In Kubernetes, the most common failure mode is trying to “disable THP inside the container.” That’s not how kernels work.

THP is a node-level setting. Treat it like vm.swappiness or overcommit policy. It lives with the node.

Two patterns that actually work:

Pattern 1: Node bootstrap / machine config (preferred)

If you run OpenShift, use a MachineConfig to set kernel args.
If you run Cluster API or managed node groups with custom AMIs, bake the kernel args into the image.

This is the highest-integrity approach. Nodes come up correct.

Pattern 2: Privileged DaemonSet (enforcement + audit)

If you can’t bake images quickly, use a privileged DaemonSet that:

Checks /sys/kernel/mm/transparent_hugepage/enabled and /defrag
Writes the desired value if it’s wrong
Exposes a node condition or metric so you can alert on drift

Be realistic: it’s still a band-aid compared to doing it at boot. But it’s miles better than hope.

Drift prevention checklist

Alert if any node reports always.
Alert if defrag is not never.
Gate node pools. Don’t schedule Postgres pods on nodes that fail the check.

If you’re doing this for a mission-critical database, treat it like a compliance control, not a one-time tweak.

Extra guardrails: kernel knobs that correlate with THP-related stalls

THP issues usually show up next to a couple other kernel behaviors. You don’t need to “tune the whole OS,” but you should know what tends to correlate.

The defrag knob is the big one

If I had to pick a single “stop the cliff” setting besides disabling THP entirely, it’s defrag=never.

That tells the kernel: don’t do expensive compaction work in the background just to satisfy huge page allocations.

Watch for memory pressure and compaction signals

Correlate Postgres p99 spikes with OS-level signals:

khugepaged CPU usage
Memory compaction activity
Swap activity (even small amounts)

If you see compaction and swap during p99 spikes, you don’t have a Postgres problem. You have a node problem.

Putting it together: the copy/paste runbook

Here’s the exact loop I’d run, in order.

Record baseline: p99/p99.9 latency for 30 minutes during representative load.
Check THP system-wide: read /sys/kernel/mm/transparent_hugepage/enabled and /defrag.
Check per-process: confirm whether the Postgres PID is using THP via /proc/<pid>/smaps(_rollup).
Enable measurement: ensure pg_stat_statements is on (restart required). (PostgreSQL docs)
Mitigate: set enabled=never and defrag=never at runtime to stop an incident.
Persist: set boot-time config via GRUB or a node image.
Validate: compare before/after p99 and pg_stat_statements max times.
Enforce (Kubernetes): machine config or privileged DaemonSet plus alerts.

Conclusion: make THP a deliberate decision, or the kernel will decide for you

The industry has a weird habit of treating kernel settings like folklore. THP isn’t folklore. It has documented modes. It has documented knobs. And for OLTP Postgres, the default Linux behavior is often the wrong one.

So don’t argue about THP in theory. Run the loop. Measure p99. Change one knob. Measure again.

My bet for the next year: as more teams run Postgres on multi-tenant Kubernetes node pools, THP misconfiguration becomes the new “noisy neighbor” incident class. The teams that win won’t be the ones with the fanciest database. They’ll be the ones who turned kernel behavior into an enforced guardrail.

Originally published on kunalganglani.com

Data Privacy in RAG Redaction and Retention [2026 Playbook]

Kunal — Tue, 11 Aug 2026 00:44:18 +0000

Data privacy in rag redaction and retention is not a “security checklist” problem. It’s an architecture problem.

Ship Retrieval-Augmented Generation (RAG) into anything regulated and you find out fast: your privacy posture is defined by a dozen tiny boundaries. Ingestion. Chunking. Embeddings. Retrieval filters. Prompt construction. Tool calls. Model outputs.

And then the graveyard where everything goes to die.

Logs, traces, analytics, replay systems. The stuff teams keep “just for a week” until it quietly becomes 180 days.

Most teams get one boundary right (usually “don’t train on my data”) and then leak sensitive data everywhere else.

Key takeaways

A RAG system has at least 8 privacy leak points. If you only redact at ingestion, you will still leak via prompts, logs, and traces.
Redact PII/PHI/PCI and secrets differently. Secrets require prevention and blocking, not just masking.
Use a retention matrix. Default to 7–30 days for operational traces, 0–7 days for raw prompts, and 90–365 days for compliance audit events that contain no raw text.
Treat embeddings as sensitive artifacts. They’re not “random vectors.” Retention and deletion must be designed, not wished into existence.
The best audit trail is structured, hashed, and policy-aware. It should prove what happened without storing what you wish never happened.

What is RAG and where privacy leaks happen

Retrieval-Augmented Generation (RAG) is the pattern where a model answers a user’s question using external knowledge retrieved at request time. Instead of fine-tuning a model on your private docs, you fetch relevant chunks from a search index or vector database and inject them into the prompt.

That’s why RAG is so attractive in enterprise and regulated domains. It’s also why privacy failures get… sneaky.

In a typical production pipeline, sensitive data can leak at eight places:

Source documents (PDFs, tickets, call transcripts, emails)
Chunked text (the “cleaned” form engineers often forget is still raw text)
Embeddings (vectors derived from sensitive text)
Vector DB metadata (tenant IDs, doc IDs, ACLs, tags)
Retrieval results (the top-k chunks your system thinks are relevant)
Prompt assembly (where you concatenate user query + system prompt + retrieved context)
Generation output (which can repeat or transform sensitive data)
Observability (logs, traces, analytics events, replay systems)

I’ve built RAG systems that handle millions of queries daily with sub-second response times for the Walmart conversational commerce chatbot. At that scale, the privacy problem stops being theoretical. One overly-verbose trace attribute doesn’t leak “a little.” It leaks thousands of secrets per hour. Not because anyone is evil. Because volume turns every mistake into an incident.

The 2026 change is that AI governance is finally becoming operational. Teams are standardizing policy-as-code for prompt/context filtering and adopting privacy-preserving tracing (hashed identifiers, structured audit events) to satisfy audits without hoarding raw prompts.

Here’s the playbook I wish more teams shipped with.

PII and secrets: what to redact (and what not to)

The fastest way to build a broken privacy system is to treat “sensitive data” like it’s one big bucket.

In RAG, you usually have four categories that behave very differently:

PII (names, emails, phone numbers, addresses, government IDs)
PHI (health info, diagnoses, prescriptions, patient IDs)
PCI (card numbers, CVV, bank account identifiers)
Secrets (API keys, OAuth tokens, session cookies, private keys, internal credentials)

Plus a fifth that matters in enterprise but gets hand-waved way too often:

Proprietary text (contracts, incident reports, roadmap docs, source code)

Redact PII/PHI/PCI, but don’t destroy meaning

PII redaction in RAG pipelines is about reducing risk without turning your index into mush.

If you delete every name, address, and number, retrieval quality falls off a cliff. And in production RAG, retrieval quality is the whole game.

I learned this building the Walmart RAG pipeline. Model choice mattered, but retrieval quality dominated at scale. If your privacy controls crater retrieval, the “fix” the product team reaches for is predictable. “Increase k.” “Log more.” “Add more context.” Congratulations, you just made privacy worse.

So for PII/PHI/PCI:

Prefer token-preserving masking over deletion. Replace names with stable placeholders like PERSON_1, PERSON_2 within a single document.
Preserve document structure (headings, tables, bullet points). Structure is signal.
Keep non-sensitive numeric values when they’re essential (prices, dimensions, dates in public policies). Over-redaction creates useless context and forces more retrieval.

Secrets are different: block, rotate, and alert

Secrets detection in LLM systems is not mainly a “masking” problem. It’s an incident prevention problem.

Once a credential gets into the pipeline:

it can land in logs,
it can get echoed by the model,
it can get cached,
it can end up in an eval dataset,
it can be pasted into a ticket by someone trying to “help.”

So for secrets:

Detect and hard-block known secret formats (API keys, JWTs, bearer tokens, private keys).
If a secret shows up in user input, respond with a safe error and trigger rotation workflows.
If a secret shows up in retrieved context (yes, this happens), quarantine the source doc and re-index.

Practical rule: if it looks like a credential, treat it like a production incident. Not like a string to redact.

Redaction enforcement points: ingestion, retrieval, prompt construction, and logging

Teams always ask: “Should we redact before embedding or at retrieval time, or both?”

My stance: both. But for different reasons. If you pick one, you’ll end up compensating somewhere else. Usually in logging. And that’s how leaks become permanent.

1) Ingestion-time redaction (before chunking and embedding)

Ingestion-time redaction is your first shot at reducing blast radius.

Do it here because:

raw docs are the richest leak source,
you can run heavier detectors (DLP, regex + ML, custom rules),
you can store two versions: a raw vault copy (restricted) and a redacted RAG copy.

In healthcare or finance, it’s common to keep raw docs in a separate evidence vault with a different retention policy. Your RAG path should almost never touch raw.

Ingestion-time output should be:

redacted chunks for indexing
a redaction manifest (what was removed, by what rule, at what time)
provenance metadata (source system, owner, tenant, classification)

2) Retrieval-time filtering (tenant, ACL, and policy checks)

Retrieval is where multi-tenant isolation either works… or you end up on a call you don’t want.

Your retriever must enforce:

Tenant isolation (hard partition or mandatory tenant filter)
Document-level ACLs (user can only retrieve what they can access)
Policy filters (e.g., “this user role cannot retrieve PHI”)

Do not rely on the LLM to “refuse” forbidden data. That’s like asking a logging library to enforce RBAC.

3) Prompt-time redaction (last-mile safety)

Prompt construction is the last place you control the content before it hits a model API.

This is where you catch:

residual PII from ingestion failures,
secrets via user input or retrieved text,
prompt injection payloads trying to force exfiltration.

If you’re already doing prompt filtering for prompt injection, extend the same policy engine to redact sensitive spans. Same machinery. Different rules.

4) Logging-time minimization (where most leaks actually happen)

LLM prompt logging retention policy is where teams self-own.

It usually starts with a reasonable intention: “we need traces to debug hallucinations.”

Then you log:

the full user query
the full retrieved chunks
the full prompt
the full model output

…and you keep it for 180 days because “compliance.”

That’s not compliance. That’s a breach backlog with a nice dashboard.

The fix is to split observability into two streams:

Compliance audit events: structured, minimal, long retention.
Debug traces: richer, short retention, access-gated, redacted.

I wrote a full schema approach in AI Agent Observability Logging Schema [2026]: OTel + Redaction. This post is the same idea, applied specifically to RAG.

Retention policies for RAG artifacts (documents, embeddings, prompts, traces)

Retention gets political fast. Security wants “keep nothing.” Debugging wants “keep everything.” Legal wants “keep whatever the policy says, and prove you did it.”

This is one of those things where the boring answer is actually the right one. Build a retention matrix. Implement it with TTLs, deletion jobs, and an audit trail that doesn’t rely on storing raw content forever.

Here’s a default retention matrix you can start with. Adjust by domain and regulation.

Default retention matrix (practical starting point)

Artifact	Contains raw text?	Sensitivity risk	Suggested default retention	Implementation pattern
Raw source docs (vault)	Yes	Very high	1–7 years (policy-driven)	Separate storage + strict access + legal hold
Redacted docs/chunks (RAG store)	Yes	High	90–365 days	Object store TTL + re-ingestion pipeline
Embeddings	No (but derived)	Medium–high	90–365 days	Vector DB TTL per record + re-embed on refresh
Vector metadata (ACL, tenant)	No	Medium	As long as embedding	Same lifecycle as embedding
User queries	Yes	High	0–30 days	Store redacted form; hash identifiers
Assembled prompts	Yes	Very high	0–7 days	Prefer not storing; store prompt hashes
Retrieved context snippets	Yes	Very high	0–7 days	Store doc IDs + chunk hashes, not text
Model outputs	Yes	High	0–30 days	Store redacted summaries; avoid raw
Debug traces (full request graph)	Sometimes	Very high	7–30 days	Separate sink + access control + sampling
Compliance audit events	No (structured)	Low–medium	1–3 years	Append-only store, tamper-evident

You’ll notice two things:

1) The stuff you want for debugging (prompts, retrieved context) has the shortest retention.

2) The long-retention trail is mostly IDs, hashes, and policy decisions.

That’s how you get both: debuggability and defensible privacy. Anything else turns into either blind debugging or an accidental data lake.

Do embeddings contain PII? How should embeddings be treated?

Embeddings are derived from sensitive text. They can leak information via membership inference or reconstruction attacks under certain conditions. And they absolutely count as “personal data” in many governance programs because they’re linked to identifiers and can be used to profile.

Practically: treat embeddings as sensitive artifacts.

Put them under retention.
Put them under deletion.
Encrypt them.
Restrict access.

If your design assumes embeddings are harmless, you will eventually build “vector analytics” that becomes a shadow data warehouse. It happens slowly, then all at once.

GDPR/CCPA deletion in a vector database: the part everyone avoids

If you support deletion requests, you need a real answer to: “How do we delete a person’s data from the vector index?”

There are only a few options that actually work:

Hard delete by document IDs: maintain a mapping from source doc → chunk IDs → embedding IDs. Cleanest.
Tombstones + async purge: mark embeddings deleted, exclude at query time, purge later.
Re-embedding: if PII was embedded into shared chunks, you may need to re-chunk and re-embed affected documents.

Operational reality: deletion is a pipeline, not a database call.

Design your indexing around stable IDs and lineage:

source document ID
chunk ID
embedding ID
tenant ID
classification

That lineage is also what makes audit trails possible.

Audit trails for RAG: what to log for compliance vs debugging

A good RAG audit trail answers two questions:

1) What happened? (who queried, what policies applied, which documents were accessed)
2) Can you prove it? (immutability, integrity, reproducibility)

A bad audit trail is “we stored every prompt for 2 years.” That’s not an audit trail. That’s a liability with extra steps.

What should an audit trail contain for RAG?

Here’s an audit event schema I’ve seen hold up well. It’s intentionally not raw-text.

timestamp
request_id (correlation ID)
tenant_id (or hashed tenant)
actor_id (hashed user ID)
actor_role (admin, support, end-user)
client_app (web, mobile, internal tool)
query_class (support, product Q&A, medical, financial)
retrieval_policy_version
redaction_policy_version
model_provider + model_id
prompt_template_version
retrieved_doc_ids (list)
retrieved_chunk_ids (list)
retrieved_chunk_hashes (optional)
decision_flags (blocked_secret, blocked_phi, allowlist_hit, denylist_hit)
output_classification (safe, contains_pii_suspected, blocked)
latency_ms (end-to-end)

Notice what’s missing: the user’s full query and the raw retrieved text.

If you need query-level detail for abuse investigations, store a redacted query or a hash plus a short-lived encrypted debug packet in a separate system.

Separate compliance logs from debug traces

This separation is not optional in regulated environments.

Compliance logs are append-only, long retention, minimal data.
Debug traces are short retention, sampled, access-gated, and aggressively redacted.

If you don’t split them, your compliance system becomes your highest-risk data store.

This is also where modern tracing helps. When you instrument your RAG pipeline like a distributed system, you can log structured spans (retrieval latency, reranker latency, generation latency) without storing raw content. When you need content occasionally, capture it on a sampling path with explicit approvals.

If you’re already working on evals, tie auditability to your regression gates. My approach in AI Engineering Evals: Regression Gates for Prompts, Tools, RAG [2026] is to treat “privacy regressions” like correctness regressions.

Encryption and access controls for vector databases and logs

Most “RAG privacy” advice stops at redaction. That’s incomplete.

Even perfectly redacted systems fail if:

anyone in the org can query the vector DB,
logs are accessible to the whole engineering org,
service-to-service permissions are broad.

Vector database security: the basics that still get missed

For your vector database:

Encrypt at rest (KMS-managed keys, ideally per environment)
Encrypt in transit (mTLS between services)
Enforce RBAC. Retrieval service accounts should not have admin permissions.
Enforce tenant isolation. Separate indexes per tenant or mandatory tenant filters enforced server-side.
Treat metadata like a boundary. A doc ID can be sensitive if it maps to a customer.

If you’re using Postgres with pgvector, lean on Postgres RBAC and row-level security. If you’re using a managed vector DB, verify that RBAC is enforced at query time, not just in dashboards.

Logs and traces: least privilege + “break glass” access

Your logging platform is often broader-access than your databases. That’s backwards.

Default debug traces to restricted access (on-call, security, a small number of engineers).
Add break-glass workflows: temporary access with approvals, fully audited.
Separate environments. Production traces shouldn’t be casually accessible from dev accounts.

In the Walmart chatbot system, we leaned heavily on event streaming for context pipeline latency. Kafka made the pipeline fast. It also creates another place to leak data if you don’t treat topics as sensitive. Partition topics by sensitivity, encrypt payloads where needed, and don’t let “observability” topics become an ungoverned dump.

Testing and monitoring: validating redaction and catching regressions

You can’t “promise” data privacy. You can only prove it continuously.

Here’s what works.

Build redaction fixtures and run them in CI

Create a small corpus of synthetic documents and user queries that contain:

multiple PII types (email, phone, address)
PHI patterns (patient ID, diagnosis codes)
PCI patterns (test card numbers)
secrets (fake API keys, JWT-like tokens)

Then test the pipeline at multiple points:

after ingestion redaction
after retrieval filtering
after prompt assembly
in the logging output

Same mindset as prompt regression testing. You’re not “hoping” you didn’t leak. You’re gating releases on it.

If you want a broader framework for these gates, Evaluate AI Agents in Production: 2026 Testing Guide is a good complement.

Monitor with metrics, not vibes

You should have metrics like:

redaction_hit_rate: % of requests where any redaction occurred
secret_block_rate: % blocked due to secrets
pii_suspected_rate: sampled detection on outputs
unredacted_span_rate: traces that contain raw content fields

Pick a baseline and alert on regressions. Even a 0.1% leak rate is catastrophic at scale.

Sampling and review: the human layer that still matters

Automated detection misses edge cases. Use sampling, but do it safely:

sample redacted traces by default
allow short-lived access to raw content only with approvals

In 2026, “privacy-preserving tracing” is the compromise that works. You debug latency and retrieval behavior with IDs and hashes. You pull raw content only on an incident path.

Incident response: how audit trails help investigate leaks

When something leaks, the worst moment to discover your logging strategy is when you’re already in incident mode and someone asks: “So… what exactly got stored?”

A strong audit trail lets you answer quickly:

Which tenant was affected?
Which user role queried it?
Which documents were retrieved?
Which policy version allowed it?
Did redaction run?
Which model version generated the response?

A practical incident workflow

Here’s a workflow I like because it respects both privacy and reality:

Contain: disable high-risk features (long-context retrieval, tool calls) for affected tenant(s).
Correlate: use request_id to trace the retrieval chain.
Verify: pull the exact retrieved document IDs and chunk IDs.
Reproduce: replay using hashes and template versions, not the raw prompt.
Remediate: fix the policy or ingestion redaction, re-index, rotate secrets if applicable.
Prove: write the incident report referencing audit events, not raw user data.

If you don’t have structured audit events, you’ll fall back to grepping logs for raw prompts. That’s slow, it’s messy, and it tends to create a second leak while investigating the first.

For broader security hardening beyond privacy, pair this with The Complete Guide to AI Security in 2026 and AI security work.

A practical end-to-end architecture (where to put the controls)

If you want a concrete system picture, here’s the enforcement map I recommend for production AI.

Controls by boundary

Client SDK / UI: detect obvious secrets before they leave the device; warn users.
API gateway: rate-limit, tenant auth, basic PII/secret scanning, request IDs.
Ingestion pipeline: heavy redaction + classification + manifests; store redacted RAG copy.
Retriever service: tenant partitioning, ACL checks, policy filters, denylist/allowlist rules.
Prompt builder: last-mile redaction; prompt injection defenses; template versioning.
LLM provider: use no-training/no-retention modes where available, but don’t pretend this is your main control.
Observability: split compliance audit events from debug traces; TTL and access controls.

This is also how you keep your production AI posture sane. When everything becomes “just one more JSON field in the trace,” you will ship a privacy regression.

The policy-as-code approach (2026 reality)

You want policies that are:

versioned
testable
deployable independently
referenced in audit events

Same discipline you apply to CI/CD.

If you’re already thinking in terms of control flow and gates, AI Agent Control Flow Patterns [2026]: Retries, HITL, Checkpoints has the same spirit. The system is the product, not the prompt.

Closing: the prediction

RAG is becoming the default “enterprise LLM” architecture. That also means RAG is becoming the default enterprise privacy failure mode.

My prediction: by late 2026, the teams that win audits won’t be the ones with the fanciest redaction model. They’ll be the ones who can point to a retention matrix, a policy version, and a tamper-evident audit event stream, and say: “Here’s exactly what we store, for how long, and why.”

If you’re building RAG today, stop treating data privacy like a bolt-on. Design it the way you design latency budgets. End-to-end. Measurable. Enforced at every boundary.

Want a challenge? Pick one artifact from the retention matrix above and actually implement TTL + deletion + audit proof for it this week. That’s the difference between a demo and a system you can defend.

Originally published on kunalganglani.com