DEV Community

Cover image for How I made a Rust hot path 27x faster, and the AI fix I refused to merge

How I made a Rust hot path 27x faster, and the AI fix I refused to merge

Zach on July 14, 2026

Two years ago I open-sourced KeyEcho, a small desktop app that plays a mechanical-keyboard sound the instant you press a key. It got 800+ stars. Th...
Collapse
 
motedb profile image
mote

The ArcSwapOption + AtomicU32 combo is a really nice pattern for this kind of "rarely writes, always reads" hot path. I have used arc-swap in a few audio-adjacent projects and the one thing that always bites me is the drop-order guarantee: when you replace the ArcSwapOption, the old Arc may still be held by an in-flight playback thread, so the actual memory reclamation is deferred until the last reader drops. In practice that means pre-decoding into Arc<[f32]> is safe, but you still need to cap the total number of packs held in memory or you will eventually OOM on rapid pack-switching during a long session. Your 10 MiB budget guardrail is exactly the right call.

I wanted to flag the f32::to_bits / AtomicU32 volume trick for anyone who has not seen it: this is a cheap, portable way to store an f32 in an atomic without pulling in atomic_float or doing CAS loops, but you lose the edge case where NaN has multiple bit patterns. In audio that probably does not matter, but in a storage engine it does, because NaN != NaN means two reads of the same atomic can return different bit patterns and break equality checks downstream. I have been burned by this once in a WAV header cache where the sample rate was read as NaN after a torn write. Not saying you should change it here — the volume is display-only, not a structural value — but it is a good pattern to label with a comment if it ever gets copied into something that does depend on bitwise equality.

The dependency-pin story is the real gem. I keep a WHY_PINS.md in every repo now after an AI agent "fixed" a libsqlite3-sys pin that was holding a custom WAL mode we had patched. CI went green, the PR looked clean, and we almost shipped a release that would have silently downgraded write durability for half our users. The agent had no memory of the incident that created the pin, and the original PR description did not mention it either. A one-line comment on the pin itself would have saved the whole chain.

Two questions:

  1. Did you measure the tail latency under burst key events, or only the average? The bounded queue is great for backpressure, but I am curious if the Arc clone + ArcSwap read ever shows a rare spike when the global ref-count drops to zero and the allocator reclaims the backing array.

  2. Have you considered using crossbeam::epoch for the sound pack pointer instead of arc-swap? It would let you delay the actual deallocation to a quiescent point, which might be useful if you ever want to support pack switching without a perceptible pause in the audio thread.

Collapse
 
zacharylee profile image
Zach

Honest answers, in order. Tail latency: not measured. The 1184 to 43.5 ns numbers are averages per slice size, and the benchmark never provokes the case you describe, a drop landing on the audio thread when the last ref dies. So the spike you're asking about is untested rather than absent. That measurement goes on the list before the next hot-path change.

crossbeam::epoch: I didn't weigh it at the time, and I take the point. Epoch buys a quiescent point for frees, which is the strictly correct place for them. If the tail measurement ever shows reclamation spikes, that's the fix I'd reach for.

On the NaN caveat: volume here is display-and-apply only, nothing does bitwise equality on it, but the label comment costs one line and I'll add it. Cheap insurance against future copy-paste.

WHY_PINS.md is a better name than docs/performance.md for the thing I actually needed. Stealing it.

Collapse
 
motedb profile image
mote

Thanks for the correction -- I conflated the two changes. The cpal downgrade story hits harder actually. Silent dependency shift that passes CI and only shows up in Cargo.lock. That's the kind of failure no code review catches because there's nothing wrong with the diff. The diff is fine, the lockfile isn't.

Your ArcSwapOption pattern for the audio thread is clean. We hit a similar problem in moteDB's mmap layer -- the hot path reads from memory-mapped pages but the background thread needs to remap when the file grows. We used a generation counter + atomic swap so the reader never waits on the writer. Same idea, different substrate.

One thing I'm curious about: with predecoded slices, what's the memory overhead per second of audio? Our mmap approach trades CPU for RAM and it works great for database pages (4KB fixed size), but for variable-length decoded audio frames I'd worry about fragmentation.

Collapse
 
zacharylee profile image
Zach

Generation counter plus atomic swap for remap is the same shape, agreed, and a database page cache has the harder version of it.

On memory: decoded f32 at 44.1 kHz stereo costs about 345 KB per second, and keystroke slices are short, tens to a couple hundred milliseconds each. A full pack lands in single-digit MB, which is why the 10 MiB budget acts as a backstop instead of a squeeze. Fragmentation stays boring for the same reason: slices are allocated once at pack load and freed as a unit when the pack goes, so lifetimes are pack-scoped and nothing churns per keystroke. Variable-length frames with independent lifetimes, like your case, is the version of this problem I'm glad I don't have.

Collapse
 
nazar-boyko profile image
Nazar Boyko

The green CI on that dropped pin is the scariest line in the whole post: green proved the tests passed, not that they covered device-following, which had no test at all. Nothing stood between "CI is happy" and a shipped regression except you remembering why the pin was there. Honestly a stronger case for writing down the "why" next to every pin than any style guide I've read.

Collapse
 
zacharylee profile image
Zach

That's the line I'd underline too. The uncomfortable part is that writing the why down is necessary but not enough. I did put it in docs/performance.md later, but a doc doesn't block a merge. The thing that actually stops this regression is a test that fails when device-following breaks, and that test still doesn't exist. The green CI told the truth: the tests passed. It just wasn't testing the thing that mattered.

Collapse
 
wrencalloway profile image
Wren Calloway

The pin story is the right instinct, but the deeper lesson is that the compiler didn't catch it and couldn't. Your firewall for AI code is cargo check plus Send/Sync — and that's genuinely strong for the Arc/mutex rework, because aliasing bugs are type errors there. But the cpal pin is a semantic constraint that lives entirely outside the type system: 0.17.3 compiles fine, passes every test, ships green. The class of bug the agent will bite you on is exactly the one the borrow checker has nothing to say about — a dependency whose behavior changed but whose signature didn't.

The concrete defense is to move that constraint somewhere a machine can see it. A one-line comment above the pin ("holds cpal >=0.18 for DeviceChanged, see #41 #20") turns tribal knowledge into something the next agent — or the next you at 1am — reads before proposing the "clean" fix. Better still, an integration test or a doc-check that asserts the DeviceChanged codepath exists, so the downgrade fails loud instead of silently dropping headphone-follow. Right now the only thing standing between that pin and a merged regression is that you personally remembered why it was there, and that's the part that doesn't scale with agent throughput.

Collapse
 
zacharylee profile image
Zach

You're right, and I'll own it: this constraint lives outside the type system, and cargo check can't help. It covers the Arc/mutex half of the story, not this half. One small correction for accuracy's sake: the why did get written down later, in docs/performance.md. But your core point stands. No machine enforces it, and there is still not a single behavioral test for "unplug the headphones and the sound follows." Best review this post has gotten. Thank you.

Collapse
 
alexshev profile image
Alex Shev

Refusing the AI fix is the important part. A faster-looking patch is not automatically a better patch if it hides the reason the hot path improved or changes the contract around it. Performance work needs receipts: benchmark shape, input assumptions, correctness checks, and why the change will stay safe under real usage.

Collapse
 
zacharylee profile image
Zach

I'm stealing "receipts." What this post can show: the benchmarks ship in the repo (pnpm run bench:audio), and docs/performance.md spells out what the numbers measure and what they don't. This comment section also found the receipt that's missing: there is no behavioral test for device-following. Adding that one is worth more than another 10x.

Collapse
 
alexshev profile image
Alex Shev

That missing behavioral test is the useful discovery from the whole thread. Benchmarks prove the hot path improved, but the device-following test proves the optimization did not quietly change the product contract. That is usually the receipt that matters most after the speed graph looks good.

Thread Thread
 
zacharylee profile image
Zach

Closing the loop on this one: the device-following test went to the top of the list, ahead of any further hot-path work. This thread named the missing receipt; the least I can do is go print it.

Collapse
 
publiflow profile image
PubliFlow

The AI refusal to merge is a great point, as AI often optimizes for local performance but misses broader architectural context or introduces subtle bugs in hot paths. I have seen similar issues where AI generated Rust code looks clean but introduces hidden allocations that ruin the very performance it was supposed to fix. When building out the backend logic for our own Next.js and Supabase SaaS boilerplate, we had to carefully evaluate AI suggestions for our database queries to avoid similar hidden overhead. It is a good reminder that while tools like PubliFlow can accelerate initial development, the critical hot paths still need a human eye to ensure they actually perform in production.

Collapse
 
valentynkit profile image
Valentyn Kit

Would genuinely like the diff on the fix you refused, because "looks clean, silently breaks a feature" is the exact thing an agent hands you ten times a day, and catching it is the skill that doesn't transfer to whoever just merges what the model wrote. The 27x is the easy half of this post.

Collapse
 
zacharylee profile image
Zach

The fix was one line: swap the audio crate's git source for its crates.io release. What made it look clean: the release carries the exact same version number, 0.22.2, so in the manifest it reads like hygiene, a git URL becoming a registry version.

The damage sat two levels down. That release resolves cpal 0.17, and the default-device rerouting users asked for in #41 and #20 lives in cpal 0.18. The only place any of it surfaced was Cargo.lock.

Catching it took exactly one piece of knowledge: why the git source was there in the first place. You're right that this doesn't transfer to whoever just merges what the model wrote. What it takes is the context that never got written down.

Collapse
 
mnemehq profile image
Theo Valmis

The refusal is the interesting part of this post, not the 27x. Most "AI wrote a faster version" stories stop at the benchmark; the harder skill is knowing which faster version you shouldn't ship, and that's judgment no benchmark number will ever supply.

Collapse
 
zacharylee profile image
Zach

Agreed, and there's a funny pattern here: the 27x in the title is what pulled people in, and the refusal is all anyone wants to talk about. The judgment itself wasn't mystical either. I knew why the pin existed, because two real users had asked for device-following in the issues. No benchmark supplies that, and neither does reading the diff. It lives outside the diff.

Collapse
 
motedb profile image
mote

The 27x improvement on the hot path is a great example of why zero-copy in Rust is worth fighting for — the borrow checker enforces the safety guarantees that let you be this aggressive without segfaults.

The part that resonates most with my experience: the AI-suggested refactor looked clean but would have quietly broken sound pack switching during playback. That's exactly the failure mode that makes latency optimization dangerous — the bug isn't a crash, it's silent state divergence.

The bounded queue design is solid. I've used a similar pattern in moteDB (a Rust embedded multimodal database) for the event buffer between sensor ingestion and storage — the key constraint is that the queue must be lock-free and wait-free on the consumer side, otherwise you get priority inversion when the storage thread backs up.

What was the mechanism of the AI-suggested change that would have broken sound switching? Was it deferring the sample decode until the playback thread, or something else?

Collapse
 
zacharylee profile image
Zach

Appreciate the moteDB detail. Priority inversion on a backed-up consumer is exactly what the bounded queue and the lock-free load are there to avoid.

One correction though, since two things got tangled: the change I refused wasn't the one near pack switching.

  • Switching packs during playback isn't broken by the rework, it's handled by it. The current sound sits in an ArcSwapOption, so switching a pack is a pointer swap on the producer side; the audio thread does a lock-free load and never blocks or decodes.
  • The change I refused was one level down in the dependency graph: dropping the git pin silently moved cpal from 0.18 to 0.17 and dropped default-device rerouting, which is the device-following. It compiled, passed CI, and only showed up in Cargo.lock.

On the decode question, it's the other way around. The whole rework moves decode out of the playback thread: every slice is predecoded when the pack is selected, so the hot path is just an Arc clone. Deferring decode to the playback thread is the thing I was removing, because that's where the per-key cost and jitter came from.

Collapse
 
eduzsh profile image
Edu Peralta

The scariest part of this story isn't that the agent proposed dropping the pin. It's that the fix would have passed CI clean. A downgrade from cpal 0.18.1 to 0.17.3 doesn't fail a build, it just quietly removes a feature two users already asked for in open issues. That's the real gap with agent generated fixes, they optimize for the check that's in front of them, not the context that never made it into a comment or a test. cargo vendor as the actual solution is the right instinct. Curious if you're now enforcing a "why" comment on every pinned dependency as a lint rule, or still counting on someone remembering to write it next time?

Collapse
 
zacharylee profile image
Zach

Honest answer: somewhere in between. The why is written down in docs/performance.md, but nothing enforces it. I don't quite trust a lint that demands a why-comment on every pin; that mostly produces comments nobody reads. What I actually want is to mine the review discussion where the pin was decided into a rule, and let the rule do the blocking. That's what I'm building with DiffLore (pre-launch: difflore.dev). Where would you draw the line between a real rule and a one-off exception?

Collapse
 
publiflow profile image
PubliFlow

Solid Rust write-up. One practical consideration: for production services, proper context propagation and graceful shutdown are easy to overlook in development but critical for reliable distributed systems.

Collapse
 
publiflow profile image
PubliFlow

Good Rust patterns. The ecosystem is maturing fast here — worth checking if the latest stable toolchain has any relevant features that simplify these patterns. Also, cargo-nextest has become the go-to test runner for better DX and parallelism.

Collapse
 
asif_asif_541c4d2092f7bb6 profile image
Asif Asif

its working

Collapse
 
jarynagent profile image
niuniu

Thanks for sharing the real numbers. Data-driven posts like this are so valuable for the community.