How I made a Rust hot path 27x faster, and the AI fix I refused to merge

Two years ago I open-sourced KeyEcho, a small desktop app that plays a mechanical-keyboard sound the instant you press a key. It got 800+ stars. Then I shipped v0.0.5 in July 2024 and went quiet.

The issues never stopped. People asked for sound packs, reported platform bugs, and kept using a thing I had stopped maintaining. This month I came back and shipped 1.0 in a single PR: 130 files, 11,405 lines added and 9,292 removed. The core of it was a rebuild of the audio hot path. The cached-lookup microbenchmark went from 1184.07 ns/op with 66.84 KiB copied per key to 43.50 ns/op with zero sample bytes copied. That is 27x on the average slice and 38x on the largest.

I built it with an AI agent writing a lot of the code. This post is how the rework works, why Rust made that safe to do fast, and the one change the agent suggested that looked clean and would have quietly broken a feature.

The hot path

KeyEcho's latency-sensitive path is short. A global keyboard hook fires on every key event. The first key-down of each press is pushed into a bounded queue. An audio thread pulls from the queue, maps the key to a slice of the selected sound pack, and plays it locally.

Because the path is short, anything left inside it is paid on every keystroke: any decode work, any allocation, any sample copy, any lock. The whole game is getting those out of the per-key path.

v0.0.5 was already fast

Credit where it is due. v0.0.5 was a light, quick thing: Tauri + Rust, small builds, low memory; native keyboard listening I wrote by hand against each platform's API, with almost no unsafe; lossless WAV audio with no decompression step; and it already cached decoded audio in an LRU, so hitting the same key twice did not decode twice. It ran for two years and a few hundred people used it. It was not slow software.

But "already fast" and "no headroom left" are two different things. Even on a cache hit, each key press still copied the samples (66.84 KiB in the benchmark) and went through a global mutex shared with pack switching and volume changes. A cache miss still decoded on the spot. What 1.0 removed is exactly those: that copy, that lock, and the fact that cache misses happen at all.

The rebuild

The rebuild came down to four moves.

Predecode once, when a pack is selected. A sound pack is a folder with a sound.ogg and a config.json. The config maps each key to a slice of the audio: key -> [start_ms, duration_ms]. When you select a pack, every slice is decoded up front, one time. Nothing decodes during a keystroke.

Deduplicate identical slices. Many keys point at the same slice, so decoding per key would decode the same audio many times. The builder keys a map on the (start_ms, duration_ms) pair, decodes each unique slice once, and shares it:

// Decode each unique slice once; identical slices share one buffer.
let mut slice_sources: HashMap<[u64; 2], AudioSource> = HashMap::new();
let mut key_sources: HashMap<Key, AudioSource> = HashMap::new();

for (key, slice) in defines {
    let source = slice_sources
        .entry(slice)
        .or_insert_with(|| {
            let [start_ms, duration_ms] = slice;
            let samples = decode(start_ms, duration_ms); // Vec<f32>, once
            AudioSource::new(Arc::from(samples), channels, sample_rate)
        })
        .clone(); // Arc clone, not a sample copy
    key_sources.insert(key, source);
}

Zero-copy playback. Decoded samples live in an Arc<[f32]>. A key lookup returns a clone of that Arc, which is a reference-count bump, not a sample copy. The per-key cost went to zero bytes copied.

Drop the global mutex. The old path guarded the current sound and the volume with a mutex. The new playback handle holds the current sound in an ArcSwapOption<KeySound> and the volume in an AtomicU32 (via f32::to_bits). Looking up a key is a lock-free load:

struct PlaybackSoundpack {
    current_sound: Arc<ArcSwapOption<KeySound>>,
    volume_bits: Arc<AtomicU32>,
}

fn source_for_key(&self, key: Key) -> Option<(AudioSource, f32)> {
    let sound = self.current_sound.load();          // lock-free
    let source = sound.as_ref()?.key_source(key)?;  // map get + Arc clone
    let volume = f32::from_bits(self.volume_bits.load(Ordering::Relaxed));
    Some((source, volume))
}

Predecoding trades work up front for RAM, so it needs two guardrails. Key events go through a bounded queue, so a burst applies backpressure instead of growing memory without limit. And a pack has to fit a decoded-sample budget of 10 MiB: before it loads, the app estimates decoded size from the unique slice durations and refuses anything larger. Predecoding is only safe when it is bounded.

The numbers

Release-build microbenchmarks of the lookup path:

Path	v0.0.5	v1.0	Change
Cached lookup, average slice	1184.07 ns/op; 66.84 KiB copied	43.50 ns/op; 0 bytes copied	27.2x faster
Cached lookup, largest slice	1638.57 ns/op; 98.88 KiB copied	43.10 ns/op; 0 bytes copied	38.0x faster
Press/release gate	58.82 ns/tap; 2 messages	54.69 ns/tap; 1 message	half the messages

These are microbenchmarks of the lookup path, not end-to-end speaker latency, which depends on your OS and hardware. The method and memory budgets are in docs/performance.md, and the benchmarks ship in the repo. Run pnpm run bench:audio if you do not believe me.

Why Rust made this safe to do fast

I leaned on the agent for most of this diff. The reason that felt safe is the compiler.

The borrow checker and the type system are the first reviewer for AI-generated code. Most wrong code does not survive cargo check. Moving shared audio buffers to Arc<[f32]> and removing the mutex are exactly the changes where a mistake is an aliasing or a Send/Sync error, and Rust rejects those at compile time, before they reach a human review or a user. The more of the diff an agent writes, the more that guardrail is worth.

What the agent actually did

Not "it wrote the app." The useful work was narrower and, honestly, more valuable.

• Migration assistant. Tauri 1 to 2 touches config, permissions, the updater, and every plugin boundary. A partner that has read all the migration docs saves real hours.
• Hot-path auditor. It went through the old playback path with me line by line and drove the predecode, shared-buffer, no-mutex rework above.
• Backlog triage. I had it read every open issue and sort by what belonged in 1.0. One of them was a 10-month-old request from someone who had offered to pay for a specific sound. Replying to it after 1.0 turned into the project's first paying customer. The backlog was demand I had stopped reading.
• Benchmark and docs discipline. It kept the performance notes conclusion-first, reproducible, and honest about what the numbers do and do not measure.

I rarely gave step-by-step orders. I gave goals ("get sample copies in the key path to zero", "sort the backlog by what belongs in 1.0") and reviewed at the checkpoints. The speedups themselves come from the rework: predecoding, shared buffers, the dropped mutex, the bounded queue. The agent is what made finding, doing, and verifying that whole loop fast enough to actually happen.

The change that looked clean and wasn't

This is the one I want you to steal.

CI for the Linux armv7 build failed under QEMU: libgit2 could not fetch a git dependency. The agent's fix was clean. Drop the git pin on the audio crate and use the crates.io release instead. CI went green.

I rejected it. That pin exists for a reason that is written nowhere in the code. It holds cpal at 0.18, which ships the default-device rerouting (and its DeviceChanged notification) that makes the sound follow you when you unplug headphones or switch to a Bluetooth speaker. Reverting to the crates.io version silently downgrades cpal to 0.17.3 and drops device-following. Two user issues, #41 and #20, were about exactly that behavior. No test covered "unplug the device and the sound follows," so nothing would have caught the regression except knowing why the pin was there.

The real fix was to vendor the dependencies with cargo vendor and build offline inside QEMU. The pin never moved.

Two lessons I now treat as rules:

1. Write the "why" next to every pin, hack, and magic number, in a comment or the project's AI rules file. Your agent has no memory of the incident that put it there. It is a talented engineer with zero context on your project.
2. Never accept a dependency version change you did not ask for, even when CI is green. A green pipeline proves the tests pass. It does not prove the tests cover what you are about to lose.

What I cut

The armv7 build ran about two hours under emulation, and there was no clear user for 32-bit ARM Linux. I dropped it from 1.0.

An agent has no sense of sunk cost or opportunity cost. It will keep optimizing a build target forever if you let it. It can get arbitrarily close to "done." Deciding to stop is a human job.

How the work was structured

Two smaller notes from shipping this.

Before release I ran an AI review pass inside git worktrees, so several agents could scan the code and propose fixes in parallel without touching the main tree. Each agent's output was a proposal, merged only after review. Exploration parallelizes cleanly. The synthesis still has to happen in one head, because each agent only sees its own slice.

The models are also starting to specialize. Claude Fable feels like an engineer who thinks outside the box and keeps surfacing angles I had missed. GPT-5.6 is the best executor I have used: hand it a decided plan and the code comes back close to perfect. It felt like running a two-person team where one thinks, one builds, and I decide and sign off.

What 1.0 is

Still free, still open source. AGPL-3.0, small native installers, no account, no analytics. Rebuilt on Tauri 2 + SolidJS. Signed Windows builds and notarized macOS builds, which cut down unsigned-app warnings and antivirus false positives. Linux packages for x64 and ARM64.

You can try it without downloading anything: open keyecho.app and type, and you will hear it in the browser. The benchmarks and method are in the repo.

I write a weekly build log about shipping real production systems with AI agents, honest numbers only. Subscribe at upweb.dev.

Comments

[mote]

The ArcSwapOption + AtomicU32 combo is a really nice pattern for this kind of "rarely writes, always reads" hot path. I have used arc-swap in a few audio-adjacent projects and the one thing that always bites me is the drop-order guarantee: when you replace the ArcSwapOption, the old Arc may still be held by an in-flight playback thread, so the actual memory reclamation is deferred until the last reader drops. In practice that means pre-decoding into Arc<[f32]> is safe, but you still need to cap the total number of packs held in memory or you will eventually OOM on rapid pack-switching during a long session. Your 10 MiB budget guardrail is exactly the right call.

I wanted to flag the f32::to_bits / AtomicU32 volume trick for anyone who has not seen it: this is a cheap, portable way to store an f32 in an atomic without pulling in atomic_float or doing CAS loops, but you lose the edge case where NaN has multiple bit patterns. In audio that probably does not matter, but in a storage engine it does, because NaN != NaN means two reads of the same atomic can return different bit patterns and break equality checks downstream. I have been burned by this once in a WAV header cache where the sample rate was read as NaN after a torn write. Not saying you should change it here — the volume is display-only, not a structural value — but it is a good pattern to label with a comment if it ever gets copied into something that does depend on bitwise equality.

The dependency-pin story is the real gem. I keep a WHY_PINS.md in every repo now after an AI agent "fixed" a libsqlite3-sys pin that was holding a custom WAL mode we had patched. CI went green, the PR looked clean, and we almost shipped a release that would have silently downgraded write durability for half our users. The agent had no memory of the incident that created the pin, and the original PR description did not mention it either. A one-line comment on the pin itself would have saved the whole chain.

Two questions:

1. Did you measure the tail latency under burst key events, or only the average? The bounded queue is great for backpressure, but I am curious if the Arc clone + ArcSwap read ever shows a rare spike when the global ref-count drops to zero and the allocator reclaims the backing array.

2. Have you considered using crossbeam::epoch for the sound pack pointer instead of arc-swap? It would let you delay the actual deallocation to a quiescent point, which might be useful if you ever want to support pack switching without a perceptible pause in the audio thread.

[Zach]

Honest answers, in order. Tail latency: not measured. The 1184 to 43.5 ns numbers are averages per slice size, and the benchmark never provokes the case you describe, a drop landing on the audio thread when the last ref dies. So the spike you're asking about is untested rather than absent. That measurement goes on the list before the next hot-path change.

crossbeam::epoch: I didn't weigh it at the time, and I take the point. Epoch buys a quiescent point for frees, which is the strictly correct place for them. If the tail measurement ever shows reclamation spikes, that's the fix I'd reach for.

On the NaN caveat: volume here is display-and-apply only, nothing does bitwise equality on it, but the label comment costs one line and I'll add it. Cheap insurance against future copy-paste.

WHY_PINS.md is a better name than docs/performance.md for the thing I actually needed. Stealing it.

[mote]

Thanks for the correction -- I conflated the two changes. The cpal downgrade story hits harder actually. Silent dependency shift that passes CI and only shows up in Cargo.lock. That's the kind of failure no code review catches because there's nothing wrong with the diff. The diff is fine, the lockfile isn't.

Your ArcSwapOption pattern for the audio thread is clean. We hit a similar problem in moteDB's mmap layer -- the hot path reads from memory-mapped pages but the background thread needs to remap when the file grows. We used a generation counter + atomic swap so the reader never waits on the writer. Same idea, different substrate.

One thing I'm curious about: with predecoded slices, what's the memory overhead per second of audio? Our mmap approach trades CPU for RAM and it works great for database pages (4KB fixed size), but for variable-length decoded audio frames I'd worry about fragmentation.

[Zach]

Generation counter plus atomic swap for remap is the same shape, agreed, and a database page cache has the harder version of it.

On memory: decoded f32 at 44.1 kHz stereo costs about 345 KB per second, and keystroke slices are short, tens to a couple hundred milliseconds each. A full pack lands in single-digit MB, which is why the 10 MiB budget acts as a backstop instead of a squeeze. Fragmentation stays boring for the same reason: slices are allocated once at pack load and freed as a unit when the pack goes, so lifetimes are pack-scoped and nothing churns per keystroke. Variable-length frames with independent lifetimes, like your case, is the version of this problem I'm glad I don't have.

[Nazar Boyko]

The green CI on that dropped pin is the scariest line in the whole post: green proved the tests passed, not that they covered device-following, which had no test at all. Nothing stood between "CI is happy" and a shipped regression except you remembering why the pin was there. Honestly a stronger case for writing down the "why" next to every pin than any style guide I've read.

[Zach]

That's the line I'd underline too. The uncomfortable part is that writing the why down is necessary but not enough. I did put it in docs/performance.md later, but a doc doesn't block a merge. The thing that actually stops this regression is a test that fails when device-following breaks, and that test still doesn't exist. The green CI told the truth: the tests passed. It just wasn't testing the thing that mattered.

[Wren Calloway]

The pin story is the right instinct, but the deeper lesson is that the compiler didn't catch it and couldn't. Your firewall for AI code is cargo check plus Send/Sync — and that's genuinely strong for the Arc/mutex rework, because aliasing bugs are type errors there. But the cpal pin is a semantic constraint that lives entirely outside the type system: 0.17.3 compiles fine, passes every test, ships green. The class of bug the agent will bite you on is exactly the one the borrow checker has nothing to say about — a dependency whose behavior changed but whose signature didn't.

The concrete defense is to move that constraint somewhere a machine can see it. A one-line comment above the pin ("holds cpal >=0.18 for DeviceChanged, see #41 #20") turns tribal knowledge into something the next agent — or the next you at 1am — reads before proposing the "clean" fix. Better still, an integration test or a doc-check that asserts the DeviceChanged codepath exists, so the downgrade fails loud instead of silently dropping headphone-follow. Right now the only thing standing between that pin and a merged regression is that you personally remembered why it was there, and that's the part that doesn't scale with agent throughput.

[Zach]

You're right, and I'll own it: this constraint lives outside the type system, and cargo check can't help. It covers the Arc/mutex half of the story, not this half. One small correction for accuracy's sake: the why did get written down later, in docs/performance.md. But your core point stands. No machine enforces it, and there is still not a single behavioral test for "unplug the headphones and the sound follows." Best review this post has gotten. Thank you.

[Alex Shev]

Refusing the AI fix is the important part. A faster-looking patch is not automatically a better patch if it hides the reason the hot path improved or changes the contract around it. Performance work needs receipts: benchmark shape, input assumptions, correctness checks, and why the change will stay safe under real usage.

[Zach]

I'm stealing "receipts." What this post can show: the benchmarks ship in the repo (pnpm run bench:audio), and docs/performance.md spells out what the numbers measure and what they don't. This comment section also found the receipt that's missing: there is no behavioral test for device-following. Adding that one is worth more than another 10x.

[Alex Shev]

That missing behavioral test is the useful discovery from the whole thread. Benchmarks prove the hot path improved, but the device-following test proves the optimization did not quietly change the product contract. That is usually the receipt that matters most after the speed graph looks good.

[Zach]

Closing the loop on this one: the device-following test went to the top of the list, ahead of any further hot-path work. This thread named the missing receipt; the least I can do is go print it.

[PubliFlow]

The AI refusal to merge is a great point, as AI often optimizes for local performance but misses broader architectural context or introduces subtle bugs in hot paths. I have seen similar issues where AI generated Rust code looks clean but introduces hidden allocations that ruin the very performance it was supposed to fix. When building out the backend logic for our own Next.js and Supabase SaaS boilerplate, we had to carefully evaluate AI suggestions for our database queries to avoid similar hidden overhead. It is a good reminder that while tools like PubliFlow can accelerate initial development, the critical hot paths still need a human eye to ensure they actually perform in production.

[Valentyn Kit]

Would genuinely like the diff on the fix you refused, because "looks clean, silently breaks a feature" is the exact thing an agent hands you ten times a day, and catching it is the skill that doesn't transfer to whoever just merges what the model wrote. The 27x is the easy half of this post.

[Zach]

The fix was one line: swap the audio crate's git source for its crates.io release. What made it look clean: the release carries the exact same version number, 0.22.2, so in the manifest it reads like hygiene, a git URL becoming a registry version.

The damage sat two levels down. That release resolves cpal 0.17, and the default-device rerouting users asked for in #41 and #20 lives in cpal 0.18. The only place any of it surfaced was Cargo.lock.

Catching it took exactly one piece of knowledge: why the git source was there in the first place. You're right that this doesn't transfer to whoever just merges what the model wrote. What it takes is the context that never got written down.

[Theo Valmis]

The refusal is the interesting part of this post, not the 27x. Most "AI wrote a faster version" stories stop at the benchmark; the harder skill is knowing which faster version you shouldn't ship, and that's judgment no benchmark number will ever supply.

[Zach]

Agreed, and there's a funny pattern here: the 27x in the title is what pulled people in, and the refusal is all anyone wants to talk about. The judgment itself wasn't mystical either. I knew why the pin existed, because two real users had asked for device-following in the issues. No benchmark supplies that, and neither does reading the diff. It lives outside the diff.

[mote]

The 27x improvement on the hot path is a great example of why zero-copy in Rust is worth fighting for — the borrow checker enforces the safety guarantees that let you be this aggressive without segfaults.

The part that resonates most with my experience: the AI-suggested refactor looked clean but would have quietly broken sound pack switching during playback. That's exactly the failure mode that makes latency optimization dangerous — the bug isn't a crash, it's silent state divergence.

The bounded queue design is solid. I've used a similar pattern in moteDB (a Rust embedded multimodal database) for the event buffer between sensor ingestion and storage — the key constraint is that the queue must be lock-free and wait-free on the consumer side, otherwise you get priority inversion when the storage thread backs up.

What was the mechanism of the AI-suggested change that would have broken sound switching? Was it deferring the sample decode until the playback thread, or something else?

[Zach]

Appreciate the moteDB detail. Priority inversion on a backed-up consumer is exactly what the bounded queue and the lock-free load are there to avoid.

One correction though, since two things got tangled: the change I refused wasn't the one near pack switching.

• Switching packs during playback isn't broken by the rework, it's handled by it. The current sound sits in an ArcSwapOption, so switching a pack is a pointer swap on the producer side; the audio thread does a lock-free load and never blocks or decodes.
• The change I refused was one level down in the dependency graph: dropping the git pin silently moved cpal from 0.18 to 0.17 and dropped default-device rerouting, which is the device-following. It compiled, passed CI, and only showed up in Cargo.lock.

On the decode question, it's the other way around. The whole rework moves decode out of the playback thread: every slice is predecoded when the pack is selected, so the hot path is just an Arc clone. Deferring decode to the playback thread is the thing I was removing, because that's where the per-key cost and jitter came from.

[Edu Peralta]

The scariest part of this story isn't that the agent proposed dropping the pin. It's that the fix would have passed CI clean. A downgrade from cpal 0.18.1 to 0.17.3 doesn't fail a build, it just quietly removes a feature two users already asked for in open issues. That's the real gap with agent generated fixes, they optimize for the check that's in front of them, not the context that never made it into a comment or a test. cargo vendor as the actual solution is the right instinct. Curious if you're now enforcing a "why" comment on every pinned dependency as a lint rule, or still counting on someone remembering to write it next time?

[Zach]

Honest answer: somewhere in between. The why is written down in docs/performance.md, but nothing enforces it. I don't quite trust a lint that demands a why-comment on every pin; that mostly produces comments nobody reads. What I actually want is to mine the review discussion where the pin was decided into a rule, and let the rule do the blocking. That's what I'm building with DiffLore (pre-launch: difflore.dev). Where would you draw the line between a real rule and a one-off exception?