DEV Community: devs

Capturing the mic and the call as two separate tracks on Windows with cpal and WASAPI loopback

devs — Mon, 26 Oct 2026 12:44:01 +0000

I'm building a Windows app, Talkaside, that transcribes both sides of a call live. The microphone is one side. The other side is whatever the call app plays through the speakers or headphones. Those need to be two separate tracks, because each one gets its own speech-to-text session, its own language and its own speaker label.

This post covers the capture layer, a Rust crate on top of cpal 0.15. It covers how the system audio is tapped, how both tracks end up as mono 16 kHz, and the gating that stops the microphone from re-transcribing the speakers.

Two pipelines, not one mixer

The obvious design is to mix everything into one stream and let the recognizer sort it out. That fails quickly: you lose who said what, and a recognizer tuned for one language sees two.

So the engine starts one pipeline per direction, called outgoing (the microphone) and incoming (the call). Each pipeline owns its own capture stream, its own resampler, its own STT session and its own recording file. When both are requested the engine sets a dual_channel_mode flag, which switches on the cross-track gating described below.

Tapping system audio

On Windows, cpal enables WASAPI loopback when you build an input stream on a render endpoint. There is no separate loopback API to call. The relevant constructor in this repo:

pub fn new_loopback(output_device_name: &str, /* ... */) -> Result<Self> {
    let device = find_output_device(output_device_name)?;
    let default_cfg = device
        .default_output_config()
        .context("Failed to get default output config for loopback capture")?;

    let channels = default_cfg.channels();
    let sample_rate = default_cfg.sample_rate().0;
    let config = StreamConfig {
        channels,
        sample_rate: default_cfg.sample_rate(),
        buffer_size: cpal::BufferSize::Default,
    };
    // ... build_input_stream_{f32,i16,u16}(&device, &config, ...)

Two choices here are deliberate. The stream runs at the device's default config instead of a requested 16 kHz, which avoids negotiating a rate the device may not support. Resampling happens later, on our side. And the sample format is matched at runtime (F32, I16, U16), with a separate callback for each.

The UI stores "system output" as a sentinel device name, __system_output_loopback__. One helper, is_system_loopback_device, is the only branch; the rest of the pipeline is shared.

Finding devices by name

Windows device names are localized strings, and cpal can expose more than one audio host. The lookup tries the default host first, then every other host cpal reports. It matches exactly, then falls back to a normalized substring match: lowercase, punctuation to spaces, whitespace collapsed. A small table maps Russian Windows names (the words for microphone, speakers, headphones, stereo mix) to their English equivalents before comparing, so a setting saved on an English install still resolves on a Russian one.

The audio callback does almost nothing

The callback runs on the driver's thread, so it only downmixes and hands off:

move |data: &[f32], _info: &cpal::InputCallbackInfo| {
    let mono: Vec<f32> = if channels == 1 {
        data.to_vec()
    } else {
        data.chunks(channels as usize)
            .map(|frame| frame.iter().sum::<f32>() / channels as f32)
            .collect()
    };
    let chunk = AudioChunk { samples: mono };
    if let Err(e) = sender.try_send(chunk) {
        note_dropped_chunk(&dropped_chunks);
        debug!("Capture channel full or disconnected: {}", e);
    }
},

It uses try_send into a bounded channel of 512 chunks. If the consumer stalls, the callback never blocks. It bumps an AtomicU64 and moves on. The consumer swaps that counter to zero on every chunk it receives and, when it was nonzero, emits a warning like [incoming] capture queue dropped count=N. Without it, a drop shows up as missing words and nothing in the logs.

Resampling to 16 kHz

Both streaming recognizers the app uses accept 16 kHz PCM, but devices run at 44.1 or 48 kHz. The engine keeps one StreamingResampler per track. It is plain linear interpolation, chosen for being cheap and predictable on speech, with one rule: the position of each output sample is computed from integer counters, never from an accumulated float.

let wanted = self.received as u128 * self.to as u128 / self.from as u128;
while (self.emitted as u128) < wanted {
    let position = self.emitted as u128 * self.from as u128;
    let index = (position / self.to as u128) as u64;
    let fraction = (position % self.to as u128) as f32 / self.to as f32;
    // ... interpolate pending[local] and pending[local + 1]
    self.emitted += 1;
}

Because the output count is a pure function of the input count, the result doesn't depend on how the driver happens to chunk the audio. A test feeds 512,000 samples through six rate pairs (48000 to 16000, 44100 to 16000, 8000 to 16000, 16000 to 16000, 24000 to 44100, 24000 to 48000) in chunks of 1, 127, 512 and 997 samples, and asserts the output is identical to converting the whole buffer at once. Another test pushes 1,000 chunks of 512 samples at 48 kHz and asserts the internal buffer never holds more than 3 samples and that exactly 170,666 samples came out.

The trade-off: linear interpolation without a low-pass filter lets some aliasing through. For speech going into a recognizer that is a reasonable trade. For music it would not be.

Stopping the microphone from transcribing the speakers

Without headphones, the microphone hears the call. Then the same sentence is transcribed twice, once per track, and the second copy is wrong and late. In dual mode the engine gates the microphone while the incoming track is active:

let level = audio_level(samples);
if above_audio_gate(level, OUTGOING_MIC_RMS, OUTGOING_MIC_PEAK) {
    *last_strong_outgoing_audio = Some(Instant::now());
    return None;
}
if let Some(last_seen) = last_strong_outgoing_audio {
    if last_seen.elapsed() <= Duration::from_millis(OUTGOING_MIC_HANGOVER_MS) {
        return None;
    }
}
if monotonic_wall_clock_ms() > incoming_audio_active_until.load(Ordering::SeqCst) {
    return None;
}
if below_audio_gate(level, OUTGOING_BLEED_RMS, OUTGOING_BLEED_PEAK) {
    return Some(AudioDropReason::QuietOutgoing);
}
Some(AudioDropReason::IncomingActive)

The incoming track marks itself active for 1,100 ms whenever a chunk crosses RMS 0.006 or peak 0.025. A microphone chunk passes through if it is clearly louder than bleed (RMS 0.010 or peak 0.035) and keeps passing for 650 ms afterwards, so the user can talk over the other person and the ends of their words aren't clipped. Anything quieter during incoming activity is dropped.

These thresholds are hand-tuned constants, not learned values. They suit typical laptop microphones and speaker volumes. A loud speaker next to a sensitive microphone can still leak through, and a very quiet speaker can be gated away. This is gating, not echo cancellation.

One more detail matters. Dropped chunks are not simply skipped: the engine calls ensure_stt_keepalive, so the STT session is not left idle while the gate is closed. The recorder also sits before the gate, so the saved file keeps the full track. Gating only decides what the recognizer sees, and a muted track is recorded as zeros of the same length so the timeline stays aligned.

Knowing when a track went quiet

A dead device and a quiet room look the same: all zeros. The pipeline logs input level over the first 10 seconds, then watches a rolling 60-second window and warns once if the track stays under the activity gate, resetting when signal returns. A separate idle-only probe opens both devices for 3 to 5 seconds, refuses to run if both selections resolve to the same device, and reports aggregate RMS and peak without keeping any PCM.

Limits

Loopback is bound to one output device, resolved when the pipeline starts. If that device disappears, the pipeline reports an audio-device-lost event instead of silently following another one.
Per-application capture isn't used: the loopback includes everything that device plays, including notification sounds.
Linear resampling and fixed gates are simple on purpose. They would be the first things to replace if a user's setup needs better rejection.

Unit tests for all of this live next to the code in capture.rs, resample.rs, probe.rs and engine.rs.

I'm building Talkaside, a Windows app for job interviews and calls in another language.

Drafted with AI assistance from this project's code and logs.

Where the time goes in a live AI call assistant: my code takes 150 ms, the model takes the rest

devs — Mon, 26 Oct 2026 11:50:43 +0000

I'm building a Windows app that listens to both sides of a call, transcribes it and drafts a short answer while the other person is still talking. People use it for job interviews and for calls in a second language. The first thing I wanted to fix was speed: the answer often arrived after the moment had passed.

Before touching the code, I logged every stage of every answer. This post is what 2,314 completed answers from my own logs say about where the seconds go.

The pipeline

Speech recognition streams partial text, then a final transcript for each utterance.
The client waits briefly for the speaker to finish, then sends a request.
The server builds the prompt: the last 30 messages of the conversation (capped at 12,000 characters), plus the user's CV and the job description.
The provider streams tokens back, and the server relays them to the UI over Server-Sent Events.

Every answer ends with one log line:

[SSE quick] done via groq final_to_request=5ms prepare=78ms ttft=1188ms total=1329ms chunks=62

final_to_request: from the final transcript to the request leaving the client
prepare: server work before the provider call (prompt assembly, context trimming)
ttft: time to the first token from the provider
total: time to the last token

Two answers per question

Each question fires two requests in parallel. The quick slot returns a sentence or two you can say straight away. The detail slot returns a fuller answer to read while you're already talking. The slots can use different providers, so the fast answer never waits for the thorough one.

The numbers

Medians for every slot and provider with at least 50 answers:

Slot / provider	n	prepare	first token	total
quick / Groq	638	234 ms	813 ms	937 ms
quick / Groq, second key	60	203 ms	1,351 ms	1,382 ms
quick / Cloudflare Workers AI	131	203 ms	1,109 ms	1,468 ms
quick / Gemini API	341	94 ms	3,063 ms	3,282 ms
quick / Claude Code CLI	52	148 ms	5,047 ms	7,250 ms
detail / Gemini API	96	125 ms	4,054 ms	5,117 ms
detail / Claude Code CLI	185	109 ms	4,203 ms	7,641 ms
detail / Codex CLI	706	172 ms	7,062 ms	10,188 ms

Across all 2,314 answers, prepare has a median of 157 ms and a 90th percentile of 609 ms. Time to first token ranges from 0.8 s to 7 s depending on the provider.

So the part I was about to optimise is roughly 150 ms. Moving the quick slot from Gemini to Groq saves more than two seconds on every answer, and that's a settings change, not a code change.

Why the CLI rows are slow

The Codex and Claude rows go through the vendors' command-line tools, which lets people use a subscription they already pay for instead of an API key. Each call starts a new CLI session, and most of the wait is on the vendor's side: in a separate test, keeping a Codex process warm barely beat starting a cold one. These work for the detail slot. They're too slow for the quick one.

The 1.2 seconds I added on purpose

In the earliest logs, final_to_request was 4 to 6 ms. In most recent rows it's about 1.2 s, and that delay is deliberate.

Speech recognisers sometimes split one sentence into two finals. Deepgram once sent "Can you tell me a little about your experience?" and then, 329 ms later, "about your experience?" as a separate final. The app drew a second bubble and cancelled the answer it had just started.

There are two fixes in place now:

A tail check: a new final from the same speaker that is the last three or more words of the previous one, arriving within 2.5 s, gets dropped.
A 1.2 s debounce after each final before asking the model, so a question split by a pause becomes one request.

That costs 1.2 s on every answer in exchange for no duplicate or cancelled answers. I haven't settled whether that's the right trade.

Skipping the long answer for small talk

"Thanks, nice to meet you" doesn't need a 100-word answer. The detail request is skipped when an utterance has no question mark, is 12 words or fewer, opens with a greeting or thanks, and contains no question word. That skips a slow request nobody would read.

Speech recognition comes on top

All of the numbers above start at the final transcript. Recognition adds its own delay before that, about a second with streaming cloud STT in my tests. With Groq in the quick slot, that puts the first words of an answer around three seconds after the speaker stops: one second of recognition, 1.2 s of debounce, a fifth of a second of prep and under a second to the first token. Without the debounce it would be closer to two.

What I took from this

Log every stage before optimising. Without a ttft field I'd have spent a week on a part that takes 150 ms.
Put the fastest provider in the slot people wait for, and let the slow, thorough one run in parallel.
Clean up recogniser output before it reaches the model. A split utterance costs a cancelled request and a second answer.

The usual caveats apply. These are medians from one person's usage on one connection. The models behind each provider changed during the period, and prompt size depends on how much CV and job text is loaded.

The app is Talkaside, a Windows interview assistant and live call translator.

Drafted with AI assistance. The numbers come straight from the app's logs.