Capturing the mic and the call as two separate tracks on Windows with cpal and WASAPI loopback

I'm building a Windows app, Talkaside, that transcribes both sides of a call live. The microphone is one side. The other side is whatever the call app plays through the speakers or headphones. Those need to be two separate tracks, because each one gets its own speech-to-text session, its own language and its own speaker label.

This post covers the capture layer, a Rust crate on top of cpal 0.15. It covers how the system audio is tapped, how both tracks end up as mono 16 kHz, and the gating that stops the microphone from re-transcribing the speakers.

Two pipelines, not one mixer

The obvious design is to mix everything into one stream and let the recognizer sort it out. That fails quickly: you lose who said what, and a recognizer tuned for one language sees two.

So the engine starts one pipeline per direction, called outgoing (the microphone) and incoming (the call). Each pipeline owns its own capture stream, its own resampler, its own STT session and its own recording file. When both are requested the engine sets a dual_channel_mode flag, which switches on the cross-track gating described below.

Tapping system audio

On Windows, cpal enables WASAPI loopback when you build an input stream on a render endpoint. There is no separate loopback API to call. The relevant constructor in this repo:

pub fn new_loopback(output_device_name: &str, /* ... */) -> Result<Self> {
    let device = find_output_device(output_device_name)?;
    let default_cfg = device
        .default_output_config()
        .context("Failed to get default output config for loopback capture")?;

    let channels = default_cfg.channels();
    let sample_rate = default_cfg.sample_rate().0;
    let config = StreamConfig {
        channels,
        sample_rate: default_cfg.sample_rate(),
        buffer_size: cpal::BufferSize::Default,
    };
    // ... build_input_stream_{f32,i16,u16}(&device, &config, ...)

Two choices here are deliberate. The stream runs at the device's default config instead of a requested 16 kHz, which avoids negotiating a rate the device may not support. Resampling happens later, on our side. And the sample format is matched at runtime (F32, I16, U16), with a separate callback for each.

The UI stores "system output" as a sentinel device name, __system_output_loopback__. One helper, is_system_loopback_device, is the only branch; the rest of the pipeline is shared.

Finding devices by name

Windows device names are localized strings, and cpal can expose more than one audio host. The lookup tries the default host first, then every other host cpal reports. It matches exactly, then falls back to a normalized substring match: lowercase, punctuation to spaces, whitespace collapsed. A small table maps Russian Windows names (the words for microphone, speakers, headphones, stereo mix) to their English equivalents before comparing, so a setting saved on an English install still resolves on a Russian one.

The audio callback does almost nothing

The callback runs on the driver's thread, so it only downmixes and hands off:

move |data: &[f32], _info: &cpal::InputCallbackInfo| {
    let mono: Vec<f32> = if channels == 1 {
        data.to_vec()
    } else {
        data.chunks(channels as usize)
            .map(|frame| frame.iter().sum::<f32>() / channels as f32)
            .collect()
    };
    let chunk = AudioChunk { samples: mono };
    if let Err(e) = sender.try_send(chunk) {
        note_dropped_chunk(&dropped_chunks);
        debug!("Capture channel full or disconnected: {}", e);
    }
},

It uses try_send into a bounded channel of 512 chunks. If the consumer stalls, the callback never blocks. It bumps an AtomicU64 and moves on. The consumer swaps that counter to zero on every chunk it receives and, when it was nonzero, emits a warning like [incoming] capture queue dropped count=N. Without it, a drop shows up as missing words and nothing in the logs.

Resampling to 16 kHz

Both streaming recognizers the app uses accept 16 kHz PCM, but devices run at 44.1 or 48 kHz. The engine keeps one StreamingResampler per track. It is plain linear interpolation, chosen for being cheap and predictable on speech, with one rule: the position of each output sample is computed from integer counters, never from an accumulated float.

let wanted = self.received as u128 * self.to as u128 / self.from as u128;
while (self.emitted as u128) < wanted {
    let position = self.emitted as u128 * self.from as u128;
    let index = (position / self.to as u128) as u64;
    let fraction = (position % self.to as u128) as f32 / self.to as f32;
    // ... interpolate pending[local] and pending[local + 1]
    self.emitted += 1;
}

Because the output count is a pure function of the input count, the result doesn't depend on how the driver happens to chunk the audio. A test feeds 512,000 samples through six rate pairs (48000 to 16000, 44100 to 16000, 8000 to 16000, 16000 to 16000, 24000 to 44100, 24000 to 48000) in chunks of 1, 127, 512 and 997 samples, and asserts the output is identical to converting the whole buffer at once. Another test pushes 1,000 chunks of 512 samples at 48 kHz and asserts the internal buffer never holds more than 3 samples and that exactly 170,666 samples came out.

The trade-off: linear interpolation without a low-pass filter lets some aliasing through. For speech going into a recognizer that is a reasonable trade. For music it would not be.

Stopping the microphone from transcribing the speakers

Without headphones, the microphone hears the call. Then the same sentence is transcribed twice, once per track, and the second copy is wrong and late. In dual mode the engine gates the microphone while the incoming track is active:

let level = audio_level(samples);
if above_audio_gate(level, OUTGOING_MIC_RMS, OUTGOING_MIC_PEAK) {
    *last_strong_outgoing_audio = Some(Instant::now());
    return None;
}
if let Some(last_seen) = last_strong_outgoing_audio {
    if last_seen.elapsed() <= Duration::from_millis(OUTGOING_MIC_HANGOVER_MS) {
        return None;
    }
}
if monotonic_wall_clock_ms() > incoming_audio_active_until.load(Ordering::SeqCst) {
    return None;
}
if below_audio_gate(level, OUTGOING_BLEED_RMS, OUTGOING_BLEED_PEAK) {
    return Some(AudioDropReason::QuietOutgoing);
}
Some(AudioDropReason::IncomingActive)

The incoming track marks itself active for 1,100 ms whenever a chunk crosses RMS 0.006 or peak 0.025. A microphone chunk passes through if it is clearly louder than bleed (RMS 0.010 or peak 0.035) and keeps passing for 650 ms afterwards, so the user can talk over the other person and the ends of their words aren't clipped. Anything quieter during incoming activity is dropped.

These thresholds are hand-tuned constants, not learned values. They suit typical laptop microphones and speaker volumes. A loud speaker next to a sensitive microphone can still leak through, and a very quiet speaker can be gated away. This is gating, not echo cancellation.

One more detail matters. Dropped chunks are not simply skipped: the engine calls ensure_stt_keepalive, so the STT session is not left idle while the gate is closed. The recorder also sits before the gate, so the saved file keeps the full track. Gating only decides what the recognizer sees, and a muted track is recorded as zeros of the same length so the timeline stays aligned.

Knowing when a track went quiet

A dead device and a quiet room look the same: all zeros. The pipeline logs input level over the first 10 seconds, then watches a rolling 60-second window and warns once if the track stays under the activity gate, resetting when signal returns. A separate idle-only probe opens both devices for 3 to 5 seconds, refuses to run if both selections resolve to the same device, and reports aggregate RMS and peak without keeping any PCM.

Limits

• Loopback is bound to one output device, resolved when the pipeline starts. If that device disappears, the pipeline reports an audio-device-lost event instead of silently following another one.
• Per-application capture isn't used: the loopback includes everything that device plays, including notification sounds.
• Linear resampling and fixed gates are simple on purpose. They would be the first things to replace if a user's setup needs better rejection.

Unit tests for all of this live next to the code in capture.rs, resample.rs, probe.rs and engine.rs.

I'm building Talkaside, a Windows app for job interviews and calls in another language.

Drafted with AI assistance from this project's code and logs.