Cursor's Real-Time Voice Chat with Agents Lands — Desktop Capture Latency Still Sets the Real Budget

On 6 November 2026 Cursor shipped Voice Chat with Agents. The changelog is short: you can start a realtime voice call from the Agents Window, the IDE, the web client, or Cursor for iOS. The voice agent listens, replies, and hands coding work to the agent already in the chat. While that agent runs you can keep talking—ask what changed in a diff, check a run, search earlier messages, or switch workers. The coding agent itself speaks short progress notes and a brief answer when it finishes. The voice layer is Grok Voice; the coding agent keeps whatever model you already selected. Privacy Mode keeps zero data retention.

That is a clean product surface. For anyone building a local Windows overlay that does speech-to-text, short translation, or reply generation on the desktop, the interesting part is not the cloud speech-to-speech model. It is the capture path that still sits between the microphone and the first sample that reaches your process.

Shared-mode WASAPI numbers that never appear in cloud benchmarks

Cursor’s voice path is a remote service. The round-trip includes network, model inference, and synthesis. A local desktop tool that captures microphone or loopback audio, runs a small STT model (GGUF, ONNX, or a quantized Whisper/Parakeet variant), and optionally feeds a short local translation or reply model has a different budget. The first 30–90 ms is almost always spent before the model even sees the buffer.

Typical shared-mode WASAPI numbers on a modern laptop:

• Device + driver buffer: 10–20 ms
• Windows Audio Engine mix + any communications enhancement or spatial APO: 10–30 ms
• Your capture client period: another 10–40 ms before you see a stable buffer

That is already 30–90 ms from acoustic input to the first sample in process memory. Exclusive mode can shrink the engine contribution by bypassing the shared mixer and APOs, but then you fight device exclusivity and the rest of the desktop. Recent Coval measurements put Qwen3-ASR 1.7B variants around 35 ms time-to-final segment and Soniox STT RT v5 at 47 ms—those numbers assume the audio has already left the capture device.

If your local model finishes inside the remaining budget, end-to-end from speech end to on-screen text can stay under a couple of hundred milliseconds without ever shipping the audio off the machine. The Cursor voice chat (or any cloud full-duplex path) hides a lot of that cost behind the network and a GPU farm. The desktop capture path does not.

Measuring the real capture latency

Do not trust the buffer size you requested. Open a shared-mode capture client, play a short impulse or click through the speakers (or use a known loopback), and timestamp the first non-zero sample that reaches your process. The number is almost always higher than the period you set.

A minimal sketch with the Windows Audio Session API looks like this (error handling omitted):

IMMDeviceEnumerator* enumerator = nullptr;
CoCreateInstance(__uuidof(MMDeviceEnumerator), nullptr, CLSCTX_ALL,
                 __uuidof(IMMDeviceEnumerator), (void**)&enumerator);

IMMDevice* device = nullptr;
enumerator->GetDefaultAudioEndpoint(eCapture, eConsole, &device);

IAudioClient* client = nullptr;
device->Activate(__uuidof(IAudioClient), CLSCTX_ALL, nullptr, (void**)&client);

WAVEFORMATEX* format = nullptr;
client->GetMixFormat(&format);

REFERENCE_TIME hnsRequested = 10 * 10000; // 10 ms in 100-ns units
client->Initialize(AUDCLNT_SHAREMODE_SHARED, 0, hnsRequested, 0, format, nullptr);

IAudioCaptureClient* capture = nullptr;
client->GetService(__uuidof(IAudioCaptureClient), (void**)&capture);

client->Start();

// In the callback or polling loop: GetBuffer, timestamp, ReleaseBuffer
// Compare against a high-resolution clock that was started when the impulse was played.

Run the same measurement in exclusive mode (AUDCLNT_SHAREMODE_EXCLUSIVE) if the device supports it. On many Realtek and USB headsets the exclusive path drops 15–40 ms. The tradeoff is that other applications can no longer use the endpoint until you release it. For an overlay that only needs the mic while a call or interview is active, that is often acceptable. For a tool that must coexist with the browser or a meeting client, you stay in shared mode and live with the engine cost.

Architecture notes for a local overlay

A practical Windows pipeline that stays under the human turn-taking gap (~200 ms) usually looks like this:

1. WASAPI shared or exclusive capture → small ring buffer in the process.
2. Lightweight VAD or energy gate so you do not run the STT model on silence.
3. Local STT (Qwen3-ASR 1.7B Q4_K_M GGUF at ~1.3 GB, or a smaller Parakeet variant) that finishes in the remaining budget.
4. Optional short local translation or reply model (another GGUF) only on the final segment.
5. Overlay render on a separate thread so the UI never blocks the audio callback.

The idle-compute tricks that papers like Speculative Listening (arXiv:2611.06982) exploit—drafting the next chunk under a silence hypothesis during the listen window—are useful once the audio is already in memory. They do not move the capture-side cost. The same is true for Cursor’s voice agent: the model can be fast, but the desktop still has to deliver clean samples on time.

Practical controls that actually move the number

• Prefer a small local STT model that finishes inside the remaining budget rather than a cloud endpoint whose time-to-final looks better on paper but adds 80–150 ms of network.
• Keep the credential and config boundary inside the process or a DPAPI-protected store. Nothing that looks like a settings file next to source code should ever contain a live key or a shell command that embeds one.
• If you later add a speculative generation step for short replies, draft only under silence or a simple VAD hypothesis and verify against the real audio. The 95 % identical-chunk rates reported in recent full-duplex papers are real, but they assume the audio has already arrived.
• Measure end-to-end from acoustic event to on-screen text, not model latency alone. The Windows Audio Engine mix and any communications enhancement are part of the number.

Cursor’s voice chat shows that speech-to-speech with coding agents is now a product surface. The same interaction pattern is useful on the desktop for interviews, calls in another language, or any workflow where typing breaks flow. The difference is that a local process owns the capture path and can keep the entire loop under the user’s control. The latency budget still starts at the WASAPI client, not at the model.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.