Speech Studio 0.5.0 Drops Local Multi-Engine TTS+STT on Windows — First-Sound Latency Still Sets the Overlay Floor

On 10 November 2026 Speech Studio hit v0.5.0. It is a free, open-source, fully local desktop studio for Windows 10/11 x64 (NVIDIA 8 GB+ preferred; experimental builds for Apple silicon and Linux). One native executable, no Python runtime. It voices text with six local engines and turns recordings back into text with speaker separation.

The engine table from the release notes is the interesting part for anyone shipping live audio tools:

Engine	Faster than real-time	First sound	Streaming	Download	VRAM
Supertonic 3	168×	103 ms	By phrase	0.1 GB	1.0 GB
Qwen3-TTS	12×	19 ms	As it speaks	2.2 GB	3.1 GB
Qwen3-TTS + ru stress LoRA	10×	26 ms	As it speaks	2.3 GB	3.5 GB
Fish Audio Drama 3	7.0×	326 ms	As it speaks	21.4 GB	~22 GB
Audio8 TTS	5.8×	2.3 s	By phrase	1.5 GB	2.0 GB
Higgs Audio v3	3.4×	—	As it speaks	5.3 GB	~7 GB

Recognition engines (Parakeet, Nemotron, CrisperWhisper via transcribe.cpp) sit beside them. Models download on demand. Everything stays on disk.

That first-sound column is the number that matters once you leave the studio and put the same engines inside a live Windows overlay.

Why first-sound latency dominates the overlay budget

A call helper or interview overlay that also speaks (translate, confirm, read back a summary) has a hard end-to-end ceiling. Users notice anything above ~150–200 ms from “intent formed” to “audio starts.”

Typical Windows budget on a dual-track capture path:

Stage	Cost
WASAPI shared-mode period	5–10 ms
Format convert + lock-free ring	1–3 ms
Lightweight VAD / energy gate	2–5 ms
STT partial (local or hosted)	35–100 ms
Routing / extraction (small model)	40–150 ms
TTS first sound	19–326 ms (engine dependent)
Overlay paint	<16 ms

Supertonic and Qwen3-TTS fit. Drama 3 and Audio8 do not, unless you pre-generate or accept the delay. The studio’s own numbers make the tradeoff explicit: 168× real-time with fixed voices and 103 ms first sound versus higher quality cloning that starts later.

On Windows the capture side is still the floor. Even a 19 ms TTS cannot hide a poorly tuned WASAPI client, a blocking callback, or a single mixed stream that loses speaker identity.

Practical local pipeline notes

Speech Studio demonstrates a working multi-engine host: native process, C ABIs for the heavy models, audio.cpp server for the rest, on-demand downloads, and a voice gate that can start a conversation from your own voice or a wake phrase without a fixed word list.

For an overlay that must stay out of the way of the call app you usually want a thinner cut:

1. Two lock-free SPSC rings (mic + render loopback). No allocation on the audio callback.
2. Consumer thread resamples to 16 kHz mono int16, runs a tiny VAD, and feeds either a streaming WebSocket or a local transcribe.cpp / whisper.cpp session.
3. On finalized or high-confidence partial, a small local or cheap hosted model decides the action (translate, summarize last N, extract names, none).
4. If speech is required, pick the engine whose first-sound number fits the remaining budget. Cache the voice embedding. Stream the first chunks immediately.

Pseudocode for the TTS hop (OpenAI-compatible local endpoint style):

async def speak_if_budget_allows(text: str, remaining_ms: int, voice_id: str):
    # Prefer engines under the remaining budget
    engine = "qwen3-tts" if remaining_ms > 40 else "supertonic"
    # Local server (audio.cpp / llama-server / Ollama-style)
    async with session.post(
        f"http://127.0.0.1:8797/v1/audio/speech",
        json={
            "model": engine,
            "input": text,
            "voice": voice_id,
            "response_format": "pcm",
            "speed": 1.0
        }
    ) as resp:
        async for chunk in resp.content.iter_chunked(4096):
            # Push to WASAPI render as soon as first buffer arrives
            await play_pcm_chunk(chunk)

Tradeoffs that still bite:

• VRAM vs latency. Qwen3-TTS at 3.1 GB is comfortable on an 8 GB card next to the call app. Drama 3 wants most of a 24 GB card. CPU-only paths exist but first-sound jumps.
• Cloning quality vs speed. Zero-shot from a few seconds is excellent for personalization; fixed Supertonic voices are faster and lighter. Interview overlays usually care more about the other party’s language than perfect voice match on the helper.
• DLL hell on Windows. The 0.5.0 notes fixed a LoadLibraryExW collision between Qwen3-TTS and the recognition engine (same-named ggml). Name your own DLLs or load order carefully when you host multiple ggml-based runtimes in one process.
• Streaming vs phrase. “As it speaks” lets you start playback early. “By phrase” is simpler but adds the full generation time before any audio.

What the studio does not change

It does not remove the need for separate tracks. Mixing mic and loopback into one stream still loses who said what and confuses language-specific recognizers. The ring buffers, the VAD, and the careful WASAPI shared-mode setup remain the latency floor. Everything after that—local TTS choice, small routing model, overlay paint—is downstream.

Speech Studio is a useful reference for how to host several open engines in one native Windows process without Python, with on-demand model downloads and measurable first-sound numbers. The numbers themselves tell you which engines can live on the hot path of a real-time overlay and which ones belong in a background or pre-generation step.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.