Most voice assistants on the desktop today feel sluggish. You press a button, wait 2 to 3 seconds for cloud processing, and get a robotic voice response while staring at a static icon.
I wanted something that felt like a true sci-fi command deck—fast, responsive, and visual.
So I spent the last few weeks building J.A.R.V.I.S., a voice-first desktop engineering assistant built with Electron, TypeScript, Three.js, and a low-latency local neural pipeline.
Here is how the architecture works, how I got the voice loop under 400ms, and what I learned along the way.
🔗 Live Website: www.sohan.cfd
📦 Beta Deck: www.sohan.cfd/beta
1. The Core Problem: The Latency Bottleneck
The biggest reason voice assistants feel unnatural is Turn-Taking Latency. Humans converse with an average pause of only 200–300ms.
In a standard voice pipeline:
- Audio recording finishes -> Send entire audio WAV to cloud STT (~800ms)
- Cloud STT returns text -> Send text to LLM (~1000ms)
- LLM returns full answer -> Send to cloud TTS (~800ms)
- Play audio back
Total delay: 2.5 to 3.5 seconds. That's far too slow for an engineering companion.
2. The Solution: Streaming Audio Chunks & Speculative Execution
To get TTFT (Time to First Token) and response time below 400ms, I redesigned the pipeline:
A. Real-Time Streaming Audio Chunker
Instead of waiting for the user to stop speaking before sending audio, audio is captured in 80ms PCM slices. Voice Activity Detection (VAD) monitors silence thresholds locally in the Electron main process.
B. Streaming Synthesis
As soon as the first sentence chunk completes from the reasoning model, it is immediately dispatched to the TTS buffer before the rest of the response finishes generating.
3. The Visuals: 60 FPS Telemetry HUD with Three.js
A voice assistant shouldn't just be an invisible background service. I wanted real visual telemetry to know the system's operational state at a glance:
- Idle: Slowly pulsating obsidian reticles.
- Listening: Real-time waveform frequency spectrum reacting to microphone input.
- Synthesizing: Holographic neural sphere rotating with dynamic cyan shaders.
By offloading the HUD animation to WebGL via Three.js in a separate renderer process, the UI maintains 60 FPS without blocking the audio processing thread in Node.js.
4. Key Lessons Learned
- Keep Heavy Audio Off the Main Thread: Audio buffer underruns cause audible stuttering. Worker threads or dedicated utility processes in Electron are essential.
- Audio Feedback Matters: Immediate subtle audio cues (chimes or reticle animations) within 50ms make any remaining latency feel instantaneous.
- Local-First is the Future: Running local models provides privacy and avoids cloud API outages.
Try it out & Feedback
The project is currently in open beta. You can check out the live site and download the installer here:
- 🌐 Website: https://www.sohan.cfd
- 🧪 Beta Flight Deck: https://www.sohan.cfd/beta
I'd love to hear your thoughts: what features would you want in a desktop voice assistant for coding and system engineering?
Top comments (0)