Building a Real-Time Desktop Voice Transcriber with Python, React, and WebSockets

Every time I needed to transcribe a meeting recording or generate subtitles for a video, I had to upload files to some cloud service, wait, download the results, and hope my data didn't end up training someone else's model.

So I built my own: a desktop voice transcription app that runs entirely locally, streams audio from microphone or file, and shows results in real-time in a React UI.

Here's the full walkthrough — code included.

---

Architecture Overview

┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌         WebSocket      ┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌┌
∜  React Frontend  ∜ ←←←←←←←←←←←← ∜  Python Backend   ☁ (Tauri Desktop) ☁                     ☁ (FastAPI +       ☁ ☁                 ☁                     ☁  faster-whisper) ☁)⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞⑞                    ≞≞≞≞≞≞≞≞≞≞≞≞≞≞≞≞≞≞≞≞≞≞≞≞≞≞≞```



Why this stack:
- **faster-whisper* is the best local transcription model right now — CTranslate2-optimized, runs on CPU in real-time
- **FastAPI*' + websockets handles streaming audio with minimal latency
- **React + Tauri** gives a native-feeling UI without Electron bloat (~5MB binary vs 150MB)

---

## Step 1: Python Backend (FastAPI + WebSocket)

First, install the dependencies:



```bash
pip install fastapi uvicorn websockets faster-whisper numpy soundfile

The backend listens for incoming audio chunks via WebSocket, accumulates them, and runs transcription in real-time:

from fastapi import FastAPI, WebSocket, WebSocketDisconnect
from faster_whisper import WhisperModel
import numpy as np
import asyncio
import json

app = FastAPI()
model = WhisperModel("base", device="cpu", compute_type="int8")

class TranscriptionSession:
    def __init__(self):
        self.buffer = b""
        self.sample_rate = 16000

    async def transcribe_chunk(self, audio_bytes: bytes) -> str:
        audio_array = np.frombuffer(audio_bytes, dtype=np.float32)
        segments, info = model.transcribe(
            audio_array,
            beam_size=1,
            vad_filter=True
        )
        result = []
        for segment in segments:
            result.append({
                "start": round(segment.start, 2),
                "end": round(segment.end, 2),
                "text": segment.text.strip()
            })
        return result

sessions: dict[str, TranscriptionSession] = {}

@app.websocket("/ws/transcribe/{session_id}")
async def websocket_endpoint(websocket: WebSocket, session_id: str):
    await websocket.accept()
    session = TranscriptionSession()
    sessions[session_id] = session
    try:
        while True:
            data = await websocket.receive_bytes()
            result = await session.transcribe_chunk(data)
            await websocket.send_text(json.dumps({
                "type": "transcription",
                "data": result
            }))
    except WebSocketDisconnect:
        del sessions[session_id]
    except Exception as e:
        await websocket.send_text(json.dumps({
            "type": "error",
            "message": str(e)
        }))

The vad_filter=True is the secret sauce — it skips silence automatically, so you only get transcription when someone is actually speaking.

---

Step 2: React Frontend (Tauri App)

Set up a new Tauri + React project:

npm create tauri-app@latest voice-transcriber -- --template react-ts
cd voice-transcribler
npm install

The React frontend captures microphone input using the Web Audio API and streams it to the backend:

// src/App.tsx
import { useState, useRef, useEffect, useCallback } from 'react';

interface TranscriptionSegment {
  start: number;
  end: number;
  text: string;
}

function App() {
  const [isRecording, setIsRecording] = useState(false);
  const [transcriptions, setTranscriptions] = useState<TranscriptionSegment[]>([]);
  const [status, setStatus] = useState('Disconnected');
  const wsRef = useRef<WebSocket | null>(null);
  const audioCtxRef = useRef<AudioContext | null>(null);
  const streamRef = useRef<MediaStream | null>(null);
  const sessionIdRef = useRef(crypto.randomUUID());

  const connect = useCallback(() => {
    const ws = new WebSocket(`ws://localhost:8000/ws/transcribe/${sessionIdRef.current}`);
    ws.onopen = () => setStatus('Connected');
    ws.onclose = () => setStatus('Disconnected');
    ws.onmessage = (event) => {
      const msg = JSON.parse(event.data);
      if (msg.type === 'transcription') {
        setTranscriptions(prev => [...prev, ...msg.data]);
      } else if (msg.type === 'error') {
        console.error('Server error:', msg.message);
      }
    };
    wsRef.current = ws;
  }, []);

  const startRecording = useCallback(async () => {
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    const audioCtx = new AudioContext({ sampleRate: 16000 });
    const source = audioCtx.createMediaStreamSource(stream);
    const processor = audioCtx.createScriptProcessor(4096, 1, 1);

    processor.onaudioprocess = (event) => {
      if (wsRef.current?.readyState === WebSocket.OPEN) {
        const inputData = event.inputBuffer.getChannelData(0);
        wsRef.current.send(inputData.buffer);
      }
    };

    source.connect(processor);
    processor.connect(audioCtx.destination);
    audioCtxRef.current = audioCtx;
    streamRef.current = stream;
    setIsRecording(true);
  }, []);

  const stopRecording = useCallback(() => {
    audioCtxRef.current?.close();
    streamRef.current?.getTracks().forEach(track => track.stop());
    setIsRecording(false);
  }, []);

  useEffect(() => {
    connect();
    return () => {
      wsRef.current?.close();
      audioCtxRef.current?.close();
    };
  }, [connect]);

  return (
    <div style={{ padding: '2rem', maxWidth: '800px', margin: '0 auto' }}>
      <h1>ø­¦ Voice Transcriber</h1>
      <p>Status: <strong>{status}</strong></p>
      <button onClick={isRecording ? stopRecording : startRecording}
        style={{ background: isRecording ? '#ef4444' : '#22c55e', color: 'white', border: 'none', padding: '12px 24px', borderRadius: '8px', fontSize: '16px', cursor: 'pointer' }}>
        {isRecording ? '⍈ Stop Recording' : '█ Start Recording'}
      </button>
      <div style={{ marginTop: '2rem', background: '#f8f9fa', padding: '1rem', borderRadius: '8px', maxHeight: '400px', overflowY: 'auto' }}>
        {transcriptions.map((seg, i) => (
          <div key={i} style={{ marginBottom: '8px' }}>
            <span style={{ color: '#6b7280', fontSize: '12px' }}>
              [{seg.start.toFixed(1)}s - {seg.end.toFixed(1)}s]
            </span> {seg.text}
          </div>
        ))}
      </div>
    </div>
  );
}

export default App;

---

Step 3: Running IT

Terminal 1 — Start the backend:

cd backend
uvicorn main:app --host 0.0.0.0 --port 8000

Terminal 2 — Start the Tauri dev server:

cd voice-transcriber
npm run tauri dev

You should see the app window open. Click "Start Recording" and speak — transcriptions appear in real-time.

---

Performance Benchmarks

Tested on a mid-range laptop (i5-1135G7, 16GB RAM, no GPU):

Model Size	VRAM	Speed (real-time factor)	Accuracy (WER)
tiny	~1 GB	8x (very fast)	~12%
base	~1.5 GB	4x	~8%
small	~2.5 GB	2x	~5.5%

I use base on CPU — it keeps up with real-time speech comfortably while maintaining decent accuracy.

---

What's Next

This is the MVP. Here's what I'm adding:

1. File drop support — transcribe pre-recorded audio/video files
2. Speaker diarization — identify who said what
3. Export to SRT/VTT — generate subtitle files
4. Screenshot + text overlay — for creating transcribyed video clips (coming in v2)

The full source code is on GitHub if you want to skip the tutorial and just run it:

github.com/yourusername/voice-transcriber (star to follow development)

---

Why This Matters

Cloud transcription services charge $0.006-0.024 per minute. A 1-hour meeting transcription costs $0.36-1.44 every time. With local whisper, the transcription is free, private, and you can process unlimited hours.

The trade-off is setup complexity — but once it's running, it just works. No API keys, no rate limits, no data leaving your machine.

---

Discussion

Have you tried building with whisper locally? What's your setup? I'm particularly interested in:

• Speaker diarization (pyannote vs NVIDIA NeMo)
• Real-time vs batch tradeoffs
• Any creative use cases for local transcription

Drop a comment below ✌／