DEV Community

Adarsh Kant
Adarsh Kant

Posted on

Building Real-Time Voice Forms with Google Gemini API: Architecture & Learnings

#ai #webdev #javascript #productivity

When you want to build voice-input forms that feel responsive and intuitive, the key challenge isn't transcription—modern APIs handle that well. It's latency. Transcription that takes 2 seconds to return feels broken. Transcription that streams back in real-time (200-400ms for first token) feels magical.

This post walks through the architecture we built at Anve Voice Forms to make real-time voice transcription feel fast and seamless in the browser.

The Challenge: Why Basic Transcription APIs Feel Slow

Most voice API approaches work like this:

  1. User speaks for N seconds
  2. Collect all audio
  3. Send entire audio file to API
  4. Wait for transcription response
  5. Display result

Round-trip latency: 2-5 seconds. That's dead time where the user is waiting and nothing is happening.

The better approach is streaming: send audio chunks as they arrive, start processing immediately, and stream back results in real-time.

The Architecture

Here's the high-level flow:

Browser (Frontend)
  Microphone API → WebAudio Processor → WebSocket Client
                                              │ Chunks
                                              ▼
Backend (Node.js/Python)
  WebSocket Server → Audio Processor → Gemini API (Streaming)
                          │
                          ▼
                    Transcript Builder → Browser updates UI
Enter fullscreen mode Exit fullscreen mode

1. Browser-Side Audio Capture 

// Capture audio from microphone
const audioContext = new (window.AudioContext || window.webkitAudioContext)();
const mediaStream = await navigator.mediaDevices.getUserMedia({ audio: true });
const source = audioContext.createMediaStreamAudioSource(mediaStream);

const processor = audioContext.createScriptProcessor(4096, 1, 1);

processor.onaudioprocess = (event) => {
  const audioData = event.inputBuffer.getChannelData(0);
  const pcmData = new Float32Array(audioData);
  const int16Data = float32ToInt16(pcmData);
  socket.emit('audio_chunk', int16Data);
};

source.connect(processor);
processor.connect(audioContext.destination);

function float32ToInt16(float32Array) {
  const int16Array = new Int16Array(float32Array.length);
  for (let i = 0; i < float32Array.length; i++) {
    int16Array[i] = float32Array[i] < 0
      ? float32Array[i] * 0x8000
      : float32Array[i] * 0x7fff;
  }
  return int16Array;
}
Enter fullscreen mode Exit fullscreen mode

Key decisions:

  • 4096 sample chunk size: 93ms at 44.1kHz (good balance between latency and overhead)
  • Int16 encoding: most APIs expect 16-bit PCM audio
  • Send immediately: don't buffer, start streaming as chunks arrive

2. Streaming to Gemini API

This is where real-time transcription happens:

const { GoogleGenerativeAI } = require("@google/generative-ai");
const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);

async function transcribeAudioStream(ws, audioChunks) {
  const model = genAI.getGenerativeModel({ model: "gemini-2.0-flash" });

  const response = await model.generateContentStream({
    contents: [{
      role: "user",
      parts: [
        { inlineData: { mimeType: "audio/mp3", data: audioStream } },
        { text: "Transcribe this audio. Return ONLY the transcription." }
      ]
    }]
  });

  for await (const chunk of response.stream) {
    const text = chunk.text();
    if (text) {
      ws.send(JSON.stringify({
        type: 'partial_transcript',
        text: text,
        timestamp: Date.now()
      }));
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

3. Handling Codec Mismatches

This was our biggest surprise issue. Browsers capture audio as PCM (44.1kHz, 16-bit mono). But APIs have different requirements — some want WAV, some MP3, some raw PCM.

const ffmpeg = require('fluent-ffmpeg');

async function convertAudioCodec(inputBuffer, outputFormat) {
  return new Promise((resolve, reject) => {
    ffmpeg(inputBuffer)
      .format(outputFormat)
      .audioFrequency(16000)
      .audioChannels(1)
      .on('end', () => resolve(outputBuffer))
      .on('error', reject)
      .pipe(outputBuffer);
  });
}
Enter fullscreen mode Exit fullscreen mode

4. Latency Optimization

Real-time means <500ms perception. Our latency breakdown:

  • Browser capture: 93ms (chunk size)
  • Network round-trip: 50ms
  • Gemini processing: 150ms
  • Response streaming: 20ms
  • Total: ~310ms before transcription appears

5. Cost Optimization 

// Don't send silence
function shouldSendChunk(audioData, threshold = 0.01) {
  const rms = Math.sqrt(
    audioData.reduce((sum, s) => sum + s ** 2, 0) / audioData.length
  );
  return rms > threshold;
}
Enter fullscreen mode Exit fullscreen mode

We estimate $0.0005 per form submission at scale.

Lessons Learned

  1. Streaming changes everything. 500ms feels slow. 200ms feels responsive.
  2. Test with real audio. Background noise, accents, quiet voices — test aggressively.
  3. Browser audio APIs are still janky. ScriptProcessorNode is deprecated but most compatible.
  4. Don't ignore codec issues. We lost 2 weeks to garbage transcription from wrong formats.
  5. Frontend UX matters. Debounce updates, show partial results clearly.

Production Stack

  • Frontend: React + WebSocket client
  • Backend: Node.js with ws library
  • API: Google Gemini 2.0 Flash
  • Codec: ffmpeg-wasm (browser) + ffmpeg (backend)
  • Hosting: Render + Cloudflare CDN

Building something with voice? We'd love to hear about it. Drop a comment or check out Anve Voice Forms if you want to see this architecture in action.

—Adarsh, Founder @ Anve Voice Forms

Top comments (0)