DEV Community: Konstantin

How we calibrated a Synthetic Focus Group from 'this looks great!' to 93% accuracy

Konstantin — Tue, 21 Apr 2026 13:00:00 +0000

When we shipped the first version of GoNoGo's Synthetic Focus Group (SFG), every persona loved everything.

The setup: a founder finishes a 30-minute voice discovery interview about their idea. From that conversation plus a stack of insights we'd scraped for the niche, we spin up five AI personas — a CTO, a budget-conscious shopper, a skeptic, an early adopter, a casual user — and ask the panel to react to the founder's two value-prop variants and a pricing test. All five voted the same way on every variant. All five "would maybe buy." Slightly different wording. Same vibe.

The problem is obvious in retrospect: we'd built a confirmation engine, not a focus group.

This is the story of the next six months — what broke, what we tried, what stuck. By the end we hit 93% predicted-vs-real accuracy across 16 niches with a 95% CI of 91.4–94.6%. Here's how.

📌 TL;DR — If you're building anything with synthetic personas, three things matter more than the rest: (1) generate persona grievances from real user data, weighted 3× over LLM-imagined ones, (2) tune sampling temperature per archetype (skeptics ≠ early adopters), (3) shuffle variant labels per persona to kill position bias. We measured the third one alone shaving 14 percentage points off label bias.

If you missed the first two posts in this series: we built A³ by accident while doing marketing and then solved sub-500ms voice latency the hard way. This post is about what happens after the voice interview ends and we have to actually validate the idea.

What SFG is actually for (and what it isn't)

Before the engineering, the use case.

A founder finishes the voice discovery and now has a stack of decisions to make: which value prop leads, what to charge, which marketing claims hold up, which feature to build first, which segment to target. Traditionally these get answered by (a) gut feel, (b) asking 5 friends, or (c) running a real survey that takes weeks and costs money to recruit the right respondents.

The SFG sits in between. It's a panel of synthetic respondents, each modelled on a specific archetype-with-real-grievances, that the founder can re-run as many times as they want against any decision they need to make — A/B variants, pricing, claims, positioning, segments.

What it gives you:

A vote with reasoning, not a number — "the Skeptic rejected variant B because pricing felt anchored to a tier she doesn't need"
Disagreement between archetypes — surfacing the trade-off you were about to ignore
Reproducible runs — same insights → same panel → same decision logic. You can re-test the same idea after changing one word in the headline

⚠️ What SFG is NOT. It is not a fortune teller. It does not predict whether your startup will succeed. It does not predict what the market will do. It models the decision behavior of a specific archetype, given the frustrations and goals we've sourced for that archetype from real public data. That's a narrower claim than "predict the future" — and a much more honest one.

When we say "93% accuracy" later in this post, that's what's being measured: how closely a synthesized archetype's modelled behavior matches the observed behavior of real users in that archetype, on data the model didn't see during synthesis. Not pre-cognition. Behavioral fidelity.

That distinction matters because it tells you what the SFG is good for (decision-stage trade-offs, claim stress-tests, segment fit, pricing) and what it's bad for (predicting macro-market outcomes, novel categories with no public user data, regulated industries where the data isn't there).

The three things that broke our personas

After watching ~50 sessions where personas all said "great idea!" we noticed three failure modes:

Personas had no real grievances. They were generated from the LLM's vague prior of "what a CTO might say." So a CTO persona evaluating a B2B SaaS would just... vibe. No specific scar tissue, no real pain.
Sampling temperature was uniform. Skeptics rolled the same temperature (0.7) as early adopters. Skeptics weren't actually skeptical — they were just slightly less enthusiastic.
Variant labels biased everything. "Option A" reliably won over "Option B" — classic position bias. Personas were anchoring on label, not content.

We fixed each one. Here's how.

Fix #1: Personas built from real grievances, not templates

The base persona-generation algorithm now does this:

Niche detection. A small LLM classifier maps the project to one of 16 niches (B2B SaaS, marketplace, dev tools, ecommerce, hardware, fitness, content, freelancers, ...). Each niche has a different archetype pool.
Insight collection. We pull real posts from Reddit, HackerNews, ProductHunt, G2, app store reviews — anywhere the niche's actual users complain. Typical project gets 100–300 raw insights.
Per-persona synthesis. For each archetype slot (3–5 per project), we sample 3–8 frustrations and 3–8 goals directly from the real insights for that persona's likely demographic.

The critical line in persona_builder.py:

# Real frustrations from source data weighted 3x over LLM-generated ones
weighted_frustrations = (
    real_frustrations * 3 +
    llm_inferred_frustrations * 1
)

That 3× multiplier is the entire difference between a persona who says "I'd want better onboarding" (LLM generic) and one who says "I bounced from the last 4 tools because none of them imported my Notion docs without breaking nested toggles" (real Reddit thread).

💬 The 3× weight on real frustrations is the cheapest, highest-leverage change in the whole pipeline. Without it, you're just paraphrasing the model's prior beliefs back to the founder.

Each persona also carries up to 5 verbatim quotes from the source data, plus a richness score (0–1) so the orchestrator can flag thin personas before they pollute results. Average richness when >100 insights are available: 0.85+.

Takeaway: persona realism is upstream of every other decision. If your input data is "what an LLM thinks a CTO sounds like," everything downstream is fan-fiction.

Fix #2: Temperature tuned per archetype

Behavioral diversity isn't a prompt problem — it's a sampling problem.

We tuned temperature per archetype in get_temperature():

ARCHETYPE_TEMPERATURE = {
    "EARLY_ADOPTER":     0.9,   # impulsive, willing to leap
    "CASUAL":            0.7,
    "MAINSTREAM":        0.6,
    "PRAGMATIST":        0.5,   # analytical, predictable
    "CTO":               0.4,
    "CFO":               0.4,
    "SKEPTIC":           0.5,   # rigid, negative-biased
    "BUDGET_CONSCIOUS":  0.5,
    # ... 15 archetypes total
}

This alone meaningfully shifted distributions. Skeptics started landing in the 4–6 appeal range by default. Early adopters jumped to 7–9. Pragmatists stayed in the 5–7 band where they belonged.

We also embedded cognitive bias hints directly into each archetype's system prompt. Pragmatists get explicit "status quo bias" framing. Skeptics get "negativity bias" framing. CFOs get loss-aversion phrasing.

The personas didn't just sound different — they actually disagreed with each other.

Takeaway: if every persona is sampled at the same temperature, you're running the same character five times in different costumes.

Fix #3: Variant shuffling

Stupid, easy, huge:

# Shuffle variants per-persona to neutralize position bias
variant_labels = ["Option 1", "Option 2", "Option 3"]
random.shuffle(variant_labels)

For an A/B/C test with 5 personas, each persona sees the variants in a different random order under neutral labels. Position effects average out across the panel.

We measured this. Before shuffling: Option A won 64% of two-variant tests across our calibration set. After shuffling: 50.3% / 49.7%. The label was carrying a 14-point bias.

⚠️ Position bias in LLM panels is real and large. If you're not shuffling labels, your A/B "winners" are partially a measurement of which slot you put them in.

Takeaway: before tuning anything sophisticated, audit for the dumb biases first. They cost you 14 points and a random.shuffle() call to fix.

By the numbers

A snapshot of where the system landed after the calibration pass:


Niches calibrated	16
Tagged insights in calibration set	2,069
Train/test split	70 / 30 per niche
Overall accuracy	93.1% (95% CI 91.4–94.6)
Decision match rate	4 / 4
Personas generated per project	3–5
Average persona richness (>100 insights)	0.85+
Position-bias reduction from shuffling	14 percentage points
Real-grievance weighting over LLM	3×
Archetypes available	15 (across B2B, consumer, marketplace)

We needed to know if any of this was working. So we built a calibration suite.

But first — what are we actually measuring?

We are not measuring "did SFG predict whether the product succeeded." We're measuring something narrower and more testable: given a known archetype and a known set of grievances, does the synthesized persona produce the same pattern of pain points, needs, sentiment, and decisions that the real users in that archetype produced — on data the model didn't see?

If yes, the persona is a faithful behavioral model of its archetype. That's what we calibrated against.

The setup: 2,069 manually-tagged insights across 16 niches, each with known ground-truth pain points, needs, sentiment distribution, and the decision a real founder would have arrived at when looking at the full dataset.

We split 70/30 — synthesize personas using 70% of insights per niche, then ask each persona to characterize the held-out 30% (without ever seeing it). Compare the persona's response to ground truth across 5 weighted dimensions:

Dimension	Metric	Weight
Pain point overlap	Semantic Jaccard (threshold 0.53)	0.30
Pain point ranking	Spearman's ρ	0.15
Needs overlap	Semantic Jaccard	0.25
Sentiment distribution	1 − √JSD	0.20
Language similarity	Cosine of embeddings	0.10

Final score across the 16 niches: 93.1% behavioral fidelity, 95% CI 91.4–94.6%. Best-performing niches: content tools (93.2%), freelancers (92.7%), fitness (90.7%).

Decision match rate (does the synthesized panel reach the same go/no-go verdict as the held-out real data on 4 axes — concern, need, verdict, recommendations): 4/4 across the calibration set.

To restate what that number means: when we ask a synthesized archetype to characterize a problem space using only the 70% it was built from, its description of pain points, needs, sentiment, and recommended decisions matches the description that real users in that archetype produced (on the held-out 30%) at 93% similarity, on average, across 16 niches. Not "predicts the future at 93%." Reproduces archetype behavior at 93%.

The most valuable thing this gave us wasn't the headline number. It was the per-niche breakdown — we could see which archetype pools were weak, which niches needed more insight sources, which prompts were drifting.

🎯 The headline accuracy number is for marketing. The per-niche breakdown is for engineering. Build both.

How an A/B test actually runs

When a founder gives us two landing page variants:

Generate panel — 3–5 personas synthesized from the project's collected insights (already done at discovery time).
Per-persona evaluation in parallel — each persona sees all variants in one prompt, with shuffled labels.
Structured response — for each variant: appeal score (1–10), willingness (would_buy / might_buy / would_not_buy), pros, cons, 2–6 sentence reasoning.
Round 2 — panel discussion — personas react to each other's reasoning. This is where the interesting stuff happens. The skeptic challenges the early adopter. Scores shift. Sometimes the panel realigns entirely.
Aggregate — winner by win count first, average appeal as tiebreak.

The output isn't just a winner. It's a transcript a founder can actually read — with reasoning that maps to specific frustrations from real users.

What else the SFG can do

Once we had calibrated personas, A/B testing turned out to be the smallest use case. The same panel can run:

Claim validation — paste 1–10 marketing claims, each persona votes agree / disagree with reasoning. Surfaces which claims a real audience would call BS on.
Pricing tests — test 3+ price points, get per-persona perceived value and conversion likelihood.
Adaptive hypothesis generation — auto-generates 5–6 testable hypotheses covering problem fit, segment fit, behavior change, switching costs, pricing.
Early adopter lead extraction — pulls 20 real handles from the source insights — actual people who described the exact problem you're solving. Not synthetic. Outreach list.

There's also a separate Reality Check feature that flips the comparison around: it lets you run a real human survey, then dual-scores the SFG prediction against the real responses. That's how we keep the 93% number honest as the model evolves.

What still doesn't work

A few things I'm still not satisfied with:

Single-model persona reasoning. Persona inference currently runs on one frontier-class LLM. We cross-verify factual claims across multiple providers in a separate feature, but the persona reasoning itself is single-backbone. That's a known shared-blind-spot risk we want to address — multi-model panels are on the roadmap.
No benchmarking against traditional focus groups. Only against holdout real-user data. Comparing AI personas to a real moderated focus group with 8 humans is the obvious next benchmark, and it's expensive enough that I keep deferring it.
Niches we don't have insight sources for (regulated industries mostly) drop to ~75% accuracy. The whole approach falls apart when you can't pull real user grievances from somewhere public.

🚧 We're not done. The 93% number is a calibration milestone, not a verdict. Anyone who tells you their AI focus group has solved the problem is selling.

Try it

The full Synthetic Focus Group lives inside gonogo.team. The free tier gives you 3 projects with the voice Discovery agent — enough to feel out whether the methodology makes sense for your idea before unlocking the full multi-agent pipeline (which is where SFG, A/B testing, pricing tests and the rest live, behind a one-time per-project credit — no subscription).

If you build something with synthetic personas yourself — the three things that mattered most for us, ranked: (1) real grievances over LLM templates (3× weight), (2) per-archetype temperature, (3) variant shuffling. Without all three, you'll just keep getting "this looks great!" forever.

Comments and corrections welcome — especially if you've benchmarked AI personas against real focus groups, I'd love to compare notes.

We Were Building Marketing for a Startup. We Accidentally Built an A3

Konstantin — Thu, 16 Apr 2026 06:22:44 +0000

We build GoNoGo — a platform where founders validate their ideas through live voice interviews with a synthetic focus group, competitor analysis, market sizing, and a dozen other tools that Anna will tell you about.

We'll get to Anna

The product was ready. Time for marketing. We did everything by the book: landing page, explainer video, banners, social posts.

The result? Standard tools delivered standard results. We expected more.

Banners get lost in the noise. Videos get skipped. Landing pages get scanned in half a second. The impression → click → signup funnel leaks at every step. Not because the product is bad. Because the format is dead.

What if advertising stopped showing — and started selling?

Not a Video. Not a Banner. Not a Chatbot

Meet Anna

This is not a recording. Anna is live right now. Press Start, allow your microphone — and ask her anything about GoNoGo. She'll answer with her voice, show you slides with real data, compare competitors.

Right here. In this article. No redirects.

Try speaking to her in Japanese, Arabic, or Spanish — she'll switch on the fly.

A³ — Autonomous Advertising Ambassador
Three letters. Three words. Not one wasted.

Autonomous — works on her own. No scripts, no decision trees. Understands context, improvises, adapts to the person she's talking to. In any language.

Advertising — lives where the audience is. Not on your website where you still need to drive traffic. In the feed, in an article, in a post.

Ambassador — represents your brand. Doesn't answer FAQs from a corner widget — she greets, explains, demonstrates, persuades. Like your best salesperson who knows the product by heart.

The difference between A³ and a chatbot is the difference between a live salesperson and a "FAQ" sign by the door.

Case 1: A Pocket Marketer for Tech Products

Anna is already live. Right now she's embedded in a post on X, in this Dev.to article, and soon on Medium.

What she does:

Explains the product by voice — differently for each person, depending on their questions
Generates analytics and slides in real time
Generates contextual CTAs during the conversation — links, sign-up prompts, demo redirects — based on what you just discussed, not a static button
Compares with competitors if you ask
Handles objections — not from a template, but in conversation
Speaks the listener's language — switches on the fly, no configuration
Not one video for a million views. A million unique conversations.

Case 2: A Welcomer in Physical Retail

Same architecture, different context. A screen at the store entrance. A customer walks up:

"Do you have the iPhone 16 Pro in black?"
→ API call to inventory → "Three in stock. Want to see the specs?"
"Compare it with the Samsung S25"
→ comparison table generated on the fly
"Order it for delivery tomorrow"
→ order placed via API, no cashier needed

Not a chatbot on a website. A voice interface on location, with live access to store data.

A tourist walks into a store in Tel Aviv and asks in Japanese — A³ answers in Japanese. No switching, no settings, no language barrier.

First Reaction

We recently demoed Anna to the owner of a retail chain. We adapted the demo to his inventory — stock checks, product comparisons, order placement.

His first question:

How many response variations did you pre-record?
We explained — none. Everything is generated in real time.
His second question:
No, seriously — how many?
We're now in partnership talks.

What Doesn't Work (Yet)

We believe in honesty more than hype.

Dev.to — fully functional widget. You just saw it.

X (Twitter) — Player Card shows the widget in the feed, but the platform blocks microphone access inside the iframe. We solved this with a popup — the browser version is fully functional.

Other platforms — some strip permissions in sandboxed iframes. We're working on a universal fallback.

These are platform limitations, not technology limitations. A³ works anywhere there's a WebSocket and a microphone.

What's Next

We're building a constructor — a platform where any business can create their own A³. Connect your data, customize voice and visuals, embed anywhere. No code required.

Anna is the first proof of concept.

Talk to her: A³ for Team GoNoGO

A³ is a GoNoGo technology. Provisional patent filed

I Built a Voice AI with Sub-500ms Latency. Here's the Echo Cancellation Problem Nobody Talks About

Konstantin — Sun, 05 Apr 2026 08:50:00 +0000

When I started building GoNoGo.team — a platform where AI agents interview founders by voice to validate startup ideas — I thought the hard part would be the AI reasoning. The multi-agent orchestration. The 40+ function-calling tools.

I was wrong.

The hard part was echo. Specifically: how do you stop an AI agent from hearing itself talk, freaking out, and interrupting its own sentence?

After 500+ voice sessions and too many late nights staring at RMS waveforms, here's what I actually learned.

The Setup: Speech-to-Speech, Not STT → LLM → TTS

GoNoGo runs on Gemini 2.5 Flash Live API — a true speech-to-speech pipeline. There's no intermediate transcription step, no text-to-speech synthesis layer bolted on afterward. Audio goes in, audio comes out. Direct.

This is important because it changes everything about how you handle audio on the client. You're not working with text buffers. You're working with raw PCM, 16kHz input from the browser mic, 24kHz output from the agent voice. Base64-encoded over WebSocket.

The browser capture side looks roughly like this:

// ScriptProcessorNode in browser — 512-sample chunks (~32ms each)
const scriptProcessor = audioContext.createScriptProcessor(512, 1, 1);

scriptProcessor.onaudioprocess = (event) => {
  const inputBuffer = event.inputBuffer.getChannelData(0);

  // Calculate RMS for VAD
  const rms = Math.sqrt(
    inputBuffer.reduce((sum, sample) => sum + sample * sample, 0) / inputBuffer.length
  );

  // VAD threshold: 0.05 RMS
  if (rms < VAD_THRESHOLD) return;

  // Convert Float32 PCM to Int16
  const int16Buffer = new Int16Array(inputBuffer.length);
  for (let i = 0; i < inputBuffer.length; i++) {
    int16Buffer[i] = Math.max(-32768, Math.min(32767, inputBuffer[i] * 32768));
  }

  // Base64 encode and send over WebSocket
  const base64Audio = btoa(String.fromCharCode(...new Uint8Array(int16Buffer.buffer)));
  ws.send(JSON.stringify({ type: 'audio_chunk', data: base64Audio }));
};

Simple enough. Until the AI starts talking.

The Echo Problem (And Why Browser AEC Isn't Enough)

Browsers have built-in acoustic echo cancellation. You enable it when you call getUserMedia:

const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,
    noiseSuppression: true,
    autoGainControl: true
  }
});

This works great for video calls between humans. It was designed for that. But it has a fundamental assumption baked in: the "far end" audio is coming through a <audio> element or Web Audio API that the browser knows about.

When you're playing 24kHz PCM chunks from a WebSocket, decoded manually and scheduled through AudioContext buffers? The browser's AEC has no idea that audio exists. It can't cancel what it can't see.

So your AI agent starts speaking. The microphone picks up the speaker output. The agent hears itself. In the best case, it gets confused and repeats something. In the worst case — and this happened constantly in early builds — you get a feedback loop where the agent interrupts itself mid-sentence, hears the interruption, tries to respond to it, hears that, and the whole session collapses.

I called these 1011 disconnects, because that was the WebSocket close code I kept seeing in logs.

The Two-Tier RMS Gate

The fix is a two-tier RMS (Root Mean Square) gate on the audio capture side. The idea is simple: measure the loudness of what the mic is picking up, and if it's probably just the speaker playing back, don't send it.

But "simple" hides a lot of edge cases.

Tier 1: Hard suppress during agent speech

While the agent is actively speaking, I track that state server-side and send it to the client. During this window, incoming audio is suppressed entirely — no chunks sent to Gemini.

let agentSpeaking = false;
let cooldownTimer: ReturnType<typeof setTimeout> | null = null;
const COOLDOWN_MS = 1500;
const COOLDOWN_THRESHOLD = 0.03; // Higher threshold during cooldown
const NORMAL_THRESHOLD = 0.05;   // Normal VAD threshold

// Called when agent audio stream starts/stops
function setAgentSpeakingState(speaking: boolean) {
  if (speaking) {
    agentSpeaking = true;
    if (cooldownTimer) clearTimeout(cooldownTimer);
  } else {
    agentSpeaking = false;
    // Start cooldown period
    cooldownTimer = setTimeout(() => {
      cooldownTimer = null;
    }, COOLDOWN_MS);
  }
}

function shouldSendAudioChunk(rms: number): boolean {
  if (agentSpeaking) return false; // Hard suppress

  if (cooldownTimer !== null) {
    // In cooldown: use higher threshold
    return rms > COOLDOWN_THRESHOLD;
  }

  return rms > NORMAL_THRESHOLD;
}

Tier 2: The 1.5-second cooldown

This is the part that took me longest to figure out. When the agent stops talking, there's still speaker resonance in the room. The RMS of captured audio doesn't drop to zero immediately — it decays. The background noise in a typical home office sits at 0.01–0.02 RMS. But for 1-2 seconds after playback stops, you're seeing 0.025–0.04 RMS — above the normal VAD threshold.

The cooldown period uses a higher threshold (0.03 vs 0.05) for 1.5 seconds after agent speech ends. This catches the decay without cutting off a founder who immediately starts talking back.

Was this threshold tuned empirically? Absolutely. I spent days listening to session replays measuring exactly how fast room resonance decays in different mic setups.

Session Resumption: The Other Half of the Problem

Echo cancellation solved the quality problem. Session resumption solved the reliability problem.

Gemini Live sessions drop. Network hiccups, mobile handoffs, Chrome deciding to do something aggressive with memory — connections fail. Early on, a dropped connection meant starting the entire 30-minute interview over. Founders would ragequit. I would understand completely.

The fix: store session handles in Firestore and resume on reconnect.

# FastAPI backend — session management
from google.genai.live import AsyncSession
from firebase_admin import firestore

async def get_or_create_session(
    project_id: str, 
    user_id: str
) -> tuple[AsyncSession, bool]:
    db = firestore.client()
    session_ref = db.collection('sessions').document(f'{user_id}_{project_id}')
    session_doc = session_ref.get()

    if session_doc.exists:
        session_data = session_doc.to_dict()
        handle = session_data.get('resumption_handle')

        if handle:
            try:
                # Attempt resume — Gemini picks up exactly where it left off
                session = await resume_gemini_session(handle)
                return session, True  # resumed=True
            except Exception:
                pass  # Fall through to new session

    # Create new session
    session = await create_gemini_session(project_id)
    session_ref.set({
        'created_at': firestore.SERVER_TIMESTAMP,
        'project_id': project_id
    })
    return session, False  # resumed=False

async def store_resumption_handle(user_id: str, project_id: str, handle: str):
    db = firestore.client()
    session_ref = db.collection('sessions').document(f'{user_id}_{project_id}')
    session_ref.update({'resumption_handle': handle})

When a session resumes, Gemini restores full context — every tool call result, every piece of market research, every persona in the synthetic focus group. The founder reconnects and the agent says "Sorry about that, where were we?" and genuinely knows where you were.

The Filler Audio Problem

One more thing nobody talks about: what do you play while the AI is thinking?

Gemini 2.5 Flash is fast. 300-500ms end-to-end is genuinely fast. But when the agent is executing a tool call — crawling a competitor site with Playwright, running Reddit scraping, calculating unit economics — you can have 3-8 second gaps.

Silence in a voice conversation feels broken. Users assume the connection dropped.

Solution: pre-computed filler audio. Short phrases like "one moment please" or "let me look that up" in 17 languages, stored as PCM chunks, played when tool execution exceeds ~800ms. The agent is triggered via text signal (not proactive_audio, which had a regression that caused double-playback — disabled entirely, use text triggers instead).

This sounds trivial. It removed about 40% of "the app is broken" support messages.

What I'd Do Differently

Start with the echo gate, not the AI logic. I spent weeks building beautiful multi-agent orchestration before I could demo it reliably. Wrong order.
Instrument RMS values from day one. Log them. Every session. You can't tune what you can't see.
Test on bad hardware. My dev setup has a good mic with physical distance from speakers. Most users have laptop mics 30cm from laptop speakers. Build for that.
Mobile is a different planet. iOS Safari handles AudioContext lifecycle in ways that will make you question your career choices. But that's an article for another day.

The Result

After solving these problems — the two-tier RMS gate, the 1.5s cooldown, the session resumption, the filler audio — GoNoGo runs 15-45 minute voice sessions with real founders, across 21 languages, with 3 AI agents handing off to each other mid-conversation. The 1011 disconnects essentially disappeared.

The voice infrastructure became invisible, which is exactly what it should be.

If you're building anything with browser mic + real-time AI audio: what's been your biggest challenge? I'm genuinely curious whether the echo problem is universal or whether I was doing something particularly wrong early on. Drop it in the comments.

How I Built a Real-Time Voice AI Interview System with Gemini Live API and WebSockets (and What Almost Broke Me)

Konstantin — Sat, 04 Apr 2026 15:45:52 +0000

When I started building GoNoGo.team -- a platform that uses AI to validate startup ideas through voice interviews -- I thought the hardest part would be the business logic. Turns out, the hardest part was keeping a duplex audio stream alive across three layers of abstraction without everything falling apart.

This is a technical post-mortem of the voice AI system I built solo. I'll cover the architecture, the ugly edge cases, and the specific patterns that finally made it stable enough to run 500+ validation interviews.

The Core Problem: Bidirectional Audio at Low Latency

The concept: a founder speaks, Gemini listens and responds with follow-up questions, in real time. No STT/TTS pipeline -- direct speech-to-speech using Gemini Live API (native audio). The pipeline looks like this:

Browser Mic -> ScriptProcessor (16kHz PCM) -> WebSocket (base64) -> Python FastAPI -> Gemini Live API
                                                                                      v
Browser Speaker <- AudioContext (24kHz PCM) <- WebSocket (base64) <- Audio Chunks <------+

Every arrow in that diagram is a potential failure point. And in production, every single one of them failed at least once.

Step 1: Capturing Audio in the Browser

The browser side uses a ScriptProcessorNode (yes, it's deprecated -- but AudioWorklet adds latency we can't afford for real-time conversation). We capture 16kHz mono PCM in 512-sample chunks -- roughly 32ms per chunk.

// Audio capture setup (simplified from useAudioInput.ts)
const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,
    noiseSuppression: true,
    autoGainControl: true,
    sampleRate: 16000,
  }
});

const audioContext = new AudioContext({ sampleRate: 16000 });
const source = audioContext.createMediaStreamSource(stream);
const processor = audioContext.createScriptProcessor(512, 1, 1);

// Analyser for RMS-based voice activity detection
const analyser = audioContext.createAnalyser();
source.connect(analyser);
analyser.connect(processor);

processor.onaudioprocess = (event) => {
  const pcmData = event.inputBuffer.getChannelData(0);
  const rms = Math.sqrt(
    pcmData.reduce((sum, x) => sum + x * x, 0) / pcmData.length
  );

  // VAD gate: only send if voice detected (RMS > 0.05)
  if (rms > 0.05 && ws.readyState === WebSocket.OPEN) {
    // Convert Float32 to Int16 PCM, then base64 encode
    const int16 = new Int16Array(pcmData.length);
    for (let i = 0; i < pcmData.length; i++) {
      int16[i] = Math.max(-32768, Math.min(32767, pcmData[i] * 32768));
    }
    ws.send(JSON.stringify({
      type: "audio",
      data: btoa(String.fromCharCode(...new Uint8Array(int16.buffer)))
    }));
  }
};

The 32ms chunk interval was a hard-won choice. It gives Gemini enough data per packet to process efficiently while keeping perceived latency under 300ms end-to-end. The VAD threshold of 0.05 RMS filters out background noise without clipping soft speech.

Step 2: The Python Backend (FastAPI + WebSockets)

The backend is Python FastAPI, deployed on Google Cloud Run. Python was the right call because Gemini's client libraries are Python-first, and the entire analysis pipeline (market research, competitor scraping with Playwright, report generation) lives in the same codebase.

# WebSocket handler (simplified from server.py)
@app.websocket("/ws_live")
async def websocket_live(ws: WebSocket):
    await ws.accept()
    session = GeminiLiveSession(model="gemini-2.5-flash-exp")

    async def forward_to_gemini():
        # Browser audio -> Gemini
        async for msg in ws.iter_json():
            if msg["type"] == "audio":
                pcm_bytes = base64.b64decode(msg["data"])
                await session.send_audio(pcm_bytes)  # 16kHz PCM

    async def forward_to_browser():
        # Gemini audio -> Browser
        async for event in session.events():
            if event.type == "audio":
                # Gemini returns 24kHz PCM
                chunk_b64 = base64.b64encode(event.data).decode()
                await ws.send_json({
                    "type": "audio",
                    "data": chunk_b64
                })
            elif event.type == "tool_call":
                result = await execute_tool(event)
                await session.send_tool_response(result)

    await asyncio.gather(forward_to_gemini(), forward_to_browser())

The asymmetric sample rates (16kHz in, 24kHz out) aren't a mistake -- Gemini natively outputs at 24kHz, and downsampling would lose audio quality. The browser's AudioContext handles the sample rate mismatch transparently.

The Echo Cancellation Problem

Gemini hears its own output through the user's speakers and tries to respond to itself. Browser-level echo cancellation (echoCancellation: true) handles most cases, but not all -- especially on laptops with poor speaker-mic isolation.

My solution: a speaking-state gate. When Gemini is outputting audio, we suppress inbound audio at the application level:

# Echo gate in the session handler
class SessionState:
    def __init__(self):
        self.agent_speaking = False
        self.last_agent_audio_time = 0.0

    def should_forward_audio(self, rms: float) -> bool:
        # Suppress during agent speech + 1.5s cooldown after
        if self.agent_speaking:
            return False
        if time.time() - self.last_agent_audio_time < 1.5:
            return rms > 0.03  # Higher threshold during cooldown
        return rms > 0.01  # Normal threshold

This two-tier threshold was the key insight: background noise sits at RMS 0.01-0.02, so during the cooldown period after the agent stops speaking, we only forward audio that's clearly human speech (> 0.03).

Failure Mode: The 1011 Disconnects

For weeks, Gemini Live API would randomly close connections with status code 1011 (Internal Server Error). No pattern, no warning. Sessions would die mid-sentence.

The fix was layered:

# Reconnection with session resumption
async def handle_disconnect(session, ws):
    for attempt in range(3):
        try:
            # Gemini supports session resumption via handle
            new_session = await GeminiLiveSession.resume(
                session.resumption_handle
            )
            # Re-send last audio chunk as context
            await new_session.send_audio(session.last_chunk)
            return new_session
        except Exception:
            await asyncio.sleep(0.5 * (attempt + 1))
    # After 3 fails, notify user with audio message
    await ws.send_json({"type": "system", "message": "reconnecting"})

Session resumption handles (persisted to Firestore) were a game-changer. Instead of starting a new conversation, Gemini picks up exactly where it left off. Users barely notice the blip.

Step 3: Playing Audio Back in the Browser

Gemini returns 24kHz PCM chunks. Playing them without glitches requires the Web Audio API with a buffer scheduler:

// Audio playback (simplified from useAudioOutput.ts)
class AudioPlayer {
  private context: AudioContext;
  private nextStartTime = 0;
  private gainNode: GainNode;

  constructor() {
    this.context = new AudioContext({
      sampleRate: 24000,
      latencyHint: /Mobi/.test(navigator.userAgent)
        ? "playback" : "interactive"
    });
    this.gainNode = this.context.createGain();
    this.gainNode.connect(this.context.destination);
  }

  playChunk(pcmBase64: string) {
    const bytes = Uint8Array.from(atob(pcmBase64), c => c.charCodeAt(0));
    const int16 = new Int16Array(bytes.buffer);
    const float32 = new Float32Array(int16.length);
    for (let i = 0; i < int16.length; i++) {
      float32[i] = int16[i] / 32768;
    }

    const buffer = this.context.createBuffer(1, float32.length, 24000);
    buffer.getChannelData(0).set(float32);

    const source = this.context.createBufferSource();
    source.buffer = buffer;
    source.connect(this.gainNode);

    const startTime = Math.max(
      this.context.currentTime, this.nextStartTime
    );
    source.start(startTime);
    this.nextStartTime = startTime + buffer.duration;
  }
}

The nextStartTime scheduler ensures seamless playback regardless of network jitter. The latencyHint switch between mobile ("playback") and desktop ("interactive") was a subtle but important optimization -- mobile browsers handle audio buffers differently.

What I Learned Building This Solo

1. Build the unhappy path first. I spent week one on the happy path. Weeks two through four were entirely edge cases -- reconnection, echo suppression, barge-in handling. If I could redo it, I'd build error recovery before a single feature.

2. Voice is a different UX paradigm. Users don't read error messages mid-conversation. Every failure needs an audio fallback. We pre-compute "filler" audio chunks ("one moment please...") as 24kHz PCM, ready to stream instantly when Gemini is slow or reconnecting.

3. Speech-to-speech beats STT+TTS. We initially considered a Whisper -> Claude -> ElevenLabs pipeline. Gemini Live API's native audio mode is faster (sub-500ms round-trip), cheaper, and handles interruptions naturally. The trade-off: less control over the voice, but the latency gain is massive.

4. Cloud Run works for WebSockets, with caveats. We deploy on Google Cloud Run (me-west1 region). WebSocket connections survive container restarts thanks to session resumption handles saved in Firestore. The key setting: request timeout of 3600s (1 hour) for long interview sessions.

The Result

The system now runs validation interviews averaging 12 minutes of continuous voice conversation. Across 500+ sessions, hard failures dropped to under 1% after implementing the echo gate and session resumption. Each interview includes 3 AI agents (Alex for discovery, Sam for architecture, Maya for design) that use ~12 function-calling tools to research markets, analyze competitors, and generate reports -- all while maintaining a natural conversation.

Building this solo meant every failure landed directly in my Telegram inbox (via a monitoring bot). Which, honestly, is the fastest feedback loop possible.

What I'm Curious About

The biggest remaining challenge is audio quality on mobile browsers. iOS Safari handles AudioContext differently from Chrome, and some Android devices have aggressive echo cancellation that clips the AI's speech. We're currently using device-specific settings, but it feels like a hack.

Has anyone found a robust cross-browser audio playback strategy for real-time AI voice? Especially interested in experiences with AudioWorklet vs ScriptProcessorNode for this use case. Drop your thoughts in the comments.