Voice AI Latency Optimization: What a 1,272ms Calendar Check Taught Us

#webdev #nextjs #ai #typescript

Originally published at parvejshah.com/blog/architecting-sub-18s-voice-ai-pipelines by Parvej Shah.

In the enterprise Voice AI space, there is a vast gulf between high-level architectural whitepapers and what actually runs on production telephony lines when real customers call in.

When building conversational booking agents for service contractors using Retell AI, n8n, EspoCRM, and Google Calendar, the technical challenge is rarely about getting a model to understand English.

The real engineering challenge is the vicious intersection of round-trip latency, external CRM tool execution overhead, and per-minute telephony economics. Here's what that actually looked like in production, and what it took to fix.

1. Two Agents, Two Models

There isn't one voice agent here — there are two live deployments, running two different models. The Horizon Realty (real estate) agent runs gemini-2.0-flash as its response engine. The Ironclad Pest (pest control) agent runs Gemini 3.1 Flash Lite. Both are Flash-tier models, chosen specifically over flagship models like GPT-4o for lower Time-To-First-Token (TTFT) and lower per-token cost on high-volume telephony.

2. The Real Latency Budget

Every external tool call — checking Google Calendar, touching EspoCRM — pauses the voice pipeline. The n8n rows below are measured directly by this project's regression suite; the STT/TTFT/TTS rows are cited industry and vendor benchmarks for the components in use.

Component	Target SLA	Benchmark
STT (Deepgram Nova-2)	< 180ms	~140ms
LLM TTFT (Gemini 2.0 Flash)	< 300ms	~220ms
TTS first packet (Cartesia Brian)	< 180ms	~120ms
n8n cache check	< 150ms	92.5ms (measured)
n8n calendar booking	< 1000ms	846.5ms (measured)
Total conversational turn	< 800ms	~600ms

3. The Actual Bottleneck Wasn't the Model

The pest-control agent's real production data told a different story than the latency table above. Reading the actual call transcripts, the problem wasn't model speed — it was conversation shape. The original Emergency Intake flow had 5 mandatory sequential nodes, each asking exactly one question, regardless of what the caller had already said in their opening statement. Add fee disclosure, time preference, a live 1,272ms Google Calendar round-trip for every single availability check, and confirmation, and a booking call needed 10-12 agent turns minimum — well past the 4-6 turns a well-designed intake needs.

4. Caching Without Redis

The fix for the 1,272ms calendar round-trip wasn't an external cache service — it's n8n's own workflow state. Here's the actual deployed node, pulled directly from the live n8n instance:

// Actual deployed check_availability Code node (n8n)
const staticData = $getWorkflowStaticData('global');
const CACHE_TTL_MS = 5 * 60 * 1000; // 5 minutes
const now = Date.now();

if (
  staticData.slotsCache &&
  staticData.slotsCachedAt &&
  (now - staticData.slotsCachedAt) < CACHE_TTL_MS
) {
  // Cache hit: <50ms, no Google Calendar call at all
  const cachedSlots = staticData.slotsCache;
  return [{ json: { available_slots: cachedSlots.slice(0, 5), cache_hit: true } }];
}

// Cache miss: compute from live Google Calendar data, then store for next time
const allFreeSlots = computeFreeSlotsFromCalendarEvents($input.all());
staticData.slotsCache = allFreeSlots;
staticData.slotsCachedAt = now;
return [{ json: { available_slots: allFreeSlots.slice(0, 5), cache_hit: false } }];

No Redis, no external cache service — just n8n's own staticData global object with a 5-minute TTL. A cold lookup (Google Calendar API round-trip) costs 1,272ms; a cache hit inside that window costs under 50ms.

5. Rebuilding the Intake as Multi-Slot Extraction

The second fix targeted turn count directly: replacing the 5 rigid sequential questions with a single extraction pass that reads what the caller already said and only asks for what's actually missing — "Got it — can I grab your name, phone, and address?" instead of five separate one-at-a-time prompts.

6. What Actually Moved

Average call duration on the pest-control agent dropped from an internal ~3m40s baseline to ~2m18s, measured across the most recent 45 live calls. The availability cache turned a 1,272ms cold path into a sub-50ms cache hit for repeat lookups.

Parvej Shah is a Lead Full-Stack Web Developer & Platform Architect based in Dhaka, Bangladesh. Explore full architecture case studies and production code at parvejshah.com.

DEV Community