DEV Community: Evie Wang

Building an AI Conversation Practice App: Part 3- From Simple Prompts to Database-Driven Dynamic AI Characters

Evie Wang — Sun, 21 Sep 2025 22:32:34 +0000

This is the third post in a series documenting how we built a browser-based English learning app with a cost-friendly conversation system.

Overview: From Static Prompts to Dynamic Character Generation

After transcribing user speech, our system needs to generate contextually appropriate responses that feel natural and authentic. I started with hardcoded prompts for our MVP, which worked well initially. As we will add more scenarios and want greater personality variation in the futhure, then I upgraded it to a database-driven system that could scale.

The complete character generation workflow involves:

Scenario Analysis → Intelligent mapping from scenario titles to character types
Character Selection → Pull the right character profile from the database
Dynamic Prompt Building → Build prompts that include personality details
Context Adaptation → Adjust responses based on timing (e.g., busy vs. relaxed moments)
GPT-4 Integration → Generate replies that fit the chosen character

Total processing time: ~1-2s (varies by network conditions)

Technical Stack:

Database: Supabase (PostgreSQL)
Prompt Engineering: Dynamic multi-layer construction
AI Model: GPT-4o with character-specific prompts
Language: JavaScript with Next.js API routes
Error Handling: JSON parsing safety and database fallbacks

From Prototype to Scale

The Starting Point

I began with a straightforward approach for our MVP:

// Simple but effective for initial scenarios
if (scenarioTitle.includes('tim hortons')) {
  const prompt = "You are a friendly Tim Hortons employee. Be helpful and use Canadian expressions...";
} else if (scenarioTitle.includes('restaurant')) {
  const prompt = "You are a restaurant server. Be professional and friendly...";
}

This worked well for getting started, but as our content expanded, we needed more flexibility and personality variation.

the Database-driven Solution---Core Implementation Details

Intelligent Scenario Mapping

We built a mapping system that automatically determines which character to use:

const mapScenarioToCharacterKey = (scenarioTitle) => {
  const titleLower = scenarioTitle.toLowerCase();

  // Tim Hortons scenarios
  if (titleLower.includes('tim hortons') || titleLower.includes('coffee shop')) {
    return 'tim_hortons';
  }

  // Restaurant scenarios with sub-categories
  if (titleLower.includes('restaurant') || titleLower.includes('dining')) {
    if (titleLower.includes('fast') || titleLower.includes('mcdonald')) {
      return 'restaurant_fast';
    } else if (titleLower.includes('fine') || titleLower.includes('upscale')) {
      return 'restaurant_fine';
    } else {
      return 'restaurant_casual';
    }
  }

  return 'directions'; // Default fallback
};

Database Schema Design

Our character system uses four tables that work together:

erDiagram
    character_profiles {
        int id PK
        string scenario_key
        string character_name
        text base_prompt_template
        string correction_style
    }

    personality_traits {
        int id PK
        int character_id FK
        string trait_type
        string trait_value
    }

    canadian_expressions {
        int id PK
        int character_id FK
        string expression_category
        json phrases
    }

    response_patterns {
        int id PK
        int character_id FK
        string time_context
        string situation_type
        json response_templates
        int avg_sentence_count
        boolean includes_questions
    }

    character_profiles ||--o{ personality_traits : "defines"
    character_profiles ||--o{ canadian_expressions : "uses"
    character_profiles ||--o{ response_patterns : "follows"

Table Relationships:

One character profile links to many personality traits
Each character has their own Canadian expressions as examples, and they are organized by category
Response patterns vary based on context (busy vs slow)

Character Generation Service

Here's our main character generation function (lib/characterService.js):

export async function generateCharacterPrompt(scenarioKey) {
  try {
    // Get the basic character information
    const { data: character, error: characterError } = await supabase
      .from('character_profiles')
      .select('*')
      .eq('scenario_key', scenarioKey)
      .single();

    if (characterError) {
      console.error('Character not found:', characterError);
      return null;
    }

    // Get personality traits
    const { data: traits, error: traitsError } = await supabase
      .from('personality_traits')
      .select('*')
      .eq('character_id', character.id);

    if (traitsError) {
      console.error('Error fetching traits:', traitsError);
      return null;
    }

    // Get Canadian expressions
    const { data: expressions, error: expressionsError } = await supabase
      .from('canadian_expressions')
      .select('*')
      .eq('character_id', character.id);

    if (expressionsError) {
      console.error('Error fetching expressions:', expressionsError);
      return null;
    }

    // Get response patterns based on random time context
    const timeContext = getRandomTimeContext();
    const { data: patterns, error: patternsError } = await supabase
      .from('response_patterns')
      .select('*')
      .eq('character_id', character.id)
      .eq('time_context', timeContext);

    if (patternsError) {
      console.error('Error fetching patterns:', patternsError);
      return null;
    }

    // Convert arrays to objects for easier access
    const personalityMap = {};
    traits.forEach(trait => {
      personalityMap[trait.trait_type] = trait.trait_value;
    });

    const expressionsMap = {};
    expressions.forEach(expr => {
      try {
        expressionsMap[expr.expression_category] = JSON.parse(expr.phrases);
      } catch (error) {
        console.error('JSON parse error for expression:', expr.expression_category);
        expressionsMap[expr.expression_category] = [];
      }
    });

    const patternsMap = {};
    patterns.forEach(pattern => {
      try {
        patternsMap[pattern.situation_type] = {
          templates: JSON.parse(pattern.response_templates),
          avg_sentence_count: pattern.avg_sentence_count,
          includes_questions: pattern.includes_questions
        };
      } catch (error) {
        console.error('JSON parse error for pattern:', pattern.situation_type);
        patternsMap[pattern.situation_type] = {
          templates: [],
          avg_sentence_count: 2,
          includes_questions: false
        };
      }
    });

    // Build the complete system prompt
    const systemPrompt = buildSystemPrompt(character, personalityMap, expressionsMap, patternsMap, timeContext);

    return {
      role: character.character_name,
      systemPrompt,
      timeContext,
      characterData: {
        personality: personalityMap,
        expressions: expressionsMap,
        patterns: patternsMap,
        correctionStyle: character.correction_style
      }
    };

  } catch (error) {
    console.error('Error generating character prompt:', error);
    return null;
  }
}

Dynamic Prompt Construction

Our prompt-building function combines all character data:

function buildSystemPrompt(character, personality, expressions, patterns, timeContext) {
  const contextDescription = timeContext === 'busy' ? 
    'It is currently a busy time, so you respond more quickly and concisely.' :
    'It is currently a slower time, so you can be more conversational and detailed.';

  // Extract expression examples for the prompt
  const greetingExamples = expressions.greetings ? expressions.greetings.join(', ') : '';
  const transitionExamples = expressions.transitions ? expressions.transitions.join(', ') : '';
  const confirmationExamples = expressions.confirmations ? expressions.confirmations.join(', ') : '';

  return `${character.base_prompt_template}

CURRENT CONTEXT: ${contextDescription}

PERSONALITY TRAITS:
- Formality: ${personality.formality_level || 'casual'}
- Energy: ${personality.energy_level || 'medium'} 
- Chattiness: ${personality.chattiness || 'moderate'}
- Service Pace: ${personality.service_pace || 'normal'}

CANADIAN EXPRESSIONS TO USE:
- Greetings: ${greetingExamples}
- Transitions: ${transitionExamples}
- Confirmations: ${confirmationExamples}

RESPONSE GUIDELINES:
- Keep responses ${timeContext === 'busy' ? '1-2 sentences' : '2-4 sentences'} maximum
- Use natural Canadian expressions from the examples above
- Stay completely in character as a real ${character.character_name}
- If you need to correct English mistakes, use the "${character.correction_style}" style
- For corrections, respond naturally first, then gently clarify. Example: "Oh, you mean a large coffee? Sure thing!"
- NEVER comment on someone's English skills or act like a language tutor
- Just have a natural conversation as if you're really in this workplace

CORRECTION EXAMPLES:
- If they say "I want big coffee" → "Oh, you mean a large coffee? Coming right up!"
- If they say "Where is bathroom?" → "Oh, the washroom? It's just down the hall there."
- If they say unclear something → "Sorry, what was that?" or "Can you repeat that?"

Remember: You are a real ${character.character_name} during ${timeContext === 'busy' ? 'busy' : 'slow'} time. Act natural and authentic!`;
}

A Real Example to help understand deeper: Tim Hortons Employee

User Scenario: "Ordering at Tim Hortons"

Step 1: Scenario Mapping

mapScenarioToCharacterKey("Ordering at Tim Hortons") 
// Returns: "tim_hortons"

Step 2: Database Queries Return

character = {
  id: 1,
  character_name: "Tim Hortons Employee",
  base_prompt_template: "You are a friendly Tim Hortons employee working the counter..."
}

personalityMap = {
  formality_level: "casual",
  energy_level: "high",
  chattiness: "moderate",
  service_pace: "fast"
}

expressionsMap = {
  greetings: ["Hey there!", "How's it going?", "Morning!"],
  confirmations: ["You bet!", "For sure!", "Absolutely!"],
  transitions: ["Alrighty", "Perfect", "Awesome"]
}

// Random timeContext = "busy"
patternsMap = {
  ordering: {
    templates: ["What can I get started for you?", "Next!"],
    avg_sentence_count: 1,
    includes_questions: true
  }
}

Step 3: Generated Prompt

const systemPrompt = `You are a friendly Tim Hortons employee working the counter...

CURRENT CONTEXT: It is currently a busy time, so you respond more quickly and concisely.

PERSONALITY TRAITS:
- Formality: casual
- Energy: high
- Chattiness: moderate
- Service Pace: fast

CANADIAN EXPRESSIONS TO USE:
- Greetings: Hey there!, How's it going?, Morning!
- Confirmations: You bet!, For sure!, Absolutely!
- Transitions: Alrighty, Perfect, Awesome

RESPONSE GUIDELINES:
- Keep responses 1-2 sentences maximum
- Use natural Canadian expressions from the examples above
- Stay completely in character as a real Tim Hortons Employee
`;

Step 4: GPT-4 Response

"Hey there! What can I get started for you today?"

Notice how the response incorporates:

Casual greeting from the expressions database
Concise format due to "busy" context
High energy personality trait
Natural Canadian friendliness

If the same scenario ran with timeContext = "slow", we might get:

"Morning! How's your day going so far? What can I get you today - maybe a nice double-double to start?"

Same character, different context, completely different response style.
Now it’s super easy to add or change characters since everything lives in the database, no need to mess with code. Each character has their own vibe and can react differently depending on the situation, and even if something goes wrong, the convo keeps going without breaking.

What's Next

In our next post, we'll explore how these generated text responses are converted back to speech using totally free browser-based text-to-speech capabilities, completing the full audio conversation loop.

Building an AI Conversation Practice App: Part 2 - Backend Speech-to-Text Processing with OpenAI Whisper

Evie Wang — Sun, 21 Sep 2025 05:40:42 +0000

This is the second post in a series documenting the technical implementation of a browser-based English learning application with cost-friendly conversation system.

Overview: The STT Pipeline

The complete STT workflow involves:

Audio Reception → FormData parsing with formidable
File Validation → WebM format verification and size checks
Stream Processing → Direct file stream to OpenAI API
Transcription → Whisper-1 model with Canadian English optimization
Response Handling → Error management and cleanup
Integration → Seamless handoff to conversation system

Total processing time: 200-500ms

Technical Stack Summary:

Primary STT: OpenAI Whisper-1
File Processing: Formidable + Node.js streams
Language: TypeScript with Next.js API routes
Error Handling: Basic try-catch with error logging
Performance: Stream processing, Node.js runtime

The Challenges I Solved

1. File Upload Complexity in Next.js

Problem: Next.js API routes have strict limitations on file uploads, especially with form-data.

Solution: Used a custom formidable-based parser:

// Disable Next.js body parsing
export const config = { api: { bodyParser: false } };

// Custom form parsing with formidable
const form = new IncomingForm({
  keepExtensions: true,
});

const formData: [Fields, Files] = await new Promise((resolve, reject) => {
  form.parse(req, (err, fields, files) => {
    if (err) return reject(err);
    resolve([fields, files]);
  });
});

The reason:

Bypasses Next.js 1MB body size limit
Handles WebM files up to 25MB
Maintains file metadata and extensions
Provides proper error handling

2. Stream Processing for Large Files

Problem: Loading entire audio files into memory causes performance issues and potential crashes after deployment.

Solution: Direct stream processing to OpenAI API:

// Create readable stream from uploaded file
const audioPath = audioFile.filepath;
const audioStream = createReadStream(audioPath);

// Stream directly to OpenAI (no memory buffering)
const transcription = await openai.audio.transcriptions.create({
  file: audioStream,
  model: "whisper-1",
  language: "en",
  prompt: "This is a conversation in Canadian English.",
});

Performance Benefits:

Significantly reduced memory usage through streaming
Faster processing for large files
Better reliability and no memory overflow crashes

3. Frontend Audio Validation

Problem: Short audio recordings (< 300ms) are often accidental and waste API calls.
Solution: Early validation on the frontend before sending to backend

// Frontend validation before API call
const recordingDuration = Date.now() - recordingStartTimeRef.current;

if (recordingDuration < 300) {
  const clarificationText = getRandomClarification();

  const assistantMessage: Message = {
    role: 'assistant',
    content: '',
    isStreaming: true
  };

  setMessages(prevMessages => [...prevMessages, assistantMessage]);
  streamText(clarificationText, messageIndex);
  return; // Don't call STT API
}

// Only send to backend if recording is long enough
const sttResponse = await fetch('/api/stt', {
  method: 'POST',
  body: formData,
});

Results:

API call reduction: ~15% fewer unnecessary calls
User experience: Immediate feedback for accidental recordings
Cost savings: Reduced unwanted OpenAI API usage

4. Canadian English Optimization

Problem: Default Whisper models aren't optimized for Canadian English expressions and pronunciation patterns.

Solution: Custom prompt engineering:

const transcription = await openai.audio.transcriptions.create({
  file: audioStream,
  model: "whisper-1",
  language: "en",
  prompt: "This is a conversation in Canadian English.",
});

Results:

Better recognition of Canadian expressions
Improved handling of slang and culture-related expressions

Core Technical Implementation

1. API Endpoint Architecture

Our main STT endpoint (/api/stt) follows a robust error-handling pattern:

export default async function handler(
  req: NextApiRequest, 
  res: NextApiResponse<ApiResponse>
) {
  if (req.method !== 'POST') {
    return res.status(405).json({ success: false, error: 'Method not allowed' });
  }

  try {
    // Parse form data
    const form = new IncomingForm({ keepExtensions: true });
    const formData: [Fields, Files] = await new Promise((resolve, reject) => {
      form.parse(req, (err, fields, files) => {
        if (err) return reject(err);
        resolve([fields, files]);
      });
    });

    const [fields, files] = formData;

    // Validate audio file
    const audioFiles = files.audio;
    if (!audioFiles || !Array.isArray(audioFiles) || audioFiles.length === 0) {
      return res.status(400).json({ success: false, error: 'No audio file provided' });
    }

    const audioFile = audioFiles[0] as File;

    // Process with OpenAI
    const audioPath = audioFile.filepath;
    const audioStream = createReadStream(audioPath);

    const transcription = await openai.audio.transcriptions.create({
      file: audioStream,
      model: "whisper-1",
      language: "en",
      prompt: "This is a conversation in Canadian English.",
    });

    // Cleanup and respond
    fs.unlinkSync(audioPath);
    return res.status(200).json({
      success: true,
      transcript: transcription.text
    });

  } catch (error) {
    console.error('STT Error:', error);
    return res.status(500).json({
      success: false,
      error: error instanceof Error ? error.message : 'Failed to transcribe audio'
    });
  }
}

2. File Validation & Security

// Access the audio file with proper type checking
const audioFiles = files.audio;
if (!audioFiles || !Array.isArray(audioFiles) || audioFiles.length === 0) {
  return res.status(400).json({ 
    success: false, 
    error: 'No audio file provided' 
  });
}

const audioFile = audioFiles[0] as File;

// Additional validation
if (!audioFile.filepath || audioFile.size === 0) {
  return res.status(400).json({ 
    success: false, 
    error: 'Invalid audio file' 
  });
}

Security Measures:

File type validation (WebM only)
Size limits (25MB max)
Temporary file cleanup
No persistent storage

3. Resource Management

// Critical: Clean up temporary files
const audioPath = audioFile.filepath;
const audioStream = createReadStream(audioPath);

// Process audio...

// Always cleanup, even on error
try {
  fs.unlinkSync(audioPath);
} catch (cleanupError) {
  console.warn('Failed to cleanup temp file:', cleanupError);
}

Resource Management Benefits:

Disk space: Prevents temp file accumulation
Security: No persistent audio storage
Performance: Clean server state

Performance Optimizations

1. Streaming vs Buffering

Before (Buffering):

// Load entire file into memory
const audioBuffer = fs.readFileSync(audioPath);
const transcription = await openai.audio.transcriptions.create({
  file: audioBuffer, // Large memory usage
});

After (Streaming):

// Stream file directly
const audioStream = createReadStream(audioPath);
const transcription = await openai.audio.transcriptions.create({
  file: audioStream, // Minimal memory usage
});

Results:

Significantly reduced memory usage through streaming
Faster processing for large files
Better support for concurrent requests

Integration with Frontend

The STT API seamlessly integrates with our frontend conversation system:

// Frontend STT call
const sttResponse = await fetch('/api/stt', {
  method: 'POST',
  body: formData,
});

const sttData = await sttResponse.json();

if (!sttData.success) {
  // Handle error gracefully
  const clarificationText = getRandomClarification();
  // Show clarification message to user
} else {
  // Continue with conversation
  const transcript = sttData.transcript;
  // Send to GPT for response generation
}

Error Handling & User Experience

1. Graceful Degradation

// If STT fails, don't break the conversation
if (!sttData.success) {
  const clarificationPhrases = [
    "Sorry, can you repeat that?",
    "Could you say that again please?",
    "I didn't quite get that. Could you repeat?",
  ];

  const randomClarification = clarificationPhrases[
    Math.floor(Math.random() * clarificationPhrases.length)
  ];

  // Continue conversation with clarification
}

2. Debugging & Monitoring

// Comprehensive logging for debugging
console.log('STT Response:', {
  success: sttData.success,
  transcript: sttData.transcript?.substring(0, 50) + '...',
  processingTime: Date.now() - startTime,
  fileSize: audioFile.size
});

Production Considerations

Rate Limiting

// Implement rate limiting for production
if (requestCount > 10) { // 10 requests per minute
  return res.status(429).json({
    success: false,
    error: 'You\'re speaking too fast! Please wait a moment before trying again.'
  });
}

frontend：

if (response.status === 429) {
  showError("Please wait a moment before recording again");
}

What's Next

In the next post, we’ll see how the transcribed text powers our AI conversation system, from selecting specific characters and crafting prompts for Canadian English, also integrating with GPT-4 and keeping conversations flowing naturally.

Building an AI Conversation Practice App: Part 1 - Browser Audio Recording with MediaRecorder API

Evie Wang — Sat, 20 Sep 2025 21:10:04 +0000

This is the first post in a series documenting the technical implementation of a browser-based English learning application with a cost-friendly conversation system.

The Story Behind This Project

When I started building my language learning app, I wanted to create something more engaging than traditional flashcards. The idea was simple: let users practice real conversations through AI-powered role-play scenarios which alighs with real-life English and be really helpful.
What seemed like a straightforward feature turned into one of the most technically challenging parts of my entire application.

The final result? A pseudo-real-time but cost-friendly conversation system where users can speak naturally with AI characters, get instant feedback, and practice scenarios like ordering in a restaurant or customer service calls. But getting there required solving problems I never expected as a student developer.

Overview: The Complete Audio-to-Conversation Workflow

Our English learning platform implements a sophisticated role-play conversation system that transforms user speech into intelligent AI responses within 2-5 seconds.
Now let me walk through what happens when a user starts a role-play conversation:

User Interaction → User presses and holds the microphone button
Audio Capture → Browser records audio using MediaRecorder API
Audio Processing → Create audio blob and prepare FormData for upload
Speech Recognition → OpenAI Whisper-1 transcribes audio to text
Character Response → GPT-4 generates contextual response using character system
Text-to-Speech → Browser Web Speech API converts response to audio
Progress Tracking → User progress and usage limits updated in database

Total response time: 2-5 seconds (optimized for conversational feel)

This series takes a closer look at each component of the workflow, starting with the foundation—frontend audio recording and processing. The first post focuses on the technical details of browser-based audio capture, while the following posts will explore the rest of the pipeline that enables real-time AI conversations.

The Challenges We Discovered

What seemed straightforward in our project planning phase revealed layer after layer of unexpected complexity:

Cost Management Reality

API Cost Explosion: Our initial implementation used detailed Azure pronunciation analysis but quickly became prohibitively expensive at scale
Usage Control Strategy: After analyzing token-based and conversation-based limits, we settled on recording-count limits (5 free recordings per day) - simple, predictable, and abuse-resistant.

Performance vs. Quality Trade-offs

Response Time Optimization: Detailed Azure analysis took ~15 seconds; we optimized to ~6 seconds while maintaining meaningful feedback
Platform Constraints: Vercel's 10-second serverless function limits forced architectural decisions
TTS Evolution: Switched from OpenAI TTS to browser Web Speech API, removed network latency, and improved user experience at the same time.

Real-World Production Constraints

Browser Compatibility: Built detection and fallback systems for MediaRecorder and Web Audio API support
Abuse Prevention: Implemented session limits and daily caps with real-time usage tracking
Graceful Error Handling: Fallback systems for failed TTS, audio processing errors, and network issues

File Structure & Architecture

components/
├── ConversationInterface.tsx # Main conversation UI & state management
├── AudioRecorder.jsx # Audio recording component
└── SlangLoader.tsx # Loading animations
pages/
├── scenario/[id]/roleplay.js # Role-play page container
└── api/
├── enhanced-chat.ts # GPT conversation API
├── stt.ts # Speech-to-text API
└── azure-analytics-client.ts # Audio analysis API
utils/
├── clientAudioProcessor.ts # Client-side audio conversion
├── browserTTS.js # Text-to-speech utility
└── usageLimits.js # Usage tracking system
lib/
└── characterService.mjs # Dynamic character generation

Technical Implementation

Because it’s too complex to explain clearly in a single post, I split it into separate posts, each covering a different stage of the workflow.

sequenceDiagram
    participant Client as Frontend (Browser)
    participant Server as Backend (Server)

    Client->>Client: User records audio
    Client->>Client: Create audioBlob
    Client->>Client: FormData
    Client->>+Server: HTTP POST /api/stt
    Server->>Server: Parse audio file
    Server->>Server: Call OpenAI Whisper
    Server->>-Client: HTTP Response (transcript)

1. Audio Recording Architecture

Our implementation uses the modern Web Audio API with React state management:

// Core state management for audio recording
const [isRecording, setIsRecording] = useState(false);
const [isProcessing, setIsProcessing] = useState(false);
const mediaRecorderRef = useRef(null);
const audioChunksRef = useRef([]);
const recordingStartTimeRef = useRef(null);

2. MediaRecorder Configuration

We chose WebM with Opus codec as our recording format for optimal compression and quality:

const startRecording = async () => {
  try {
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    mediaRecorderRef.current = new MediaRecorder(stream, {
      mimeType: 'audio/webm;codecs=opus',
    });

    audioChunksRef.current = [];
    recordingStartTimeRef.current = Date.now();

    // Event handlers for data collection
    mediaRecorderRef.current.ondataavailable = (event) => {
      audioChunksRef.current.push(event.data);
    };

    mediaRecorderRef.current.onstop = () => {
      handleAudioSubmission();
    };

    mediaRecorderRef.current.start();
    setIsRecording(true);
  } catch (error) {
    console.error('Error starting recording:', error);
  }
};

3. Resource Management & Cleanup

const stopRecording = () => {
  if (mediaRecorderRef.current && isRecording) {
    // Stop the MediaRecorder
    mediaRecorderRef.current.stop();

    // Critical: Release microphone permissions
    mediaRecorderRef.current.stream.getTracks().forEach(track => track.stop());

    setIsRecording(false);
  }
};

So, this can

Stops the “microphone active” light from staying on
Releases system resources
Addresses user privacy concerns
Allows other applications to use the microphone

4. Audio Data Processing

The MediaRecorder API delivers audio in chunks, which we collect and process:

const handleAudioSubmission = async () => {
  if (audioChunksRef.current.length === 0) return;

  // Quality check: Filter out accidental short recordings
  const recordingDuration = Date.now() - recordingStartTimeRef.current;
  if (recordingDuration < 300) {
    // Handle too-short recordings gracefully
    return;
  }

  // Create blob from audio chunks
  const audioBlob = new Blob(audioChunksRef.current, { type: 'audio/webm' });

  // Prepare for backend processing
  const formData = new FormData();
  formData.append('audio', audioBlob, 'recording.webm');

  // Send to backend API
  const response = await fetch('/api/stt', {
    method: 'POST',
    body: formData,
  });
};

Technical Decisions & Trade-offs

When building a speech recognition app that integrates with multiple AI services, one of the biggest decisions is choosing the right audio format. Each API has different requirements, and the wrong choice can impact performance, user experience, and development complexity.

After testing various formats, we chose WebM + Opus as our frontend recording format. Here's why:

Browser Compatibility

Works on all modern browsers (except some iOS Safari versions)
Native support in Chrome, Firefox, Edge
Graceful fallback for older browsers

File Size Optimization

10:1 compression ratio compared to WAV
30-second recording: ~50KB vs ~500KB
Faster uploads, especially on mobile networks

Real-time Performance

Designed for streaming applications
Low latency encoding
Minimal CPU usage during recording

Audio Quality

High quality at low bitrates
Opus codec optimized for speech
Maintains clarity and quality for AI processing

Instead of forcing one format for all APIs, we implemented intelligent routing:

// Frontend: Always record as WebM
const mediaRecorder = new MediaRecorder(stream, {
  mimeType: 'audio/webm;codecs=opus',
});

// Backend: Route based on API requirements
if (apiType === 'openai') {
  // Direct WebM support - zero conversion
  return await openai.audio.transcriptions.create({
    file: formData, 
    model: "whisper-1"
  });
} else if (apiType === 'azure') {
  // Convert to WAV for Azure
  const wavBuffer = await convertWebMToWav(audioData);
  return await azureSTT(wavBuffer);
}

Real-world measurements:

Recording latency: <50ms (WebM) vs <100ms (WAV)
File upload time: 3-4x faster with WebM
API response time: OpenAI 300ms, Azure 800ms

Frontend Recording

// Smart format detection
const getSupportedMimeType = () => {
  const types = [
    'audio/webm;codecs=opus',
    'audio/webm',
    'audio/mp4',
    'audio/wav'
  ];

  for (const type of types) {
    if (MediaRecorder.isTypeSupported(type)) {
      return type;
    }
  }
  return 'audio/webm'; // fallback
};

Next Steps

In the next post, I'll cover the backend processing pipeline, including:

Audio format conversion and file handling
Integration with OpenAI Whisper for speech-to-text
The complete /api/stt implementation
Error handling and performance optimization strategies

The complete audio workflow achieves our target latency of under 2-5 seconds from speech capture to AI-generated response, which is a natural conversation flow for language learners.

Coming Up Next: Part 2 - Backend Speech-to-Text Processing with OpenAI Whisper