DEV Community

Mart Schweiger
Mart Schweiger

Posted on • Originally published at assemblyai.com

Vapi Voice Agent with AssemblyAI Universal-3 Pro Streaming

#python #ai #tutorial #vapi

Vapi Voice Agent with AssemblyAI Universal-3 Pro Streaming

Use AssemblyAI Universal-3 Pro Streaming as the speech-to-text engine inside your Vapi voice agent — and get neural turn detection, keyterm prompting, and 307ms P50 latency inside Vapi's managed voice platform.

What is Vapi?

Vapi handles telephony, turn-taking, and orchestration so you don't have to. It supports 14+ speech-to-text providers. You bring your AssemblyAI key, Vapi handles the rest.

Setup: Add AssemblyAI to Vapi

Step 1 — Add Your API Key

  1. Go to dashboard.vapi.ai
  2. Navigate to Settings > Transcriber Providers
  3. Add your AssemblyAI API key

Step 2 — Create an Assistant (Dashboard)

  1. Click Create Assistant
  2. Under Transcriber, select Assembly AI
  3. Choose u3-rt-pro under Model
  4. Save and test via the web call button

Step 3 — Create an Assistant (API) 

git clone https://github.com/kelseyefoster/voice-agent-vapi-assemblyai
cd voice-agent-vapi-assemblyai

pip install -r requirements.txt
cp .env.example .env

python create_assistant.py create
Enter fullscreen mode Exit fullscreen mode

This creates a fully configured assistant:

{
  "transcriber": {
    "provider": "assembly-ai",
    "model": "u3-rt-pro",
    "language": "en",
    "keytermsPrompt": ["YourBrand", "SpecialTerm"],
    "confidenceThreshold": 0.4
  }
}
Enter fullscreen mode Exit fullscreen mode

Quick Start (Full) 

# Create an assistant
python create_assistant.py create

# Make an outbound test call
python create_assistant.py call --assistant-id <id> --phone +1XXXXXXXXXX

# Start the webhook server
uvicorn webhook_server:app --port 8000
Enter fullscreen mode Exit fullscreen mode

Keyterm Prompting 

"keytermsPrompt": [
    "hemoglobin A1c",
    "HIPAA",
    "Jardiance",
    "deductible"
]
Enter fullscreen mode Exit fullscreen mode

Up to 100 keyterms, 50 characters each. Changes take effect on the next call without restarting the assistant.

When to Choose AssemblyAI in Vapi

Use Case Recommendation
Fastest streaming latency AssemblyAI (307ms P50)
Account/serial codes AssemblyAI (+21% fewer alphanumeric errors)
Medical terminology AssemblyAI (keyterm prompting)
Interruption handling AssemblyAI (punctuation-based turn detection)
Multilingual callers AssemblyAI (native code switching)

Resources

Top comments (0)