The problem we started with
Most small businesses still handle phone calls the old way: a human answers, or a menu tree routes the caller to a department. If it's after hours, the call goes to voicemail. If the customer speaks a language the team doesn't cover, the call drops. If the issue needs two departments, the customer repeats themselves twice.
We wanted to build an AI agent that could take over those calls — not as a chatbot on a website, but as a real-time voice agent on the phone. And we wanted it to work in 70+ languages without building a separate model for each one.
The result is RingBot, an Agent-as-a-Service platform. This post is a short technical walkthrough of what we learned building it.
What "real-time voice" actually means
A traditional voice bot works like this:
- Speech-to-text (STT) transcribes the audio
- An LLM generates a text response
- Text-to-speech (TTS) reads it back
That works, but the latency is noticeable. Every step adds 500ms–1s, and the conversation feels robotic.
Real-time voice models like GPT Realtime, Gemini Live, and Grok Voice use an end-to-end audio pipeline. The model listens to raw audio and generates audio back directly, without going through text in the middle. That cuts latency dramatically and makes turn-taking feel natural.
Why we use multiple models
We built RingBot to be model-agnostic. Each real-time model has trade-offs:
- GPT Realtime is strong at following instructions and handling complex multi-turn tasks.
- Gemini Live is excellent across many languages and has a very natural voice persona.
- Grok Voice gives us a different latency/cost profile and a distinct voice character.
Instead of betting on one model, we route calls to the best model for the job. A caller can even switch models mid-conversation if that improves the experience.
The architecture in four pieces
Our production stack looks like this:
- Telephony gateway — handles SIP/WebRTC, streams audio to the agent, and manages call state.
- Agent orchestrator — picks the right model, loads the business context, and runs the conversation loop.
- Knowledge base — semantic search over product docs, FAQs, and catalogs so the agent answers with live information, not stale scripts.
- Handoff layer — when a human needs to take over, it transfers the full transcript, intent summary, and customer context to the support agent.
The agent orchestrator is the interesting part. It doesn't just call a model. It also manages intent classification, routing rules, tool calls (e.g., check order status, book an appointment), and escalation thresholds.
The multilingual challenge
The biggest surprise was that multilingual voice is now mostly a solved problem — for the model layer. The harder part is the surrounding system:
- Business context translation: Your knowledge base is probably in one language. We use retrieval that works across languages, so a Spanish caller can get answers from English docs.
- Routing intent across languages: The agent must recognize that "quiero hablar con ventas" and "I want to speak to sales" mean the same thing.
- Voice persona consistency: We want the same agent personality across languages, not a different voice for each locale.
- Latency on low-resource languages: Some languages are slower on certain models. We monitor real-time latency per language and route accordingly.
Context is the hard part
Voice conversations move fast. If a caller says, "Actually, I want to change my appointment, not cancel it," the agent has to update its plan immediately. If it then transfers to a human, the receiving agent needs to know:
- What the original request was
- What changed
- What still needs to be done
- What tools the AI already called
We solve this by maintaining a structured conversation state that travels with the call. The human agent sees a short summary plus the full transcript, instead of asking the customer to repeat the story.
What we would do differently
- Start with evaluation tooling. It's hard to tell if a voice agent is "good" without listening to calls. We built automated evals for latency, turn-taking, and task completion, but we should have done it earlier.
- Fail loud and fast. A confused voice agent that loops forever is worse than one that immediately transfers to a human. We added escalation triggers based on repetition and confidence scores.
- Respect platform limits. Each real-time API has its own quirks around audio formats, rate limits, and session duration. We abstracted these into a provider layer so callers don't need to know which model is running.
The "no code" layer
Under the hood, this is a fairly technical system. But the end user — a small business owner — shouldn't have to think about telephony, embeddings, or model selection. So we wrapped everything in a config flow: connect your business info, upload your docs, pick a phone number, and deploy.
That's the Agent-as-a-Service idea. The user hires an agent, not a toolkit.
Try it or ask questions
If you're building with real-time voice APIs, I'd love to hear what you're running into. And if you want to see a production example, you can check out RingBot.
Top comments (1)
Really like the point about failing fast instead of looping. In my experience, users forgive a quick handoff much more than repeating the same question three times.