DEV Community

Cover image for AI voice agent routing: send small talk to a fast model and real work to an agent
MatIntellect
MatIntellect

Posted on Edited on Originally published at matintellect.com AI-assisted

AI voice agent routing: send small talk to a fast model and real work to an agent

Problem: a voice front end that sends every sentence to a heavy agent feels slow, and one that answers everything itself does no real work. Result of this article: a small, testable router that decides, for each spoken instruction, whether the fast conversational model replies, an agent with tools does the work, or the action stops for human approval. Stack: plain Python 3.10+ (tested on 3.15.2), no external packages. Boundaries: speech recognition, text to speech and the agents themselves are out of scope; the router works on text that recognition has already produced

AI-generated illustration of an AI voice agent and the system behind it

Where the pattern comes from

I built a voice agent called Ami for the owner of an aesthetic medicine clinic. The owner opens a private page, presses Start and talks. Small talk goes to a fast model so the conversation has no pauses. Real requests, such as checking the server, go to the clinic's agent with tools, and the answer comes back as speech. Memory is shared with the Telegram assistant. When the voice arrived, the clinic already had a team of 6 agents, so the voice had someone to hand work to

My own agent, Jes, works the same way on the input side: voice notes and text are treated alike, and behind her are 10 scheduled jobs and 7 other bots. Since July she has passed on 165 tasks over 134 days

The code below is a simplified, standalone teaching version of the routing idea. It is not production code from either system

Input, processing, output

Input: one recognised utterance as text

Processing: small talk is answered by the fast model straight away. Anything else gets an immediate acknowledgement from the fast model, then is split into parts and matched against the skills the system actually has. Each matched skill goes to the agent layer, unless it is in the set that needs a human yes, in which case it goes to the approval layer. If nothing matches, the fast model says so honestly

Output: an ordered list of steps, each with a layer, an action and the part of the sentence it came from

Two rules matter more than the matching itself. First, the acknowledgement comes before the work, so the speaker never sits in silence while tools run. Second, an unknown request produces "I can't do that yet" rather than a confident reply with nothing behind it

A sound mixing console

The example

from dataclasses import dataclass

SMALL_TALK = ("hello", "thanks", "how are you")
NEEDS_YES = {"pay", "message_client"}
SKILLS = {
    "check server": "check_server",
    "remind": "schedule_reminder",
    "pay": "pay",
    "write to client": "message_client",
}

@dataclass
class Step:
    layer: str
    action: str
    detail: str

def route(utterance: str) -> list[Step]:
    text = utterance.lower().strip()
    if any(text.startswith(p) for p in SMALL_TALK):
        return [Step("fast_model", "reply", text)]
    steps = [Step("fast_model", "reply", "got it, working on it")]
    found = False
    for part in text.split(" and "):
        for phrase, action in SKILLS.items():
            if phrase in part:
                found = True
                layer = "approval" if action in NEEDS_YES else "agent"
                steps.append(Step(layer, action, part.strip()))
    if not found:
        return [Step("fast_model", "reply", "I can't do that yet")]
    return steps

if __name__ == "__main__":
    for said in [
        "Hello there",
        "Check server and remind me at 10 about the supplier invoice",
        "Pay the supplier invoice",
        "Book me a flight to Tokyo",
    ]:
        print(f"> {said}")
        for s in route(said):
            print(f"  {s.layer:10} {s.action:18} {s.detail}")
Enter fullscreen mode Exit fullscreen mode

Output, exactly as it printed when I ran it:

> Hello there
  fast_model reply              hello there
> Check server and remind me at 10 about the supplier invoice
  fast_model reply              got it, working on it
  agent      check_server       check server
  agent      schedule_reminder  remind me at 10 about the supplier invoice
> Pay the supplier invoice
  fast_model reply              got it, working on it
  approval   pay                pay the supplier invoice
> Book me a flight to Tokyo
  fast_model reply              I can't do that yet
Enter fullscreen mode Exit fullscreen mode

Design notes

Keep the skill list next to the agents, not in the voice layer. The voice should only know what the system can do, not how. When an agent gains a new tool, one entry is added and the voice can route to it

Split before matching. People combine instructions when they speak. "Check server and remind me at 10" is two jobs with different owners: one runs now, one goes to a schedule. Treating the sentence as a single intent loses one of them

Approval is a layer, not a phrase. Saying "yes, pay it" into a microphone is not a confirmation in this design. The approval layer should send a card to a channel where the owner reads what, where and how much, because recognition can mishear a number

Honest refusal beats a guess. The flight booking line shows the most important behaviour of the whole router: when the system has no skill for a request, it says so

Errors, timeouts and trade-offs

Recognition errors: a misheard word may match the wrong skill or none. Because risky actions stop at approval, a mismatch costs a confirmation card, not a payment

Slow agents: the fast model has already replied, so the speaker waits for the result, not for the first words. Decide in advance how long to wait before saying "this is taking longer, I will send the result to the chat"

Keyword matching is crude: a production router would use the model to classify intents. The contract stays the same, though: ordered steps, explicit layers, and an approval set that is checked in one place

Testing: feed the router every phrase from a week of real voice notes and check two numbers. How many risky actions bypassed approval, which must be zero, and how many requests the system could not handle, which tells you which agent or tool to build next

Network cables in server equipment

How to reproduce the check

  1. Save the example as voice_router.py
  2. Run python3 voice_router.py with Python 3.10 or newer
  3. Compare the output with the block above
  4. Add "refund": "refund" to SKILLS and "refund" to NEEDS_YES, then route "refund the last order" and confirm it lands in the approval layer

For the business side of the same idea, including why the system has to exist before the voice, see why an AI voice agent needs a system behind it

This article was created with the help of AI.

Top comments (0)