Problem: a voice front end that sends every sentence to a heavy agent feels slow, and one that answers everything itself does no real work. Result of this article: a small, testable router that decides, for each spoken instruction, whether the fast conversational model replies, an agent with tools does the work, or the action stops for human approval. Stack: plain Python 3.10+ (tested on 3.15.2), no external packages. Boundaries: speech recognition, text to speech and the agents themselves are out of scope; the router works on text that recognition has already produced
Where the pattern comes from
I built a voice agent called Ami for the owner of an aesthetic medicine clinic. The owner opens a private page, presses Start and talks. Small talk goes to a fast model so the conversation has no pauses. Real requests, such as checking the server, go to the clinic's agent with tools, and the answer comes back as speech. Memory is shared with the Telegram assistant. When the voice arrived, the clinic already had a team of 6 agents, so the voice had someone to hand work to
My own agent, Jes, works the same way on the input side: voice notes and text are treated alike, and behind her are 10 scheduled jobs and 7 other bots. Since July she has passed on 165 tasks over 134 days
The code below is a simplified, standalone teaching version of the routing idea. It is not production code from either system
Input, processing, output
Input: one recognised utterance as text
Processing: small talk is answered by the fast model straight away. Anything else gets an immediate acknowledgement from the fast model, then is split into parts and matched against the skills the system actually has. Each matched skill goes to the agent layer, unless it is in the set that needs a human yes, in which case it goes to the approval layer. If nothing matches, the fast model says so honestly
Output: an ordered list of steps, each with a layer, an action and the part of the sentence it came from
Two rules matter more than the matching itself. First, the acknowledgement comes before the work, so the speaker never sits in silence while tools run. Second, an unknown request produces "I can't do that yet" rather than a confident reply with nothing behind it
The example
from dataclasses import dataclass
SMALL_TALK = ("hello", "thanks", "how are you")
NEEDS_YES = {"pay", "message_client"}
SKILLS = {
"check server": "check_server",
"remind": "schedule_reminder",
"pay": "pay",
"write to client": "message_client",
}
@dataclass
class Step:
layer: str
action: str
detail: str
def route(utterance: str) -> list[Step]:
text = utterance.lower().strip()
if any(text.startswith(p) for p in SMALL_TALK):
return [Step("fast_model", "reply", text)]
steps = [Step("fast_model", "reply", "got it, working on it")]
found = False
for part in text.split(" and "):
for phrase, action in SKILLS.items():
if phrase in part:
found = True
layer = "approval" if action in NEEDS_YES else "agent"
steps.append(Step(layer, action, part.strip()))
if not found:
return [Step("fast_model", "reply", "I can't do that yet")]
return steps
if __name__ == "__main__":
for said in [
"Hello there",
"Check server and remind me at 10 about the supplier invoice",
"Pay the supplier invoice",
"Book me a flight to Tokyo",
]:
print(f"> {said}")
for s in route(said):
print(f" {s.layer:10} {s.action:18} {s.detail}")
Output, exactly as it printed when I ran it:
> Hello there
fast_model reply hello there
> Check server and remind me at 10 about the supplier invoice
fast_model reply got it, working on it
agent check_server check server
agent schedule_reminder remind me at 10 about the supplier invoice
> Pay the supplier invoice
fast_model reply got it, working on it
approval pay pay the supplier invoice
> Book me a flight to Tokyo
fast_model reply I can't do that yet
Design notes
Keep the skill list next to the agents, not in the voice layer. The voice should only know what the system can do, not how. When an agent gains a new tool, one entry is added and the voice can route to it
Split before matching. People combine instructions when they speak. "Check server and remind me at 10" is two jobs with different owners: one runs now, one goes to a schedule. Treating the sentence as a single intent loses one of them
Approval is a layer, not a phrase. Saying "yes, pay it" into a microphone is not a confirmation in this design. The approval layer should send a card to a channel where the owner reads what, where and how much, because recognition can mishear a number
Honest refusal beats a guess. The flight booking line shows the most important behaviour of the whole router: when the system has no skill for a request, it says so
Errors, timeouts and trade-offs
Recognition errors: a misheard word may match the wrong skill or none. Because risky actions stop at approval, a mismatch costs a confirmation card, not a payment
Slow agents: the fast model has already replied, so the speaker waits for the result, not for the first words. Decide in advance how long to wait before saying "this is taking longer, I will send the result to the chat"
Keyword matching is crude: a production router would use the model to classify intents. The contract stays the same, though: ordered steps, explicit layers, and an approval set that is checked in one place
Testing: feed the router every phrase from a week of real voice notes and check two numbers. How many risky actions bypassed approval, which must be zero, and how many requests the system could not handle, which tells you which agent or tool to build next
How to reproduce the check
- Save the example as
voice_router.py - Run
python3 voice_router.pywith Python 3.10 or newer - Compare the output with the block above
- Add
"refund": "refund"toSKILLSand"refund"toNEEDS_YES, then route "refund the last order" and confirm it lands in the approval layer
For the business side of the same idea, including why the system has to exist before the voice, see why an AI voice agent needs a system behind it
This article was created with the help of AI.



Top comments (0)