OpenAI’s current Realtime documentation names three live-session tool paths: function tools, remote MCP servers, and connectors.
It does not verify hosted web search or sandboxed Python inside an active speech session.
Start with the documented boundary
Begin with evidence, not release language. Function tools, remote MCP servers, and connectors are supported during live conversations. The cited documentation does not support the more specific claim that a realtime voice agent can search the web or run sandboxed Python inside an active WebSocket speech session without custom proxy handling.
That is why this design should be treated as a proposal rather than a shipped capability. The distinction affects technical planning. If a team assumes hosted execution already exists, it may design around a runtime and permission boundary that the available source does not establish.
A useful architecture starts by labeling each component as documented, proposed, or application-owned. That keeps an implementation plan from quietly turning an aspiration into an API dependency.
Define what native execution would change
A voice workflow that needs current information or a calculated answer may route a tool request through custom proxy logic that pauses and resumes audio. That path adds latency and state handling.
A native hosted-tool design could remove that specific detour and shorten the route between a caller’s question and a spoken answer. It would not remove the need for an application backend. Teams would still be accountable for which tool runs, what it can access, what it costs, and how its behavior is reviewed.
The practical distinction is simple: native hosting could change where execution happens, but not who owns the decision.
Give search and code separate jobs
Web search fits questions that require current public information. Controlled code execution fits calculations and transformations. Treating them as interchangeable creates an unnecessarily broad permission surface.
For a proposed search tool, define when public information is actually needed and how tool accuracy will be evaluated. For a proposed code tool, constrain the operation to the calculation or transformation the conversation requires. Neither tool should become a route to unrestricted secrets, production credentials, or direct production-system access.
This separation also improves evaluation. A search case can test whether the right public information was retrieved. A calculation case can test whether the intended transformation was performed. Both can test whether the agent chose the right tool at all.
Use production gates that reflect live speech
A tool call that works in isolation is not enough. Production readiness depends on the behavior of the entire spoken interaction. Require evidence for these checks:
- Latency: confirm the tool path keeps the conversation usable.
- Barge-in recovery: verify the session recovers when the caller interrupts.
- Tool accuracy: inspect both the selected tool and the returned result.
- Cost: observe the operational effect of tool use.
- Token budgets: enforce limits throughout the interaction.
- Logs: preserve enough detail to review each tool decision.
- Held-out evaluations: test behavior beyond the cases used during development.
These checks belong together. A correct calculation that arrives too late can still fail the conversation. Fast retrieval with poor tool selection can produce a confidently spoken error. Good average behavior can also conceal a weak failure path if evaluation covers only expected prompts.
Keep the security boundary explicit
A live voice agent must never receive unrestricted secrets, production credentials, or direct production-system access. A sandbox label, if such execution becomes available, would not answer the authorization question by itself.
Map permissions before connecting a tool. Record the tool decision in logs. Define review gates for changes to tool scope. Document what happens when access is denied, a tool fails, the token budget is reached, or the caller interrupts the response.
This is application governance, even when part of execution is hosted. Less proxy plumbing does not make the resulting system self-governing.
Apply an operator lens
At Van Data Team, we start by mapping evidence, permissions, review gates, and failure paths. Vanaxity applies that method across research, writing, illustration, publishing, and syndication. The same discipline is useful here because a voice answer can compress tool selection, execution, and presentation into a single conversational moment.
Native hosted tools, if verified, could make that moment faster and less fragile. They would also make weak controls less visible behind a smoother interaction. The bar for observability should therefore rise as orchestration becomes simpler.
If you were designing the first held-out evaluation for this proposed tool path, would you prioritize barge-in recovery, tool-selection accuracy, or permission denial—and why?
📖 Read the full guide → Realtime Voice Agents and Proposed Native Tools
Top comments (0)