DEV Community

cheng zhang
cheng zhang

Posted on

Gemini Live API Async Function Calling for Real-Time Voice Agents

#ai #security #gemini #functional

Article Summary

One of the biggest usability problems in real-time voice agents is tool latency. If the model needs to query CRM, orders, databases, search, or ticketing systems and every function call blocks the conversation, the user experiences several seconds of silence. Gemini Live API supports asynchronous function calling in its cascaded architecture. A function can be marked NON_BLOCKING, allowing the live conversation to continue while the tool executes in the background. This does not mean the model can invent tool-dependent facts. It means developers need explicit task state, timeouts, cancellation, concurrency controls, result correlation, permissions, and idempotency.

1. Why tool latency hurts voice agents

Text users may tolerate a few seconds. Voice users notice silence immediately.

A normal live interaction can include speech understanding, CRM, databases, search, model generation, and audio output. External systems have unpredictable latency.

If every tool call blocks the session, natural conversation disappears.

2. Blocking function calls

A traditional flow is:

user asks
→ model calls get_refund_status
→ conversation pauses
→ backend waits four seconds
→ result returns
→ model resumes
Enter fullscreen mode Exit fullscreen mode

The user hears a long gap.

That may be acceptable for some transactional workflows, but it is poor conversational design.

3. What NON_BLOCKING changes

With asynchronous function calling:

user request
→ model starts tool call
├── tool executes in background
└── conversation continues
→ result returns
→ model incorporates result
Enter fullscreen mode Exit fullscreen mode

The agent can ask a clarification or explain the process while waiting.

4. Asynchronous does not mean speculative

If the refund status has not returned, the agent must not claim that the refund is complete.

Separate state into:

known
pending
unknown
Enter fullscreen mode Exit fullscreen mode

Only confirmed data should be presented as fact.

5. Good non-blocking tools

Useful candidates include customer lookup, order status, tickets, inventory, shipping, knowledge retrieval, web search, recommendations, and other independent read operations.

Several independent read calls can also execute in parallel.

6. Poor non-blocking candidates

Be cautious with payments, deletes, production changes, irreversible operations, and any eligibility decision that is required before downstream logic can continue.

These often require blocking behavior and explicit approval.

7. Recommended architecture 

microphone
→ Gemini Live session
→ tool router
   ├── blocking
   ├── non-blocking
   └── approval-required
→ async task manager
   ├── timeout
   ├── retry
   ├── cancellation
   └── idempotency
→ enterprise APIs
→ tool result
→ live session
Enter fullscreen mode Exit fullscreen mode

Add authorization, tracing, cost controls, and audit logs.

8. Tool policy belongs in the application

Maintain metadata such as:

name: get_refund_status
mode: non_blocking
timeout_ms: 5000
retry: 1
idempotent: true
risk: read
requires_confirmation: false
Enter fullscreen mode Exit fullscreen mode

For a financial write:

name: create_refund
mode: blocking
timeout_ms: 10000
retry: 0
idempotent: true
risk: financial
requires_confirmation: true
Enter fullscreen mode Exit fullscreen mode

The model should not classify business risk by itself.

9. Conceptual Gemini declaration

Exact syntax should follow the active SDK version, but conceptually:

tool = {
    "function_declarations": [
        {
            "name": "get_order_status",
            "description": "Get current order status",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string"}
                },
                "required": ["order_id"]
            },
            "behavior": "NON_BLOCKING"
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

The important behavior is that the live session does not have to stop while the function runs.

10. Use an async task manager

Do not block a WebSocket callback with synchronous HTTP work.

Instead:

async def execute_tool(call):
    task_id = create_task_id(call)
    task = asyncio.create_task(run_with_timeout(call))
    registry[task_id] = task
Enter fullscreen mode Exit fullscreen mode

When it completes:

async def on_tool_done(task_id):
    result = await registry[task_id]
    await send_tool_result(result)
Enter fullscreen mode Exit fullscreen mode

11. Every tool needs a timeout

External services may respond in 300 milliseconds, two seconds, eight seconds, or never.

A timed-out task should return structured state instead of remaining pending forever.

12. Continue the conversation during delays

Instead of dead air, the agent can say:

“The customer system is taking a little longer. I can confirm a few details while it finishes.”

This is one of the main user-experience benefits of asynchronous tools.

13. Limit parallelism

A model may want to call CRM, orders, tickets, payments, email, and analytics at once.

Set a practical limit such as:

max_parallel_tools = 3
Enter fullscreen mode Exit fullscreen mode

Queue the rest.

This protects downstream systems and keeps state manageable.

14. Correlate results correctly

Store:

session_id
call_id
tool
arguments
start time
conversation turn
Enter fullscreen mode Exit fullscreen mode

When a result returns, match the call ID, confirm that the session is still valid, decide whether the result remains relevant, and only then deliver it back to the live model.

15. Handle topic changes

The user may request order A and then immediately correct the request to order B.

If possible, cancel A. If cancellation is impossible, mark its result stale and do not inject it into the active conversation.

Useful states include:

PENDING
RUNNING
COMPLETED
TIMEOUT
FAILED
CANCELLED
STALE
Enter fullscreen mode Exit fullscreen mode

16. Use idempotency for writes

Async systems create retry ambiguity. A remote write may succeed while the client loses the response.

Use an idempotency key for side-effecting operations such as ticket creation or order changes. The target service should guarantee one execution per key.

17. User interruptions must stop current speech

A real voice agent must support barge-in.

When a user interrupts, stop current audio, process the new input, decide whether existing tool tasks remain relevant, cancel or mark them stale, and continue with the updated goal.

18. What the agent can do while waiting

Useful actions include asking clarifying questions, collecting identity details, explaining process, and running independent read tools.

The agent should never invent pending results, promise unconfirmed outcomes, or claim that an operation completed before confirmation.

A useful instruction is:

Never state that a tool-dependent fact is confirmed
until the corresponding tool result is received.
Enter fullscreen mode Exit fullscreen mode

19. Authorization remains mandatory

A model-generated tool call is not authorization.

The application still needs:

user identity
→ role
→ resource permission
→ tool permission
→ argument validation
→ execution
Enter fullscreen mode Exit fullscreen mode

Otherwise the voice agent becomes an authorization bypass.

20. Minimize sensitive tool output

If a CRM result contains identity documents, bank information, contact details, and order state while the user asked only for order status, return only the necessary field to the model.

Minimizing model exposure is a basic production principle.

21. Observability

Track:

session_id
call_id
tool
mode
start
end
latency
status
retry count
cancellation
token use
business result
Enter fullscreen mode Exit fullscreen mode

Important metrics include tool P50/P95 latency, timeout rate, cancellation rate, parallel call count, task success, first-audio latency, and conversation silence time.

22. Conversation silence time is a key metric

Backend tool latency alone does not describe user experience.

A tool may take four seconds, but if the agent keeps the conversation useful, the user may experience less than a second of dead air.

Measure silence, not only backend latency.

23. Customer-support example

A customer asks why a package has not arrived.

The agent starts shipping lookup while asking whether this is the same order discussed yesterday.

The user responds while the tools execute.

When the result returns, the agent provides the confirmed shipping status.

The backend was slow, but the conversation never fully stopped.

24. Blocking versus non-blocking policy

A practical classification is:

READ_FAST
READ_SLOW
WRITE_LOW_RISK
WRITE_HIGH_RISK
FINANCIAL
PRODUCTION
Enter fullscreen mode Exit fullscreen mode

Possible defaults:

READ_FAST → NON_BLOCKING
READ_SLOW → NON_BLOCKING + status
WRITE_LOW_RISK → confirmation
WRITE_HIGH_RISK → BLOCKING + confirmation
FINANCIAL → BLOCKING + strong approval
PRODUCTION → BLOCKING + multi-party approval
Enter fullscreen mode Exit fullscreen mode

25. Launch testing

Test slow responses, timeouts, disconnects, retries, out-of-order results, user interruptions, requirement changes, cancellations, authorization failures, prompt injection, sensitive data, and duplicate writes.

The final business state must remain correct even when the conversation is interrupted.

Conclusion

Gemini Live API asynchronous function calling addresses a fundamental real-time-agent problem:

slow tools should not automatically make the conversation slow.

With non-blocking calls, an agent can continue clarification, explanation, and independent work while external systems execute.

But the architecture needs task state, timeouts, cancellation, call IDs, concurrency controls, idempotency, authorization, sensitive-data filtering, and observability.

The central rule is simple: the agent may continue speaking while a tool runs, but it must never pretend to know a tool-dependent result before that result arrives.

For more practical Gemini API, voice-agent, function-calling, and production AI engineering guides, visit Zyentor Picks: https://www.zyentorpicks.com/.

Originally published on Zyentor Picks.

Top comments (0)