Building a useful voice agent connects media, speech, reasoning, business
actions, security controls, data retention, and teardown.
I tested this with a fictional restaurant-ordering agent. A real
phone call entered through LiveKit SIP, reached
a LiveKit worker, used Cartesia for speech
recognition and synthesis, and asked a text model to manage the conversation.
After the caller confirmed the order, the worker sent a signed request to an
API. A small AWS Lambda
function validated it, encrypted the contact details, and wrote the simulated
order to DynamoDB.
The call produced an API 201 response and a new encrypted test record. That
was the first point at which I considered the agent meaningful. The result
demonstrated a completed action, a durable record, and the privacy and security
boundaries I needed to examine around that action.
The working unit is a chain of boundaries. The model is one component in that
chain.
The System Behind the Voice
The working shape has a live conversation path and a separate durable-write
path:
LIVE CONVERSATION
Caller speaks
-> LiveKit SIP and room
-> Cartesia STT: audio to text
-> LiveKit worker + OpenRouter LLM: Gemini 2.5 Flash Lite
- menu tools
- per-call cart: items, name, pickup time, contact choice, order ID
-> Cartesia TTS: response text to audio
-> LiveKit SIP and room
-> Caller hears the response
DURABLE WRITE, AFTER READ-BACK CONFIRMATION
LiveKit worker cart
-> signed HTTPS order request
-> API Gateway
-> Lambda: validate catalog, timestamp, nonce, and signature
-> KMS: encrypt contact details
-> DynamoDB: simulated order record
The cart stays in the worker's memory during the call. The worker builds it
through validated order tools and reads it back to the caller. Cartesia converts
caller audio to text before the model acts and converts the model's response
text back to streaming audio.
The text-model layer is separate from the speech layer. The phone test used
Gemini 2.5 Flash Lite through
OpenRouter for the
ordering dialogue and tool selection. OpenRouter lists the model at $0.10 per
million input tokens and $0.40 per million output tokens, which made it a
low-cost choice for this controlled text workload. Cartesia handled speech
recognition and synthesis, so the LLM did not process the live audio stream.
An OpenRouter-hosted
Qwen3-8B remains a concrete
open-weight, Apache-2.0 alternative. I did not test that substitution in this
phone flow. Changing models requires a fresh conversation-quality and
tool-reliability evaluation.
Each arrow is a different contract. The phone layer determines how a caller is
routed. The media layer determines when the agent hears and speaks. The model
decides whether the conversation has enough information to act. The tool
boundary determines whether a spoken request becomes a durable write. The
data boundary determines what survives after the call.
The visible conversation relies on deliberate design for identity,
confirmation, validation, storage, observability, and deletion.
Restaurant ordering is a credible voice-agent workflow because it has a bounded
catalog, a spoken interaction, a clear confirmation point, and an action a
backend can validate. LiveKit publishes a restaurant-agent
recipe and a
drive-through example
with order state and database-backed order management.
My test used an external HTTPS gateway so the worker could invoke one narrow
write operation without cloud credentials. The request carried an HMAC
signature over the exact body, a timestamp, a single-use nonce, and a random
order ID. The API was Internet-reachable and accepted any request with that
shared secret. This was a request-integrity pattern for one controlled test,
not per-worker identity, user authorization, or a complete production security
design.
The Lambda validated catalog values and request integrity, encrypted the
contact details, and wrote the simulated order. Its DynamoDB policy allowed
writes to the dedicated tables and excluded scan, query, delete, and unrelated
data access. The function also needed narrowly scoped KMS permissions to
encrypt the contact data.
Confirmation Is Part of the Write Path
The conversational path needed an explicit finalization rule.
The agent collected menu items, a pickup time, and a name. The worker's final
tool required a read-back flag and complete fields before it submitted the
order. I also separated saving order details from finalizing an order after the
model retried a finalization attempt with missing fields.
This is a useful pattern for any voice action with consequences:
Collect -> summarize -> caller confirms -> validate -> persist -> return confirmation
In this test, the read-back flag came from the model's tool call. It was a
conversational guard, not independently verifiable evidence of caller consent.
The server validated the catalog, quantities, and other business rules again.
A production write path needs a durable confirmation event that the server can
verify independently.
LiveKit's guidance on agent tools
captures one part of the pattern: tools need narrow, clear purposes. The
larger lesson is that a tool is a policy boundary. Its input schema, validation,
idempotency behavior, and audit trail are product decisions.
Privacy Across the System
The test identified a privacy limit.
The full caller-number attribute entered the worker process for inbound SIP
calls. The agent spoke only the last four digits, then asked whether that
number should be recorded or whether the caller wanted to provide another one.
When selected, the full number went to the order gateway. Caller ID is routing
metadata and personal data, not identity proof or authority to disclose
account information, take payment, or change an account.
In the August 6 test, application API and Lambda logs were configured not to
contain the caller's name or phone number. The encrypted DynamoDB record did
not expose either field in plaintext. I also observed a LiveKit framework
lifecycle log with a SIP participant identity containing the caller number.
That observation applies to this test configuration. Hosted-runtime metadata
and retention settings remain part of the privacy model.
That observation expands the architecture discussion. A privacy review tracks
where data appears at every stage:
| Boundary | Question |
|---|---|
| Telephony | Does caller identity enter the room, recording, or provider metadata? |
| Worker | Which participant attributes and secrets can the agent process read? |
| Model | Which spoken details reach the model provider and its logs? |
| API | Are requests authenticated, replay-resistant, and free of unnecessary PII in logs? |
| Storage | What is encrypted, which key is used, and when does the record expire? |
| Operations | Who can access traces, transcripts, recordings, and lifecycle logs? |
Caller speech can reach Cartesia before the model acts. Names, order details,
and an alternate contact number can then reach the configured text-model
provider. This happens before the AWS encryption boundary. Every row can
retain data. Every row needs a retention decision.
LiveKit documents that
HidePhoneNumber suppresses the sip.phoneNumber participant attribute. In
that configuration, the agent asks the caller to provide a contact number. The
setting applies to that documented attribute and does not establish the absence
of all caller-related provider metadata. LiveKit Cloud's agent observability
documentation treats transcripts,
recordings, traces, and runtime logs as session data, with a documented
30-day retention window. Its Build plan can retain anonymized session data
longer for model improvement. Cartesia offers zero data
retention for its
STT and TTS inference APIs on eligible enterprise plans, while retaining
operational metadata. This test did not establish that entitlement. Those
controls are part of the architecture decision, not deployment settings to
address later.
Teardown Is Also an Architecture Boundary
The system also needs a defined deletion path.
This test required a dedicated phone route, worker, API, Lambda function,
DynamoDB tables, secret, encryption key, and logs. The teardown plan removes
the route before the number, removes the API before its execution path becomes
orphaned, deletes the data and secrets, and schedules the encryption key for
deletion after its recovery period. It also records what was removed and what
will disappear later.
That work belongs in the design from the first call. It limits cost, reduces
the chance that test data persists unseen, and gives the architecture a
defined end state.
So What
The operating question is whether the system can complete a task with bounded
authority and an understood data trail.
LiveKit supplied the phone and real-time media layer. Cartesia supplied speech
components. AWS supplied the validation, encryption, and durable-write
boundary. The layers sit behind defined interfaces, but changing any one of
them requires a fresh integration, privacy, and quality evaluation.
This was a controlled technical pattern, not a restaurant deployment or a
compliance determination. A production team needs channel- and
jurisdiction-specific review of recording, privacy, caller-ID, disclosure,
retention, access, deletion, and escalation requirements. It also needs a
written retention and access review for the telephony provider, hosted agent
runtime, speech provider, text-model provider, and its own observability
systems.
That review, alongside confirmation and durable-write controls, determines
whether a voice agent is ready to support a real workflow.
I have not resolved the right way to prove caller confirmation independently
without making the conversation less natural. That boundary remains open.
Top comments (0)