A production voice agent has to do more than understand a request. It must respond within the rhythm of a phone conversation, coordinate live APIs, protect sensitive data, and transfer the caller without losing context when automation reaches its limit. In our travel booking implementation, that meant completing the full shop, select, pay, book, and confirm workflow through an existing Amazon Connect environment. The system reached a 92% successful end-to-end transaction rate using live APIs and saved 10 to 15 minutes per successful automated booking.
The useful lesson was not simply that a large language model could handle a booking. The result depended on decisions around the model: how work was divided, how latency was masked, where sensitive operations stopped, and when a person took over.
Add AI as a path through the existing system
The client already had Amazon Connect and the APIs required to complete a reservation. We added conversational AI as a new path inside that environment instead of replacing the contact center or recreating its transactional systems.
Amazon Lex transcribes the caller's freeform request and identifies the basic intent. Claude Haiku handles quick classification, while Claude Sonnet performs the deeper reasoning required to compare options and run the booking APIs. The agent can search live inventory, discuss alternatives, adjust to caller preferences, send a secure payment link by text, confirm the reservation through the client's Payment API, and trigger the confirmation email while the caller remains on the line.
This approach keeps the AI layer focused on conversation and coordination. Existing systems remain responsible for the operations they already perform. For teams working with mature contact center infrastructure, that is a useful boundary: first identify the smallest new path that can orchestrate the current systems, then replace infrastructure only when the existing components block the required workflow.
Treat latency as an end-to-end budget
Voice exposes delays that would be less disruptive in a text interface. Our end-to-end response latency target was less than 2 seconds, but transcription and text-to-speech already consumed 1 to 2 seconds before the model had much room to respond. The implemented response time was 2 to 3 seconds.
True streaming was not ready for this implementation, so we used a practical workaround. The model streamed its reply into DynamoDB. Amazon Lex checked the stored output every few hundred milliseconds and began speaking once it found complete sentences. This was not true streaming, but it reduced the silence the caller experienced and allowed speech to sound natural.
The transferable principle is to budget latency across the whole turn, not just model inference. Measure speech recognition, orchestration, API calls, model generation, buffering, and speech synthesis as one user experience. When a platform capability is missing, optimize the part the caller perceives. Here, sentence-level delivery mattered more than waiting for a technically ideal streaming path.
Choose orchestration and models by workload
We began with LangChain because it handled tool calling and context well. When AWS introduced Bedrock AgentCore as part of the Strands framework, we moved the orchestration into that stack. Its built-in memory management simplified session handling, and tighter integration with Amazon Connect, AWS Lambda, and DynamoDB reduced the infrastructure the team had to maintain. Keeping orchestration within AWS also matched the client's requirement that the system stay in its AWS environment.
Model selection followed the same workload-specific reasoning. We tested newer AWS models for speed and integration, but they did not provide the quality required for complex, multistep booking logic. Claude Haiku remained the fast classifier, and Claude Sonnet handled option selection and API execution.
This split is more useful than choosing one model for every turn. Classification and transactional reasoning have different latency and quality requirements. Test them as separate workloads, then assign each to the least expensive or fastest model that still meets its acceptance criteria. A newer model or a tighter platform integration is not an upgrade if it weakens the critical path.
Keep sensitive operations outside the AI boundary
The agent creates a payment link, but it never handles payment data. Customer identity is verified through the client's own lookup system, and payment remains behind the existing secure Payment API. Prompt guardrails prevent the model from storing or repeating sensitive details beyond the current call. Conversation state is stored in DynamoDB, while policies and rules are retrieved from a Bedrock Knowledge Base.
That separation narrows what the model can expose or mishandle. The model can guide the transaction and invoke approved operations without becoming the system of record for identity or payment. For transactional agents, define this boundary before refining prompts: list what the model may interpret, what it may initiate, and what must remain entirely inside trusted systems.
Automation also needs a designed exit. If the agent could not resolve a request within 2 turns, it transferred the call to a human agent with the conversation context and identified customer account intact. The caller did not have to repeat the request.
This makes escalation part of the architecture rather than an exception. A useful handoff policy needs a trigger, the context to transfer, and a destination capable of continuing the workflow. Together, those pieces let routine calls stay automated while disputes, irregular operations, and distressed customers reach people equipped to help.
Top comments (0)