I built a Digital Human you can talk to

Most digital-human demos start with a prepared script or audio file. I wanted a live conversation.

A user had to speak through the browser. The system had to understand the question, retrieve relevant information, generate a response, synthesize speech, animate a face, and stream the result back as video.

The notes from our 21 June 2024 demo recorded about five seconds to stream video for a short, 20-token response. We ran that demo on an AWS G5.2xlarge instance using Groq, Chroma, AWS Polly, GeneFace++, FastAPI, and WebRTC.

A separate end-to-end test on the AWS A10G measured an 18-second minimum and a 25-second average for a ten-word response. The experiment log does not contain enough detail to reconcile the difference. I treat the five-second figure as a demo observation and the end-to-end table as the repeatable benchmark.

The five-second result came from a pipeline of separate systems. Each system had its own latency, output format, failure modes, and hardware requirements.

Defining the conversation

The proof of concept supported a turn-based, two-way loop:

1. The user spoke to the avatar through a browser.
2. The browser sent the audio to the backend.
3. The backend transcribed the speech and generated an answer.
4. The avatar spoke the answer and streamed video back to the browser.

Full-duplex barge-in remained outside the PoC. The notes do not show a user interrupting the avatar mid-response and forcing the system to cancel queued speech and video frames. That requires a different session state machine and cancellation path.

The architecture

Browser microphone → FastAPI session → Deepgram VAD + STT → Groq + Chroma → AWS Polly → GeneFace++ → WebRTC video → Browser

The system processed one turn through eight stages:

1. Capture: React captured the user’s microphone input in the browser.
2. Session handling: FastAPI accepted the audio stream and maintained the conversation session.
3. Turn detection: Deepgram’s voice activity detection identified speech boundaries.
4. Transcription: Deepgram converted the recorded turn into text.
5. Retrieval and generation: Chroma retrieved relevant document chunks, and Groq generated the response.
6. Chunking: The backend split the response into units that the speech and animation stages could process without waiting for the full answer.
7. Speech and animation: AWS Polly produced audio. GeneFace++ used that audio and the trained avatar data to generate video frames.
8. Delivery: aiortc sent the frames to the browser over WebRTC.

Chroma held indexed reference material for retrieval. It did not train the language model. That distinction matters because teams often label any data connected to an LLM as training data.

The latency budget

T_visible ≈ T_endpoint + T_retrieval + T_{first token} + T_{speech chunk} + T_{avatar chunk} + T_transport

The individual benchmark values below cannot be added into one exact end-to-end total. We tested some components with different input sizes, GPUs, cloud regions, and levels of overlap. The equation shows where the delay came from and where we had to measure it.

The avatar renderer consumed much of the budget. GeneFace++ generated ten seconds of video in about five seconds on an RTX 4070. A one-second LLM response still felt slow if the system waited for a complete audio track and complete video before sending the first frame.

Text chunking gave the pipeline a way to overlap work. The speech and animation stages could process the first chunk while the LLM produced later chunks. Chunk size created a trade-off: small chunks reduced the wait but could damage prosody and create visible transitions; large chunks produced smoother output at the cost of a longer pause.

Technical specification

Layer	PoC choice	Role
Compute	AWS G5.2xlarge with NVIDIA A10G	Hosted the API and GPU avatar workload
Backend	Python 3.8 and FastAPI	Managed sessions and pipeline orchestration
Voice activity and STT	Deepgram	Detected turns and converted speech to text
LLM	Groq, with Mixtral 8x7B in the RAG tests	Generated the response
Retrieval	Local Chroma DB	Retrieved document context without a managed search service
Text-to-speech	AWS Polly	Generated the response audio
Avatar renderer	GeneFace++	Generated lip-synced video frames for the demo
Video transport	WebRTC through Python aiortc	Streamed low-latency video to the browser
Frontend	TypeScript and React with chat UI components	Captured audio and displayed the avatar conversation

The proposed architecture kept the avatar engine replaceable. We also evaluated SyncTalk and several other lip-sync systems. GeneFace++ powered the measured demo stack.

The source notes did not record the microphone codec, sample rate, chunk duration, VAD thresholds, WebRTC topology, or concurrent-session capacity. Those values belong in a production specification before another team tries to reproduce the system.

LLM and retrieval benchmarks

We averaged each result across ten runs. These numbers describe this PoC environment rather than a general ranking of the services.

Without a knowledge base

Setup	Throughput	Average latency	Observation
Groq	107 tokens/s	1.77 s	Best response time in this test; API variance remained a dependency
AWS Bedrock with Titan Express in Singapore	9.2 tokens/s	2.12 s	Easier managed setup; answer quality failed some basic test questions

With retrieval

Setup	Throughput	Average latency	Observation
Groq, Mixtral 8x7B, local Chroma	168.1 tokens/s	1.00 s	Fastest RAG configuration in the test
Groq, Mixtral 8x7B, AWS OpenSearch	9 tokens/s	11.00 s	Managed retrieval added a large delay in this setup
AWS Bedrock Claude Haiku and AWS OpenSearch in Oregon	10.46 tokens/s	10.22 s	Slowest managed configuration tested

The Chroma and OpenSearch rows include more than database time. Region, model endpoint, networking, and implementation choices also changed between configurations. The results support a decision about this PoC stack; they do not isolate the retrieval engine as the sole cause.

Text-to-speech benchmarks

Engine	Generation latency	Test input	Generated audio	Notes recorded during the evaluation
AWS Polly	1.25 s	337 characters	17.1 s	Fast managed API; limited custom-voice options for our use case
ElevenLabs	4.73 s	337 characters	17.1 s	Custom voice support; slower in this test
Picovoice	2.00 s	337 characters	22 s	Strong inference speed; custom voice required vendor coordination
Speechify	1.52 s	337 characters	26 s	Custom voice and good perceived quality
Deepgram	1.41 s	337 characters	17.1 s	Custom voice options, pause control, and local-hosting potential

AWS Polly gave us the clearest path to the demo because it generated a 17.1-second clip in 1.25 seconds and required no voice-model training.

Lip-sync benchmarks

We ran these tests on an RTX 4070 and measured the time required to generate ten seconds of video.

Framework	Generation time	Evaluation note
Wav2Lip	4 s	Fast setup, lower resolution, and no head movement in our test
GeneFace++	5 s	Fast inference and better lip movement; model training and environment setup took more work
SyncTalk	10 s	Best lip movement and image quality in our evaluation
Real3DPortrait	29 s	Incorrect head, lip, and torso movement in our test
SadTalker	45 s	Slow inference and weak lip sync for this use case
Hallo	More than 10 min	Natural body movement, but the inference time ruled it out for conversation

The evaluation notes also recorded commercial-use concerns for some frameworks. Licences and repository terms can change, so any production team should verify the current terms before selecting an engine.

End-to-end GPU results

We used Groq, AWS Polly, and GeneFace++ for the end-to-end GPU comparison. Prices show the hourly figures recorded during the 2024 evaluation.

Provider and GPU	Recorded price/hour	Minimum response time for 10 words	Inference speed	Average response time
AWS G5.2xlarge, A10G, Sydney	$1.212	18 s	45 it/s	25 s
Runpod, RTX 4090, Slovakia	$0.74	4.6 s	130 it/s	11.26 s
Runpod, RTX 4090, Romania	$0.74	5.6 s	123 it/s	11.44 s
Runpod, H100, Netherlands	$4.96	6 s	86 it/s	9.14 s

The RTX 4090 produced the shortest minimum response and the highest iteration rate in this workload. The H100 delivered the lowest average response time. Its recorded hourly price exceeded the 4090 instances by more than six times. Model architecture, clock speed, software optimisation, and provider variance affected the outcome. The GPU name alone did not predict the best user experience.

Why WebRTC

The avatar needed a browser-native path with low delay. HLS introduced segmenting and caching delays. RTMP lacked direct browser playback. Downloading complete files produced broken transitions between responses.

WebRTC handled media delivery in the browser and supported a future path toward two-way media. Python’s aiortc let us keep the media server in the same language as the AI pipeline, though it added setup work around tracks, timing, and connection state.

Production changes I would make

The PoC proved the conversation loop. A production version needs more control around cancellation, load, and measurement.

Add a session state machine

Each session should move through LISTENING, TRANSCRIBING, THINKING, and SPEAKING. A new user turn should invalidate stale LLM, TTS, and frame-generation jobs.

Introduce bounded queues and backpressure

The LLM can generate text faster than the avatar can render it. Unbounded audio and frame queues increase memory use and make interruptions feel broken.

Measure first-frame latency

Total response time hides the pause the user feels. I would record speech-end to transcript, first token, first audio chunk, first video frame, and playback start, with P50 and P95 values.

Separate GPU workers from the API

FastAPI should manage sessions while dedicated workers handle avatar inference. That separation makes GPU scheduling and horizontal scaling easier to control.

Design for barge-in

Full-duplex conversation needs interruption detection, job cancellation, audio ducking, and a clean return to the listening state. The session controller must drop stale frames from the previous response after the user starts a new turn.

Synchronise with timestamps

The media layer should align audio samples and video presentation timestamps. Queue length alone cannot preserve lip sync under network jitter or variable inference time.

What I learned

End-to-end latency followed the slowest media stage. The one-second RAG result helped, while the user still waited for speech generation, avatar inference, and the first playable WebRTC frame.

Streaming changed the useful metric. The system did not need to finish the whole response before it could begin playback. It needed to deliver the first coherent audio-and-video chunk, then keep producing chunks faster than the browser consumed them.

The hardware tests also challenged a common assumption. The most expensive GPU did not produce the shortest first response in our stack. Testing the complete pipeline gave us a better answer than choosing hardware from model specifications.

The PoC gave us a working digital-human conversation and a clear list of the remaining engineering work. The next version would focus on barge-in, cancellation, percentile latency, and concurrent sessions.

Demo

June 2024 digital-human proof of concept

---

Benchmark figures in this post come from the PoC tests recorded in June 2024. Cloud pricing, model endpoints, framework licences, and service performance may have changed since the evaluation.