DEV Community: Gokula Krishna

I built a Digital Human you can talk to

Gokula Krishna — Mon, 03 Aug 2026 09:07:58 +0000

Most digital-human demos start with a prepared script or audio file. I wanted a live conversation.

A user had to speak through the browser. The system had to understand the question, retrieve relevant information, generate a response, synthesize speech, animate a face, and stream the result back as video.

The notes from our 21 June 2024 demo recorded about five seconds to stream video for a short, 20-token response. We ran that demo on an AWS G5.2xlarge instance using Groq, Chroma, AWS Polly, GeneFace++, FastAPI, and WebRTC.

A separate end-to-end test on the AWS A10G measured an 18-second minimum and a 25-second average for a ten-word response. The experiment log does not contain enough detail to reconcile the difference. I treat the five-second figure as a demo observation and the end-to-end table as the repeatable benchmark.

The five-second result came from a pipeline of separate systems. Each system had its own latency, output format, failure modes, and hardware requirements.

Defining the conversation

The proof of concept supported a turn-based, two-way loop:

The user spoke to the avatar through a browser.
The browser sent the audio to the backend.
The backend transcribed the speech and generated an answer.
The avatar spoke the answer and streamed video back to the browser.

Full-duplex barge-in remained outside the PoC. The notes do not show a user interrupting the avatar mid-response and forcing the system to cancel queued speech and video frames. That requires a different session state machine and cancellation path.

The architecture

Browser microphone → FastAPI session → Deepgram VAD + STT → Groq + Chroma → AWS Polly → GeneFace++ → WebRTC video → Browser

The system processed one turn through eight stages:

Capture: React captured the user’s microphone input in the browser.
Session handling: FastAPI accepted the audio stream and maintained the conversation session.
Turn detection: Deepgram’s voice activity detection identified speech boundaries.
Transcription: Deepgram converted the recorded turn into text.
Retrieval and generation: Chroma retrieved relevant document chunks, and Groq generated the response.
Chunking: The backend split the response into units that the speech and animation stages could process without waiting for the full answer.
Speech and animation: AWS Polly produced audio. GeneFace++ used that audio and the trained avatar data to generate video frames.
Delivery: aiortc sent the frames to the browser over WebRTC.

Chroma held indexed reference material for retrieval. It did not train the language model. That distinction matters because teams often label any data connected to an LLM as training data.

The latency budget

Tvisible ≈ Tendpoint + Tretrieval + Tfirst token + Tspeech chunk + Tavatar chunk + Ttransport

The individual benchmark values below cannot be added into one exact end-to-end total. We tested some components with different input sizes, GPUs, cloud regions, and levels of overlap. The equation shows where the delay came from and where we had to measure it.

The avatar renderer consumed much of the budget. GeneFace++ generated ten seconds of video in about five seconds on an RTX 4070. A one-second LLM response still felt slow if the system waited for a complete audio track and complete video before sending the first frame.

Text chunking gave the pipeline a way to overlap work. The speech and animation stages could process the first chunk while the LLM produced later chunks. Chunk size created a trade-off: small chunks reduced the wait but could damage prosody and create visible transitions; large chunks produced smoother output at the cost of a longer pause.

Technical specification

Layer	PoC choice	Role
Compute	AWS G5.2xlarge with NVIDIA A10G	Hosted the API and GPU avatar workload
Backend	Python 3.8 and FastAPI	Managed sessions and pipeline orchestration
Voice activity and STT	Deepgram	Detected turns and converted speech to text
LLM	Groq, with Mixtral 8x7B in the RAG tests	Generated the response
Retrieval	Local Chroma DB	Retrieved document context without a managed search service
Text-to-speech	AWS Polly	Generated the response audio
Avatar renderer	GeneFace++	Generated lip-synced video frames for the demo
Video transport	WebRTC through Python `aiortc`	Streamed low-latency video to the browser
Frontend	TypeScript and React with chat UI components	Captured audio and displayed the avatar conversation

The proposed architecture kept the avatar engine replaceable. We also evaluated SyncTalk and several other lip-sync systems. GeneFace++ powered the measured demo stack.

The source notes did not record the microphone codec, sample rate, chunk duration, VAD thresholds, WebRTC topology, or concurrent-session capacity. Those values belong in a production specification before another team tries to reproduce the system.

LLM and retrieval benchmarks

We averaged each result across ten runs. These numbers describe this PoC environment rather than a general ranking of the services.

Without a knowledge base

Setup	Throughput	Average latency	Observation
Groq	107 tokens/s	1.77 s	Best response time in this test; API variance remained a dependency
AWS Bedrock with Titan Express in Singapore	9.2 tokens/s	2.12 s	Easier managed setup; answer quality failed some basic test questions

With retrieval

Setup	Throughput	Average latency	Observation
Groq, Mixtral 8x7B, local Chroma	168.1 tokens/s	1.00 s	Fastest RAG configuration in the test
Groq, Mixtral 8x7B, AWS OpenSearch	9 tokens/s	11.00 s	Managed retrieval added a large delay in this setup
AWS Bedrock Claude Haiku and AWS OpenSearch in Oregon	10.46 tokens/s	10.22 s	Slowest managed configuration tested

The Chroma and OpenSearch rows include more than database time. Region, model endpoint, networking, and implementation choices also changed between configurations. The results support a decision about this PoC stack; they do not isolate the retrieval engine as the sole cause.

Text-to-speech benchmarks

Engine	Generation latency	Test input	Generated audio	Notes recorded during the evaluation
AWS Polly	1.25 s	337 characters	17.1 s	Fast managed API; limited custom-voice options for our use case
ElevenLabs	4.73 s	337 characters	17.1 s	Custom voice support; slower in this test
Picovoice	2.00 s	337 characters	22 s	Strong inference speed; custom voice required vendor coordination
Speechify	1.52 s	337 characters	26 s	Custom voice and good perceived quality
Deepgram	1.41 s	337 characters	17.1 s	Custom voice options, pause control, and local-hosting potential

AWS Polly gave us the clearest path to the demo because it generated a 17.1-second clip in 1.25 seconds and required no voice-model training.

Lip-sync benchmarks

We ran these tests on an RTX 4070 and measured the time required to generate ten seconds of video.

Framework	Generation time	Evaluation note
Wav2Lip	4 s	Fast setup, lower resolution, and no head movement in our test
GeneFace++	5 s	Fast inference and better lip movement; model training and environment setup took more work
SyncTalk	10 s	Best lip movement and image quality in our evaluation
Real3DPortrait	29 s	Incorrect head, lip, and torso movement in our test
SadTalker	45 s	Slow inference and weak lip sync for this use case
Hallo	More than 10 min	Natural body movement, but the inference time ruled it out for conversation

The evaluation notes also recorded commercial-use concerns for some frameworks. Licences and repository terms can change, so any production team should verify the current terms before selecting an engine.

End-to-end GPU results

We used Groq, AWS Polly, and GeneFace++ for the end-to-end GPU comparison. Prices show the hourly figures recorded during the 2024 evaluation.

Provider and GPU	Recorded price/hour	Minimum response time for 10 words	Inference speed	Average response time
AWS G5.2xlarge, A10G, Sydney	$1.212	18 s	45 it/s	25 s
Runpod, RTX 4090, Slovakia	$0.74	4.6 s	130 it/s	11.26 s
Runpod, RTX 4090, Romania	$0.74	5.6 s	123 it/s	11.44 s
Runpod, H100, Netherlands	$4.96	6 s	86 it/s	9.14 s

The RTX 4090 produced the shortest minimum response and the highest iteration rate in this workload. The H100 delivered the lowest average response time. Its recorded hourly price exceeded the 4090 instances by more than six times. Model architecture, clock speed, software optimisation, and provider variance affected the outcome. The GPU name alone did not predict the best user experience.

Why WebRTC

The avatar needed a browser-native path with low delay. HLS introduced segmenting and caching delays. RTMP lacked direct browser playback. Downloading complete files produced broken transitions between responses.

WebRTC handled media delivery in the browser and supported a future path toward two-way media. Python’s aiortc let us keep the media server in the same language as the AI pipeline, though it added setup work around tracks, timing, and connection state.

Production changes I would make

The PoC proved the conversation loop. A production version needs more control around cancellation, load, and measurement.

Add a session state machine

Each session should move through LISTENING, TRANSCRIBING, THINKING, and SPEAKING. A new user turn should invalidate stale LLM, TTS, and frame-generation jobs.

Introduce bounded queues and backpressure

The LLM can generate text faster than the avatar can render it. Unbounded audio and frame queues increase memory use and make interruptions feel broken.

Measure first-frame latency

Total response time hides the pause the user feels. I would record speech-end to transcript, first token, first audio chunk, first video frame, and playback start, with P50 and P95 values.

Separate GPU workers from the API

FastAPI should manage sessions while dedicated workers handle avatar inference. That separation makes GPU scheduling and horizontal scaling easier to control.

Design for barge-in

Full-duplex conversation needs interruption detection, job cancellation, audio ducking, and a clean return to the listening state. The session controller must drop stale frames from the previous response after the user starts a new turn.

Synchronise with timestamps

The media layer should align audio samples and video presentation timestamps. Queue length alone cannot preserve lip sync under network jitter or variable inference time.

What I learned

End-to-end latency followed the slowest media stage. The one-second RAG result helped, while the user still waited for speech generation, avatar inference, and the first playable WebRTC frame.

Streaming changed the useful metric. The system did not need to finish the whole response before it could begin playback. It needed to deliver the first coherent audio-and-video chunk, then keep producing chunks faster than the browser consumed them.

The hardware tests also challenged a common assumption. The most expensive GPU did not produce the shortest first response in our stack. Testing the complete pipeline gave us a better answer than choosing hardware from model specifications.

The PoC gave us a working digital-human conversation and a clear list of the remaining engineering work. The next version would focus on barge-in, cancellation, percentile latency, and concurrent sessions.

Demo

June 2024 digital-human proof of concept

Benchmark figures in this post come from the PoC tests recorded in June 2024. Cloud pricing, model endpoints, framework licences, and service performance may have changed since the evaluation.

How I Migrated a 350,000-Line Java/JSP Application to TypeScript in Five Days

Gokula Krishna — Fri, 31 Jul 2026 06:30:06 +0000

I recently migrated a 350,000-line Java/JSP application to TypeScript using AI agents.

Our original estimate for a conventional migration was three engineers and close to a year. Instead, one engineer orchestrating pools of up to 20 sub-agents at each stage got the application running in a development environment in five days.

The headline sounds like a story about faster code generation. It is not.

The decisive factor was verification. Once the agents had a reliable way to compare the old and new systems, parallelism became useful rather than chaotic.

350,000

lines of Java/JSP

5 days

to a running development environment

20

sub-agents in each stage

Why parallel agents were only half the solution

Jarred Sumner’s account of the Bun rewrite gave me useful ideas for running agents in parallel. Bun had an important advantage: a language-independent TypeScript test suite. The implementation could change while the expected behaviour remained measurable.

Our application had no equivalent safety net.

Only about half of the existing tests worked. Many had weak or missing assertions, and others depended directly on infrastructure. A large agent pool could generate code quickly, but without an independent oracle it could also produce 20 plausible versions of the wrong system.

We needed a way to answer a simple question at every stage: does the new application behave like the one already running?

The migration sequence

Database
APIs
UI
Business logic
Improvements

That order was deliberate. Each stage created a more stable foundation for the next, and it limited the number of variables that could change at once.

1. Reconstructing the database contract

The legacy application did not have a reliable database schema we could treat as authoritative.

We reconstructed the application-facing Oracle schema from the SQL queries embedded throughout the codebase. The agents identified tables, columns, joins, constraints and data assumptions from the queries, then we validated the reconstructed contract against the running system.

This was more than a database exercise. The schema defined the vocabulary shared by every API, page and business rule that followed. Starting anywhere else would have forced the agents to build on guesses.

2. Rebuilding the APIs

Once the data contract was stable, agents could migrate API components in parallel. Each work item was narrow: rebuild a component, exercise it against both systems, compare the result and resolve the discrepancy.

This gave the UI migration a dependable backend target instead of asking agents to infer behaviour simultaneously at every layer.

3. Migrating the UI

The UI required more than checking HTTP status codes. We needed to know whether the same data appeared, whether the same elements were present and whether the same interactions produced the same outcomes.

This is where the parity harness became the centre of the migration.

Building a parity harness where tests were missing

The parity harness ran the legacy and TypeScript applications side by side and compared them at three levels:

API responses and resulting database state
Page elements and screenshots
User interactions in both applications

The parity harness compared observable behaviour across the legacy and TypeScript systems.

The harness did not need to understand the intended design of the whole system. It needed to expose observable differences.

That distinction mattered. It turned the running legacy application into an executable specification, including behaviours that were not documented and edge cases that the surviving tests did not cover.

Parity before correctness

During the migration, we treated existing bugs as expected behaviour.

That may sound counterintuitive, but fixing bugs while changing languages, frameworks and architecture would have made failures ambiguous. When a result differed, we would not know whether we had introduced a regression or intentionally changed behaviour.

First reproduce the system. Then improve it.

We recorded known defects and fixed them only after achieving parity. This separated migration risk from product change and made discrepancies much easier to diagnose.

How the agent pools worked

At each migration stage, a pool of up to 20 sub-agents handled bounded pieces of work. Their jobs generally fell into three categories:

Rebuild a component in TypeScript.
Compare the old and new implementations through the parity harness.
Investigate and fix the differences.

The primary engineering task shifted from writing every line to designing work packets, maintaining dependency order, interpreting failures and improving the verification loop.

Parallelism helped because the work was decomposable. The harness helped because every agent could check its own result against the same external standard.

Without that feedback loop, adding more agents would have increased review load. With it, discrepancies became actionable inputs for the next iteration.

The cost

The migration consumed approximately 100 million output tokens.

At Fable 5 output-token pricing, that was roughly US$5,000 , excluding input tokens and other usage costs.

That number needs context. It is not the total cost of productionizing a modernized core system, and getting the application running in development is not the same as completing security review, performance testing, deployment and operational handover.

But it changes the economics of the first major migration milestone. A project estimated at three engineers for close to a year reached a working development environment with one engineer in five days.

What made the five-day migration possible

The result did not come from asking a model to “rewrite the application in TypeScript.”

It came from combining five practices:

Establish an observable source of truth before scaling code generation.
Migrate in dependency order so every stage has a stable foundation.
Give agents bounded work with clear inputs and outputs.
Preserve legacy behaviour until parity removes ambiguity.
Make comparison and repair part of every agent’s loop.

The parity harness was the force multiplier. It reduced the amount of trust required, made parallel work measurable and let agents detect many of their own mistakes before human review.

AI agents supplied the throughput. Verification converted that throughput into progress.

What this means for legacy modernization

The biggest shift may not be that AI writes code faster. It is that migration work can be reorganized around machine-executable feedback.

Systems once considered too large, too poorly documented or too risky to touch may become practical candidates when their observable behaviour can be captured and compared incrementally.

That does not remove the need for engineering judgment. It makes judgment more leveraged. The engineer still decides the sequence, defines the boundaries, builds the oracle and determines when parity is good enough to move forward.

As migration time and cost continue to fall, the question for many organizations may change from “Can we afford to modernize?” to “What would we need to verify it safely?”

Would you consider modernizing your core systems?

Get in touch if you are exploring an AI-assisted migration or need help designing a verification strategy.