DEV Community

Cover image for Open Source Project #160: LiveTalking — Real-Time Interactive Streaming Digital Human Engine, Wav2Lip/MuseTalk/ER-NeRF
WonderLab
WonderLab

Posted on

Open Source Project #160: LiveTalking — Real-Time Interactive Streaming Digital Human Engine, Wav2Lip/MuseTalk/ER-NeRF

#opensource #ai #python #digitalhuman

Introduction

"Real-time interactive streaming digital human engine — synchronized audio/video dialogue."

This is article #160 in the "One Open Source Project a Day" series. Today's project is LiveTalking — a real-time interactive streaming digital human engine with 9,140 Stars, created by lipku (Li Hengzhong), Apache-2.0, and already widely deployed in production.

LiveTalking solves a specific engineering problem: given an existing video clip of a person, how do you make them "say" arbitrary text or audio input in real time, output as a low-latency audio/video stream, while supporting multi-user concurrency, mid-speech interrupts, and streaming to live platforms — all at production grade.

What You'll Learn

  • The four rendering models LiveTalking supports and their respective use cases
  • The four-layer system architecture: API / Logic / Rendering / Streaming
  • Three output modes: WebRTC (browser), RTMP (live streaming), virtual camera
  • Plugin extension mechanism (TTS, Avatar, Output modules)
  • Production features: multi-session concurrency, interrupt dialog, action orchestration, voice cloning

Prerequisites

  • Basic Python and command-line experience
  • General familiarity with deep learning inference (models, GPU acceleration)
  • Some background in live streaming or WebRTC is helpful but not required

Project Background

Overview

LiveTalking is a production-grade digital human streaming framework, not an experimental demo. Its positioning is squarely at the engine layer: receive text or audio input, run TTS and lip-sync inference, output a synchronized audio/video stream, so that upstream applications (LLM dialogs, live streaming systems, customer service platforms) can plug in directly.

Author / Team

Project Stats

  • ⭐ GitHub Stars: 9,140+
  • 🍴 Forks: 1,449+
  • 📄 License: Apache-2.0
  • 📅 Created: 2023-12-19

Core Pipeline 

User input (text / audio)
    ↓
LLM generates reply (optional — connect Qwen or any LLM)
    ↓
TTS synthesizes speech (EdgeTTS / GPT-SoVITS / CosyVoice / Tencent TTS, etc.)
    ↓
Acoustic feature extraction (Mel spectrogram, etc.)
    ↓
Lip-sync inference (Wav2Lip / MuseTalk / ER-NeRF)
    ↓
Post-processing (blend generated lip region back onto high-res source video)
    ↓
Audio/video streaming (WebRTC / RTMP / virtual camera)
Enter fullscreen mode Exit fullscreen mode

Each connection is assigned a unique sessionid. API calls route to the correct session by ID, supporting multi-user concurrency with isolated compute per session.

Supported Rendering Models

LiveTalking supports four lip-sync rendering models with distinct trade-offs:

Model Characteristics Recommended GPU Real-time FPS
Wav2Lip Fastest, most compatible, 2D video-based RTX 3060+ 60–120
MuseTalk Higher quality, more natural, 2D video-based RTX 3080Ti+ 42–72
ER-NeRF 3D neural radiance field, supports multi-angle head motion High-end GPU —
Ultralight-Digital-Human Lightweight, low GPU requirements Entry-level GPU —

Real-time inference FPS benchmarks (≥25 FPS required for real-time):

Model GPU Inference FPS
wav2lip256 RTX 3060 60
wav2lip256 RTX 3080Ti 120
musetalk RTX 3080Ti 42
musetalk RTX 3090 45
musetalk RTX 4090 72

Backend logs expose two metrics: inferfps (GPU inference frame rate) and finalfps (final streaming frame rate). Both must be ≥ 25 for a real-time output.

Quick Start

Installation 

git clone https://github.com/lipku/LiveTalking.git
conda create -n livetalking python=3.12
conda activate livetalking

# Install PyTorch matching your CUDA version (CUDA 12.8 example)
pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 \
  --index-url https://download.pytorch.org/whl/cu128

cd LiveTalking
pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

Tested on Ubuntu 22.04 + Python 3.12 + PyTorch 2.9.1 + CUDA 12.8.

Download Models

Download pre-trained models from Quark Drive or Google Drive:

# Place wav2lip256.pth in models/ and rename to wav2lip.pth
# Extract wav2lip256_avatar1.tar.gz into data/avatars/
Enter fullscreen mode Exit fullscreen mode

Start the Server 

# Wav2Lip model with WebRTC transport
python app.py --transport webrtc --model wav2lip --avatar_id wav2lip256_avatar1

# MuseTalk model
python app.py --transport webrtc --model musetalk --avatar_id musetalk_avatar1

# RTMP output (push to live platforms)
python app.py --transport rtmp --model wav2lip --avatar_id wav2lip256_avatar1
Enter fullscreen mode Exit fullscreen mode

Note: TCP:8010 and UDP:1-65536 must be open on the server (WebRTC uses a wide UDP port range).

Connect and Use

  • Browser: Open http://serverip:8010/index.html, click "Start Connection," type text in the input box
  • API: Call POST /human with text input, or POST /humanaudio with an audio file

System Architecture: Four Layers

API Layer

Endpoint Description
POST /human Accept text; supports echo (direct repeat) and chat (LLM conversation) modes
POST /humanaudio Accept an audio file and play it directly
POST /record Start/stop recording; used for batch video generation

Each connection gets a unique sessionid; API calls route to the correct session by ID.

Logic Layer

  • LLM engine: Connects to Qwen or any LLM to generate conversation replies (optional — you can bypass this and pass text directly to TTS)
  • TTS engine: Modular design supporting EdgeTTS, GPT-SoVITS, CosyVoice, Tencent Cloud TTS, and others
  • Voice cloning: Clone the target person's voice so the digital human speaks with a specific vocal identity
  • Feature extraction: Synchronously extract acoustic features (Mel spectrogram) from audio, used as input to lip-sync inference

Rendering Layer

  • Model inference: Use Wav2Lip, MuseTalk, or other models to generate lip-sync frames driven by audio features
  • Post-processing: Smoothly blend the generated lip region back onto the original high-res video, preserving quality outside the mouth area

Streaming Layer

Three output modes for different deployment scenarios:

Output Latency Use case
WebRTC Very low (< 500ms) Browser real-time interaction, AI customer service
RTMP Low (1–3s) Bilibili/YouTube/Douyin live stream push
Virtual camera Very low Video conferencing, desktop apps

Plugin Extension Architecture

LiveTalking uses a decentralized registry (registry.py) for plugin-style extension of three module types:

TTS module: Register a new speech synthesis engine

from registry import register_tts

@register_tts("my_tts")
class MyTTS:
    def speak(self, text: str) -> bytes:
        # return audio data
        ...
Enter fullscreen mode Exit fullscreen mode

Avatar module: Register a new rendering model (e.g., a custom lip-sync algorithm)

Output module: Register a new streaming output (e.g., a proprietary streaming protocol)

This design lets third-party developers extend the system without modifying core code.

Production Features

Interrupt and Resume

While the digital human is speaking, a new input can interrupt it — the current audio stops and new lip-sync rendering starts immediately. This is a production requirement for AI customer service: without it, users must wait for the current speech to finish before the digital human can respond, creating an unnatural experience.

Action Orchestration

When the digital human isn't speaking, play a custom idle video (nodding, smiling, body movement) rather than showing a static freeze frame, making the digital human feel more alive.

Full-Body Video Compositing

Composite the lip-sync region (face) onto a full-body video, producing a full-body digital human rather than just a talking head portrait.

Multi-Session Concurrency

  • Silent sessions: concurrency is CPU-bound (video encoding)
  • Speaking sessions: concurrency is GPU-bound (lip-sync inference)
  • Higher output resolution consumes more CPU; mid-resolution (256px) gives the best performance-per-cost ratio

Admin Dashboard

/admin.html provides a real-time monitoring panel: view all active session states, manage global configuration, and force-stop any session.

Use Cases

Scenario Key technical points
Live commerce streaming RTMP push to Bilibili/Douyin + LLM generates sales scripts + action orchestration
AI digital human customer service WebRTC low latency + interrupt feature + enterprise knowledge base integration
Online education API-driven teacher digital clone + course content as text input
Batch short video production /human + /record API, batch-submit scripts to generate videos at scale
Exhibition/event screens Virtual camera output + local large display

LiveTalking also provides a dedicated virtual livestreamer solution: LiveStream, optimized specifically for 24/7 unattended live streaming.

Web Pages

Page Path Function
Main control /index.html WebRTC connection + text/audio input + recording control
Avatar creation /avatar.html Upload a video to auto-generate a custom digital human avatar
Admin dashboard /admin.html Real-time monitoring of all sessions and global configuration

Resources

Official Links

Summary

LiveTalking represents a typical path for digital human technology moving from research into production engineering. Three decisions stand out:

Multi-model support rather than a single bet: Wav2Lip (speed-first), MuseTalk (quality-first), and ER-NeRF (3D) each address different hardware and quality requirements. Users can match the model to their actual GPU budget and latency constraints. This is far more practical than projects that only support one rendering approach.

Four-layer decoupled architecture: API, logic, rendering, and streaming layers are independent, and TTS, rendering, and output modules are all replaceable via plugin registration. You can swap EdgeTTS for your company's proprietary TTS, or replace WebRTC with a private protocol, without touching the core code.

Interrupt handling is a genuine engineering challenge: Having a digital human interrupted mid-speech requires coordinating audio buffer queue flushing, GPU inference task cancellation, and immediate takeover by new input. Getting this right signals that the project was designed for real interactive scenarios, not just batch video generation.

Explicit performance baselines: The README provides actual FPS numbers for different GPU models and a hard requirement that inferfps ≥ 25 for real-time output. This lets users make informed hardware decisions before buying or renting a GPU.

If you're building an AI digital human product, LiveTalking provides a production-tested streaming engine that plugs directly into LLM + TTS stacks — one of the most fully engineered options currently available in the open-source community.

Explore PrimeSkills — a curated marketplace of AI agents and skills, each validated against real enterprise workflows. No hype, just what actually works.

Visit my personal site for more insights and interesting products.

Top comments (0)