This is a simplified guide to an AI model called Dreamtalk maintained by Cjwbw. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.
Model overview
dreamtalk is a diffusion-based audio-driven expressive talking head generation framework developed by cjwbw. It can produce high-quality talking head videos across diverse speaking styles, including songs, speech in multiple languages, noisy audio, and out-of-domain portraits. dreamtalk exhibits robust performance, building upon the foundations of previous works like PIRenderer, AVCT, StyleTalk, and others. Similar models by the same creator include sadtalker, video-retalking, aniportrait-audio2vid, analog-diffusion, and voicecraft.
Model inputs and outputs
dreamtalk takes in an audio file, a reference speaking style, a head pose, and an input portrait. It then generates a talking head video that matches the audio and exhibits the desired speaking style and head pose. The model can handle a variety of audio file formats, including wav, mp3, m4a, and mp4.
Inputs
- Audio: The input audio file
- Style clip: A reference speaking style, specified as a 3DMM parameter sequence
- Pose: The desired head pose, also specified as a 3DMM parameter sequence
- Image: The input portrait, which should be larger than 256x256 and will be cropped to that size
Outputs
- Talking head video: A high-quality video of a talking head that matches the input audio, style, and pose
Capabilities
dreamtalk can generate expressive ta...
Top comments (0)