One model to rule them all? In the world of Video AI, we've traditionally been forced to pick our poison: one model for VQA (Understanding), one for T2V (Generation), and another for SDEdit (Editing).
UniVideo changes the game. Released recently by the KlingTeam, it unifies these three pillars into a single Dual-Stream framework.
Why should devs care?
Most video models are "black boxes" that take text and spit out pixels. UniVideo is different because it links a Multimodal LLM (MLLM) directly to a Diffusion Transformer (DiT).
- Semantic-to-Video: The MLLM acts as the "encoder" that actually understands the scene logic before the DiT starts drawing.
- Mask-Free Editing: No more fighting with segmentation masks. You can literally tell the model: "Change that car's material to gold" or "Apply a green screen background," and it just works.
- Identity Preservation: It hits a 0.88 score in subject consistency, solving the "jittery character" problem we've all struggled with in open-source pipelines.
Getting Started: Deploying UniVideo
Ready to get your hands dirty? Here is the step-by-step guide to getting UniVideo running locally.
1. Environment Setup
You'll need a Beefy GPU (NVIDIA A100/H100 recommended for training, though inference can run on smaller cards with optimization).
# Clone the repo
git clone https://github.com/univideo/UniVideo
cd UniVideo
# Create a clean environment
conda create -n univideo python=3.10 -y
conda activate univideo
# Install dependencies
pip install -r requirements.txt
2. Download Weights
The model weights are hosted on Hugging Face. You'll need the DiT checkpoints and the VAE.
# Ensure you have git-lfs installed
git lfs install
git clone https://huggingface.co/KlingTeam/UniVideo weights/
3. Basic Inference Script
You can run a simple text-to-video generation or an image-to-video task using the provided inference CLI.
python sample.py \
--model_path "weights/univideo_model.pt" \
--prompt "A futuristic cyberpunk city in the rain, high quality, 4k" \
--save_path "./outputs/demo.mp4"
4. Advanced: Visual Prompting
UniVideo supports "visual prompts" (like drawing an arrow to indicate motion). To use this, you'll need to pass an image and a motion-hint mask to the sampler.
# Example for Image-to-Video with motion guidance
python sample_i2v.py --image_path "./assets/car.jpg" --motion_mask "./assets/arrow_mask.png"
Performance Benchmarks
If you're looking at the numbers, UniVideo is punching way above its weight:
MM Bench: 83.5 (Visual Reasoning)
VBench (T2V): 82.6 (State-of-the-Art Quality)
Consistency: 0.88 (Identity Preservation)
Resources & Links
Try it online (No Setup Required): UniVideo Official Site
Full Paper: Technical PDF
Source Code: GitHub - UniVideo
Weights: Hugging Face
What are you planning to build with this? I'm personally looking into how the "mask-free editing" can be integrated into automated VFX pipelines. Let's discuss in the comments!
Top comments (1)
It's really clear and practical, especially how you explain why the dual-stream design actually matters for developers. I like the direction UniVideo is heading; unifying understanding, generation, and editing feels like the right solution to a lot of current pipeline pain. I’m especially interested in the mask-free editing idea and how far it can be pushed in real production workflows. Curious to see where this goes and what kind of tools people start building on top of it.