Magic Animate

magic-research / magic-animate

[CVPR 2024] Official repository for "MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model"

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

Zhongcong Xu · Jianfeng Zhang · Jun Hao Liew · Hanshu Yan · Jia-Wei Liu · Chenxu Zhang · Jiashi Feng · Mike Zheng Shou

National University of Singapore | ByteDance

📢 News

[2023.12.4] Release inference code and gradio demo. We are working to improve MagicAnimate, stay tuned!
[2023.11.23] Release MagicAnimate paper and project page.

🏃‍♂️ Getting Started

Download the pretrained base models for StableDiffusion V1.5 and MSE-finetuned VAE.

Download our MagicAnimate checkpoints.

Please follow the huggingface download instructions to download the above models and checkpoints, git lfs is recommended.

Place the based models and checkpoints as follows:

magic-animate
|----pretrained_models
  |----MagicAnimate
    |----appearance_encoder
      |----diffusion_pytorch_model.safetensors
      |----config.json
    |----densepose_controlnet
      |----diffusion_pytorch_model.safetensors
      |----config.json
    |----temporal_attention
      |----temporal_attention.ckpt
  |----sd-vae-ft-mse
    |----config.json
    |----diffusion_pytorch_model.safetensors
  |----stable-diffusion-v1-5
    |----scheduler
       |----scheduler_config.json
    |----text_encoder
       |----config.json
       |----pytorch_model.bin
    |----tokenizer

…

View on GitHub

I tried Magic Animate with a super famous Japanese comedian’s photo on Google Colab T4(free tier), but the result wasn’t what I expected lol.

But still, I think Magic Animate is awesome.

Another one that is kind of funny.

The following is the Google Colab I used.

llm_on_GoogleColab/MagicAnimate_colab.ipynb at main · koji/llm_on_GoogleColab
Contribute to koji/llm_on_GoogleColab development by creating an account on GitHub.
github.com

AI-Colab-Notebooks

Google Colab で動く AI モデルのテストノート集。LLM / 画像生成 / 動画 / 音声など。各ノートのバッジからそのまま Colab で開ける。

LLM / Chat

Notebook	内容
Mistral-7B.ipynb	Mistral 7B (OpenHermes GGUF + llama-cpp-python)
Calm2_7B_Chat_llama_cpp_python.ipynb	Calm2 7B Chat (GGUF + llama-cpp-python)
Japanese_StableLM_Instruct_Beta_7B_llama_cpp_python.ipynb	Japanese StableLM Instruct Beta 7B
Google_Gemma.ipynb	Google Gemma
Rakuten_7B_chat.ipynb	Rakuten 7B Chat
RakutenAI_7B_chat_Quantized_model_test.ipynb	RakutenAI 7B Chat (Quantized)
phi_4_reasoning.ipynb	Phi-4-mini-reasoning
Liquid-AI_decision-model/	Liquid AI decision model (600M / 3B)

Ollama / Agents

Notebook	内容
ollama.ipynb	Ollama セットアップ
ollama_deepseek_r1_32b_4.ipynb	Ollama + DeepSeek R1 32B
ollama_deepseek_r1_32b_with_oyama.ipynb	Ollama + DeepSeek R1 32B + oyama
smolagents_with_ollama.ipynb	smolagents + Ollama
Gemini_CLI.ipynb	Gemini CLI
output2prompt.ipynb	output2prompt

Vision / Multimodal

Notebook	内容
llama3_2_vision.ipynb	Llama 3.2 Vision + Ollama
llama3_2_vision+transformers/	Llama 3.2 Vision + transformers (Gradio app付き)
imp_v1_3b.ipynb	imp-v1-3b VLM
Janus_Pro.ipynb	Janus Pro
Qwen-Audio_test.ipynb	Qwen-Audio

Image Generation

Notebook	内容
sdxl_turbo_simple_ui/	SDXL Turbo + Simple Gradio UI
SDXL_Lightning.ipynb	SDXL Lightning
sd3_test_with_gradio.ipynb	Stable Diffusion 3 + Gradio
flux_schnell_dev_with_transformers.ipynb	FLUX schnell/dev + transformers
CogView4.ipynb	CogView4
diffusers_v0_27.ipynb	diffusers v0.27
gemini_api_generate_image.ipynb

…

Note

If you run the above Colab on T4, you will need to reduce sampling from 25 to around 15(12–15) because of T4’s memory size.

The entire process will take around 10 minutes.

DEV Community

Generate a Video from an Image and Motion Magic Animate on Google Colab