Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation
Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation
๐ธ Showcase
๐ฌ Honoring Classic Films
| Devil Wears Prada | Green Book | Infernal Affairs |
 |
 |
 |
| Patch Adams | Tough Love | Shawshank Redemption |
 |
 |
 |
Explore more examples.
๐ฐ News
2024/06/28: ๐๐๐ We are proud to announce the release of our model training code. Try your own training data. Here is tutorial.2024/06/21: ๐๐๐ Cloned a Gradio demo on ๐คHuggingface space.2024/06/20: ๐๐๐ Received numerous contributions from the community, including a Windows version, ComfyUI, WebUI, and Docker template.2024/06/15: โจโจโจ Released some images and audios for inference testing on ๐คHuggingface.2024/06/15: ๐๐๐ Launched the first version on ๐ซกGitHub.
๐ค Community Resources
Explore the resources developed by our community to enhance your experience with Hallo:
- TTS x Hallo Talking Portrait Generator - Check out this awesome Gradio demo by @Sylvain Filoni! With this tool, you can conveniently prepare portrait image and audio for Hallo.
- Demo on Huggingface - Check out this easy-to-use Gradio demo by @multimodalart.
- hallo-webui - Explore the WebUI created by @daswer123.
- hallo-for-windows - Utilize Hallo on Windows with the guide by @sdbds.
- ComfyUI-Hallo - Integrate Hallo with the ComfyUI tool by @AIFSH.
- hallo-docker - Docker image for Hallo by @ashleykleynhans.
- RunPod Template - Deploy Hallo to RunPod by @ashleykleynhans.
- JoyHallo - JoyHallo extends the capabilities of Hallo, enabling it to support Mandarin
Thanks to all of them.
Join our community and explore these amazing resources to make the most out of Hallo. Enjoy and elevate their creative projects!
๐ง๏ธ Framework
โ๏ธ Installation
- System requirement: Ubuntu 20.04/Ubuntu 22.04, Cuda 12.1
- Tested GPUs: A100
Create conda environment:
conda create -n hallo python=3.10
conda activate hallo
Install packages with pip
pip install -r requirements.txt
pip install .
Besides, ffmpeg is also needed:
apt-get install ffmpeg
๐๏ธ๏ธ Usage
The entry point for inference is scripts/inference.py. Before testing your cases, two preparations need to be completed:
- Download all required pretrained models.
- Prepare source image and driving audio pairs.
- Run inference.
๐ฅ Download Pretrained Models
You can easily get all pretrained models required by inference from our HuggingFace repo.
Clone the pretrained models into ${PROJECT_ROOT}/pretrained_models directory by cmd below:
git lfs install
git clone https://huggingface.co/fudan-generative-ai/hallo pretrained_models
Or you can download them separately from their source repo:
- hallo: Our checkpoints consist of denoising UNet, face locator, image & audio proj.
- audio_separator: Kim_Vocal_2 MDX-Net vocal removal model. (Thanks to KimberleyJensen)
- insightface: 2D and 3D Face Analysis placed into
pretrained_models/face_analysis/models/. (Thanks to deepinsight) - face landmarker: Face detection & mesh model from mediapipe placed into
pretrained_models/face_analysis/models. - motion module: motion module from AnimateDiff. (Thanks to guoyww).
- sd-vae-ft-mse: Weights are intended to be used with the diffusers library. (Thanks to stablilityai)
- StableDiffusion V1.5: Initialized and fine-tuned from Stable-Diffusion-v1-2. (Thanks to runwayml)
- wav2vec: wav audio to vector model from Facebook.
Finally, these pretrained models should be organized as follows:
./pretrained_models/
|-- audio_separator/
| |-- download_checks.json
| |-- mdx_model_data.json
| |-- vr_model_data.json
| `-- Kim_Vocal_2.onnx
|-- face_analysis/
| `-- models/
| |-- face_landmarker_v2_with_blendshapes.task # face landmarker model from mediapipe
| |-- 1k3d68.onnx
| |-- 2d106det.onnx
| |-- genderage.onnx
| |-- glintr100.onnx
| `-- scrfd_10g_bnkps.onnx
|-- motion_module/
| `-- mm_sd_v15_v2.ckpt
|-- sd-vae-ft-mse/
| |-- config.json
| `-- diffusion_pytorch_model.safetensors
|-- stable-diffusion-v1-5/
| `-- unet/
| |-- config.json
| `-- diffusion_pytorch_model.safetensors
`-- wav2vec/
`-- wav2vec2-base-960h/
|-- config.json
|-- feature_extractor_config.json
|-- model.safetensors
|-- preprocessor_config.json
|-- special_tokens_map.json
|-- tokenizer_config.json
`-- vocab.json
๐ ๏ธ Prepare Inference Data
Hallo has a few simple requirements for input data:
For the source image:
- It should be cropped into squares.
- The face should be the main focus, making up 50%-70% of the image.
- The face should be facing forward, with a rotation angle of less than 30ยฐ (no side profiles).
For the driving audio:
- It must be in WAV format.
- It must be in English since our training datasets are only in this language.
- Ensure the vocals are clear; background music is acceptable.
We have provided some samples for your reference.
๐ฎ Run Inference
Simply to run the scripts/inference.py and pass source_image and driving_audio as input:
python scripts/inference.py --source_image examples/reference_images/1.jpg --driving_audio examples/driving_audios/1.wav
Animation results will be saved as ${PROJECT_ROOT}/.cache/output.mp4 by default. You can pass --output to specify the output file name. You can find more examples for inference at examples folder.
For more options:
usage: inference.py [-h] [-c CONFIG] [--source_image SOURCE_IMAGE] [--driving_audio DRIVING_AUDIO] [--output OUTPUT] [--pose_weight POSE_WEIGHT]
[--face_weight FACE_WEIGHT] [--lip_weight LIP_WEIGHT] [--face_expand_ratio FACE_EXPAND_RATIO]
options:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
--source_image SOURCE_IMAGE
source image
--driving_audio DRIVING_AUDIO
driving audio
--output OUTPUT output video file name
--pose_weight POSE_WEIGHT
weight of pose
--face_weight FACE_WEIGHT
weight of face
--lip_weight LIP_WEIGHT
weight of lip
--face_expand_ratio FACE_EXPAND_RATIO
face region
Training
Prepare Data for Training
The training data, which utilizes some talking-face videos similar to the source images used for inference, also needs to meet the following requirements:
- It should be cropped into squares.
- The face should be the main focus, making up 50%-70% of the image.
- The face should be facing forward, with a rotation angle of less than 30ยฐ (no side profiles).
Organize your raw videos into the following directory structure:
dataset_name/
|-- videos/
| |-- 0001.mp4
| |-- 0002.mp4
| |-- 0003.mp4
| `-- 0004.mp4
You can use any dataset_name, but ensure the videos directory is named as shown above.
Next, process the videos with the following commands:
python -m scripts.data_preprocess --input_dir dataset_name/videos --step 1
python -m scripts.data_preprocess --input_dir dataset_name/videos --step 2
Note: Execute steps 1 and 2 sequentially as they perform different tasks. Step 1 converts videos into frames, extracts audio from each video, and generates the necessary masks. Step 2 generates face embeddings using InsightFace and audio embeddings using Wav2Vec, and requires a GPU. For parallel processing, use the -p and -r arguments. The -p argument specifies the total number of instances to launch, dividing the data into p parts. The -r argument specifies which part the current process should handle. You need to manually launch multiple instances with different values for -r.
Generate the metadata JSON files with the following commands:
python scripts/extract_meta_info_stage1.py -r path/to/dataset -n dataset_name
python scripts/extract_meta_info_stage2.py -r path/to/dataset -n dataset_name
Replace path/to/dataset with the path to the parent directory of videos, such as dataset_name in the example above. This will generate dataset_name_stage1.json and dataset_name_stage2.json in the ./data directory.
Training
Update the data meta path settings in the configuration YAML files, configs/train/stage1.yaml and configs/train/stage2.yaml:
#stage1.yaml
data:
meta_paths:
- ./data/dataset_name_stage1.json
#stage2.yaml
data:
meta_paths:
- ./data/dataset_name_stage2.json
Start training with the following command:
accelerate launch -m \
--config_file accelerate_config.yaml \
--machine_rank 0 \
--main_process_ip 0.0.0.0 \
--main_process_port 20055 \
--num_machines 1 \
--num_processes 8 \
scripts.train_stage1 --config ./configs/train/stage1.yaml
Accelerate Usage Explanation
The accelerate launch command is used to start the training process with distributed settings.
accelerate launch [arguments] {training_script} --{training_script-argument-1} --{training_script-argument-2} ...
Arguments for Accelerate:
-m, --module: Interpret the launch script as a Python module.--config_file: Configuration file for Hugging Face Accelerate.--machine_rank: Rank of the current machine in a multi-node setup.--main_process_ip: IP address of the master node.--main_process_port: Port of the master node.--num_machines: Total number of nodes participating in the training.--num_processes: Total number of processes for training, matching the total number of GPUs across all machines.
Arguments for Training:
{training_script}: The training script, such asscripts.train_stage1orscripts.train_stage2.--{training_script-argument-1}: Arguments specific to the training script. Our training scripts accept one argument,--config, to specify the training configuration file.
For multi-node training, you need to manually run the command with different machine_rank on each node separately.
For more settings, refer to the Accelerate documentation.
๐ ๏ธ Roadmap
| Status | Milestone | ETA |
|---|---|---|
| โ | Inference source code meet everyone on GitHub | 2024-06-15 |
| โ | Pretrained models on Huggingface | 2024-06-15 |
| โ | Releasing data preparation and training scripts | 2024-06-28 |
| ๐ | Improving the modelโs performance on Mandarin Chinese | TBD |
Other Enhancements
- [x] Enhancement: Test and ensure compatibility with Windows operating system. #39
- [x] Bug: Output video may lose several frames. #41
- [ ] Bug: Sound volume affecting inference results (audio normalization).
- [ ]
Enhancement: Inference code logic optimization. This solution doesnโt show significant performance improvements. Trying other approaches.
๐ Citation
If you find our work useful for your research, please consider citing the paper:
@misc{xu2024hallo,
title={Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation},
author={Mingwang Xu and Hui Li and Qingkun Su and Hanlin Shang and Liwei Zhang and Ce Liu and Jingdong Wang and Yao Yao and Siyu zhu},
year={2024},
eprint={2406.08801},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
๐ Opportunities Available
Multiple research positions are open at the Generative Vision Lab, Fudan University! Include:
- Research assistant
- Postdoctoral researcher
- PhD candidate
- Master students
Interested individuals are encouraged to contact us at [email protected] for further information.
โ ๏ธ Social Risks and Mitigations
The development of portrait image animation technologies driven by audio inputs poses social risks, such as the ethical implications of creating realistic portraits that could be misused for deepfakes. To mitigate these risks, it is crucial to establish ethical guidelines and responsible use practices. Privacy and consent concerns also arise from using individualsโ images and voices. Addressing these involves transparent data usage policies, informed consent, and safeguarding privacy rights. By addressing these risks and implementing mitigations, the research aims to ensure the responsible and ethical development of this technology.
๐ค Acknowledgements
We would like to thank the contributors to the magic-animate, AnimateDiff, ultimatevocalremovergui, AniPortrait and Moore-AnimateAnyone repositories, for their open research and exploration.
If we missed any open-source projects or related articles, we would like to complement the acknowledgement of this specific work immediately.
๐ Community Contributors
Thank you to all the contributors who have helped to make this project better!