---
license: apache-2.0
---
# RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wild

<a href='https://arxiv.org/abs/2504.14977'>
<img src='https://img.shields.io/badge/arXiv-red'></a> 
<a href='https://thefoxofsky.github.io/project_pages/RealisDance-DiT/index'>
<img src='https://img.shields.io/badge/Project-Page-orange'></a> 
<a href='https://huggingface.co/datasets/theFoxofSky/RealisDance-Val'>
<img src='https://img.shields.io/badge/Test-Dataset-blue'></a> 
<a href='https://github.com/damo-cv/RealisDance'>
<img src='https://img.shields.io/badge/Git-Hub-green'></a> 

This repository is the official model checkpoint of [RealisDance-DiT](https://arxiv.org/abs/2504.14977).
RealisDance-DiT is a structurally simple, empirically robust, and experimentally strong baseline model for controllable character animation in the wild.

## Gallery
Here are several character animation generated by RealisDance-DiT. 
Note that the GIFs shown here have some degree of visual quality degradation.
Please visit our [project page](https://thefoxofsky.github.io/project_pages/RealisDance-DiT/index) for more original videos 

<table class="center">
    <tr>
    <td><img src="https://github.com/damo-cv/RealisDance/blob/main/__assets__/samples/1.gif?raw=true"></td>
    <td><img src="https://github.com/damo-cv/RealisDance/blob/main/__assets__/samples/2.gif?raw=true"></td>
    </tr>
    <tr>
    <td><img src="https://github.com/damo-cv/RealisDance/blob/main/__assets__/samples/3.gif?raw=true"></td>
    <td><img src="https://github.com/damo-cv/RealisDance/blob/main/__assets__/samples/4.gif?raw=true"></td>
    </tr>
    <tr>
    <td><img src="https://github.com/damo-cv/RealisDance/blob/main/__assets__/samples/5.gif?raw=true"></td>
    <td><img src="https://github.com/damo-cv/RealisDance/blob/main/__assets__/samples/6.gif?raw=true"></td>
    </tr>
</table>

## Quick Start

### 1. Setup Repository and Environment

```
git clone https://github.com/theFoxofSky/RealisDance.git
cd RealisDance

conda create -n realisdance python=3.10
conda activate realisdance

pip install -r requirements.txt

# FA3 (Optional)
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
git checkout ea3ecea97a1393c092863330aff9a162bb5ce443  # very important, using other FA3 will yield bad results
cd hopper
python setup.py install
```

### 2. Quick Inference

- Inference with Demo sequences

```commandline
python inference.py \
    --ref __assets__/demo/ref.png \
    --smpl __assets__/demo/smpl.mp4 \
    --hamer __assets__/demo/hamer.mp4 \
    --prompt "A blonde girl is doing somersaults on the grass. Behind the grass is a river, \
    and behind the river are trees and mountains. The girl is wearing black yoga pants and a black sports vest." \
    --save-dir ./output
```

## Disclaimer
This project is released for academic use.
We disclaim responsibility for user-generated content.

## Contact Us
Jingkai Zhou: [fs.jingkaizhou@gmail.com](mailto:fs.jingkaizhou@gmail.com)


## BibTeX
```
@article{zhou2025realisdance-dit,
  title={RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wild},
  author={Zhou, Jingkai and Wu, Yifan and Li, Shikai and Wei, Min and Fan, Chao and Chen, Weihua and Jiang, Wei and Wang, Fan},
  journal={arXiv preprint arXiv:2504.14977},
  year={2025}
}

@article{zhou2024realisdance,
  title={RealisDance: Equip controllable character animation with realistic hands},
  author={Zhou, Jingkai and Wang, Benzhi and Chen, Weihua and Bai, Jingqi and Li, Dongyang and Zhang, Aixi and Xu, Hao and Yang, Mingyang and Wang, Fan},
  journal={arXiv preprint arXiv:2409.06202},
  year={2024}
}
```


## Acknowledgements
Thanks [Shikai Li](https://scholar.google.com/citations?user=WXGg2rgAAAAJ&hl) for condition paraperation and [Chenjie Cao](https://ewrfcas.github.io/) for pose align.