ToonCrafter: Generative Cartoon Interpolation

   
    

Jinbo Xing, Hanyuan Liu, Menghan Xia, Yong Zhang, Xintao Wang, Ying Shan, Tien-Tsin Wong

From CUHK and Tencent AI Lab.

at SIGGRAPH Asia 2024, Journal Track

🔆 Introduction

⚠️ Please check our disclaimer first.

🤗 ToonCrafter can interpolate two cartoon images by leveraging the pre-trained image-to-video diffusion priors. Please check our project page and paper for more information.

1.1 Showcases (512x320)

Input starting frame	Input ending frame	Generated video

1.2 Sparse sketch guidance

Input starting frame	Input ending frame	Input sketch guidance	Generated video

2. Applications

2.1 Cartoon Sketch Interpolation (see project page for more details)

Input starting frame	Input ending frame	Generated video

2.2 Reference-based Sketch Colorization

Input sketch	Input reference	Colorization results

📝 Changelog

• Add sketch control and colorization function.
• [2024.05.29]: 🔥🔥 Release code and model weights.
• [2024.05.28]: Launch the project page and update the arXiv preprint.

🧰 Models

Model	Resolution	GPU Mem. & Inference Time (A100, ddim 50steps)	Checkpoint
ToonCrafter_512	320x512	~24G & 24s (perframe_ae=True)	Hugging Face

We get the feedback from issues that the model may consume about 24G~27G GPU memory in this implementation, but the community has lowered the consumption to ~10GB.

Currently, our ToonCrafter can support generating videos of up to 16 frames with a resolution of 512x320. The inference time can be reduced by using fewer DDIM steps.

⚙️ Setup

Install Environment via Anaconda (Recommended)

conda create -n tooncrafter python=3.8.5
conda activate tooncrafter
pip install -r requirements.txt

💫 Inference

1. Command line

Download pretrained ToonCrafter_512 and put the model.ckpt in checkpoints/tooncrafter_512_interp_v1/model.ckpt.

  sh scripts/run.sh

2. Local Gradio demo

Download the pretrained model and put it in the corresponding directory according to the previous guidelines.

  python gradio_app.py 

🤝 Community Support

1. ComfyUI and pruned models (fp16): ComfyUI-DynamiCrafterWrapper (Thanks to kijai)

Model	Resolution	GPU Mem.	Checkpoint
ToonCrafter	512x320	12GB	Hugging Face

2. ComfyUI. ComfyUI-ToonCrafter (Thanks to Yorha4D)

3. Colab. Code (Thanks to camenduru), Code (Thanks to 0smboy)

4. Windows platform support: ToonCrafter-for-windows (Thanks to sdbds)

5. Sketch-guidance implementation: ToonCrafter_with_SketchGuidance (Thanks to mattyamonaca)

😉 Citation

Please consider citing our paper if our code is useful:

@article{xing2024tooncrafter,
  title={ToonCrafter: Generative Cartoon Interpolation},
  author={Xing, Jinbo and Liu, Hanyuan and Xia, Menghan and Zhang, Yong and Wang, Xintao and Shan, Ying and Wong, Tien-Tsin},
  journal={arXiv preprint arXiv:2405.17933},
  year={2024}
}

🙏 Acknowledgements

We would like to thank Xiaoyu for providing the sketch extractor, and supraxylon for the Windows batch script.

📢 Disclaimer

We have not set up any official profit-making projects or web applications. Please be cautious.

Calm down. Our framework opens up the era of generative cartoon interpolation, but due to the variaity of generative video prior, the success rate is not guaranteed.

⚠️This is an open-source research exploration, instead of commercial products. It can't meet all your expectations.

This project strives to impact the domain of AI-driven video generation positively. Users are granted the freedom to create videos using this tool, but they are expected to comply with local laws and utilize it responsibly. The developers do not assume any responsibility for potential misuse by users.

---