X-Adapter

This repository is the official implementation of X-Adapter.

X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model


Lingmin Ran,
Xiaodong Cun,
Jia-Wei Liu,
Rui Zhao,
Song Zijie,
Xintao Wang,
Jussi Keppo,
Mike Zheng Shou

X-Adapter enables plugins pretrained on the old version (e.g. SD1.5) directly work with the upgraded Model (e.g., SDXL) without further retraining.

Thank @kijai for CumfyUI implementation here! Please refer to this tutorial for hyperparameter setting.

News

• [17/02/2024] Inference code released

Setup

Requirements

conda create -n xadapter python=3.10
conda activate xadapter

pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116
pip install -r requirements.txt

Installing xformers is highly recommended for high efficiency and low GPU cost.

Weights

[Stable Diffusion] Stable Diffusion is a latent text-to-image diffusion model capable of generating photo-realistic images given any text input. The pre-trained Stable Diffusion models can be downloaded from Hugging Face (e.g., Stable Diffusion v1-5). You can also use fine-tuned Stable Diffusion models trained on different styles (e.g., Anything V4.0, Redshift, etc.).

[ControlNet] Controlnet is a method to control diffusion models with spatial conditions. You can download the ControlNet family here.

[LoRA] LoRA is a lightweight adapter to fine-tune large-scale pretrained model. It is widely used for style or identity customization in diffusion models. You can download LoRA from the diffusion community (e.g., civitai).

Checkpoint

Models can be downloaded from our Hugging Face page. Put the checkpoint in folder ./checkpoint/X-Adapter.

Usage

After preparing all checkpoints, we can run inference code using different plugins. You can refer to this tutorial to quickly get started with X-Adapter.

Controlnet Inference

Set --controlnet_canny_path or --controlnet_depth_path to ControlNet’s path in the bash script. The default value is its Hugging Face model card.

sh ./bash_scripts/canny_controlnet_inference.sh
sh ./bash_scripts/depth_controlnet_inference.sh

LoRA Inference

Set --lora_model_path to LoRA’s checkpoint in the bash script. In this example we use MoXin, and we put it in folder ./checkpoint/lora.

sh ./bash_scripts/lora_inference.sh

Controlnet-Tile Inference

Set --controlnet_tile_path to ControlNet-tile’s path in the bash script. The default value is its Hugging Face model card.

sh ./bash_scripts/controlnet_tile_inference.sh

Cite

If you find X-Adapter useful for your research and applications, please cite us using this BibTeX:

@article{ran2023xadapter,
  title={X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model},
  author={Lingmin Ran and Xiaodong Cun and Jia-Wei Liu and Rui Zhao and Song Zijie and Xintao Wang and Jussi Keppo and Mike Zheng Shou},
  journal={arXiv preprint arXiv:2312.02238},
  year={2023}
}