Kolors IP Adapter Plus
par Kwai-Kolors
Open source · 11k downloads · 118 likes
2.6
(118 avis)ImageAPI & LocalÀ propos
Kolors IP Adapter Plus est un modèle d'IA spécialisé dans la génération d'images à partir de descriptions textuelles tout en intégrant des éléments visuels de référence. Il se distingue par son extracteur d'images renforcé, basé sur le modèle Openai-CLIP-336, qui préserve davantage de détails des images d'entrée. Grâce à un jeu de données d'entraînement plus large et de meilleure qualité, il produit des résultats plus diversifiés et fidèles à la fois au texte et à l'image de référence. Le modèle excelle particulièrement dans la fidélité sémantique, c'est-à-dire sa capacité à conserver les caractéristiques visuelles essentielles des images sources. Il est idéal pour des applications créatives comme la retouche d'images, la génération de variations stylistiques ou l'intégration d'éléments spécifiques dans de nouvelles compositions.
Documentation
Kolors-IP-Adapter-Plus weights and inference code
📖 Introduction
We provide IP-Adapter-Plus weights and inference code based on Kolors-Basemodel. Examples of Kolors-IP-Adapter-Plus results are as follows:
Our improvements
- A stronger image feature extractor. We employ the Openai-CLIP-336 model as the image encoder, which allows us to preserve more details in the reference images
- More diverse and high-quality training data: We construct a large-scale and high-quality training dataset inspired by the data strategies of other works. We believe that paired training data can effectively improve performance.
📊 Evaluation
For evaluation, we create a test set consisting of over 200 reference images and text prompts. We invite several image experts to provide fair ratings for the generated results of different models. The experts rate the generated images based on four criteria: visual appeal, text faithfulness, image faithfulness, and overall satisfaction. Image faithfulness measures the semantic preservation ability of IP-Adapter on reference images, while the other criteria follow the evaluation standards of BaseModel. The specific results are summarized in the table below, where Kolors-IP-Adapter-Plus achieves the highest overall satisfaction score.
| Model | Average Overall Satisfaction | Average Image Faithfulness | Average Visual Appeal | Average Text Faithfulness |
|---|---|---|---|---|
| SDXL-IP-Adapter-Plus | 2.29 | 2.64 | 3.22 | 4.02 |
| Midjourney-v6-CW | 2.79 | 3.0 | 3.92 | 4.35 |
| Kolors-IP-Adapter-Plus | 3.04 | 3.25 | 4.45 | 4.30 |
The ip_scale parameter is set to 0.3 in SDXL-IP-Adapter-Plus, while Midjourney-v6-CW utilizes the default cw scale.
Kolors-IP-Adapter-Plus employs chinese prompts, while other methods use english prompts.
🛠️ Usage
Requirements
The dependencies and installation are basically the same as the Kolors-BaseModel.
- Repository Cloning and Dependency Installation
Bash
apt-get install git-lfs
git clone https://github.com/Kwai-Kolors/Kolors
cd Kolors
conda create --name kolors python=3.8
conda activate kolors
pip install -r requirements.txt
python3 setup.py install
- Weights download link：
Bash
huggingface-cli download --resume-download Kwai-Kolors/Kolors-IP-Adapter-Plus --local-dir weights/Kolors-IP-Adapter-Plus
or
Bash
git lfs clone https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-Plus weights/Kolors-IP-Adapter-Plus
- Inference：
Bash
python ipadapter/sample_ipadapter_plus.py ./ipadapter/https://raw.githubusercontent.com/junqiangwu/Kolors/master/ipadapter/asset/test_ip.jpg "穿着黑色T恤衫，上面中文绿色大字写着“可图”"
python ipadapter/sample_ipadapter_plus.py ./ipadapter/https://raw.githubusercontent.com/junqiangwu/Kolors/master/ipadapter/asset/test_ip2.png "一只可爱的小狗在奔跑"
# The image will be saved to "scripts/outputs/"
Note
The IP-Adapter-FaceID model based on Kolors will also be released soon!
Acknowledgments
- Thanks to IP-Adapter for providing the codebase.
Liens & Ressources