by TheMistoAI
Open source · 4k downloads · 457 likes
MistoLine is an SDXL-ControlNet model designed to interpret and leverage various types of line drawings, whether hand-sketched, preprocessed, or generated by AI models. Through an innovative line preprocessing approach and optimized training, it delivers high precision and remarkable stability, even in complex scenarios, to produce high-quality images (exceeding 1024 pixels). The model stands out for its ability to adapt to different line sources without requiring specific models, simplifying image generation workflows. It is particularly well-suited for artists, designers, and content creators looking to transform sketches or outlines into detailed and coherent visuals. MistoLine distinguishes itself through its balance of flexibility, performance, and adherence to ethical constraints, while remaining compatible with most SDXL models and other tools like LCM.
We are thrilled to announce that MistoLine has officially surpassed 300,000 total downloads! 🚀
(Verified via official Hugging Face API)
MistoLine also achieved #1 in the Text-to-Image Trending list and Top 4 Overall on Hugging Face! These milestones reflect the incredible support and adoption from the open-source AI community. Check out our trending highlights:
MistoLine: A Versatile and Robust SDXL-ControlNet Model for Adaptable Line Art Conditioning.
MistoLine is an SDXL-ControlNet model that can adapt to any type of line art input, demonstrating high accuracy and excellent stability. It can generate high-quality images (with a short side greater than 1024px) based on user-provided line art of various types, including hand-drawn sketches, different ControlNet line preprocessors, and model-generated outlines. MistoLine eliminates the need to select different ControlNet models for different line preprocessors, as it exhibits strong generalization capabilities across diverse line art conditions.
We developed MistoLine by employing a novel line preprocessing algorithm Anyline and retraining the ControlNet model based on the Unet of stabilityai/ stable-diffusion-xl-base-1.0, along with innovations in large model training engineering. MistoLine showcases superior performance across different types of line art inputs, surpassing existing ControlNet models in terms of detail restoration, prompt alignment, and stability, particularly in more complex scenarios.
MistoLine maintains consistency with the ControlNet architecture released by @lllyasviel, as illustrated in the following schematic diagram:
reference：https://github.com/lllyasviel/ControlNet
More information about ControlNet can be found in the following references:
https://github.com/lllyasviel/ControlNet
https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet_sdxl
The model is compatible with most SDXL models, except for PlaygroundV2.5, CosXL, and SDXL-Lightning(maybe). It can be used in conjunction with LCM and other ControlNet models.
The following usage of this model is not allowed:
If you use or distribute this model for commercial purposes, you must comply with the following conditions:
署名条款
如果您在商业用途中使用或分发本模型,您必须满足以下条件:
The model output is not censored and the authors do not endorse the opinions in the generated content. Use at your own risk.
The following case only utilized MistoLine as the controlnet：
The following case only utilized Anyline as the preprocessor and MistoLine as the controlnet.
sampler steps:30
CFG:7.0
sampler_name:dpmpp_2m_sde
scheduler:karras
denoise:0.93
controlnet_strength:1.0
stargt_percent:0.0
end_percent:0.9
Make sure to first install the libraries:
pip install accelerate transformers safetensors opencv-python diffusers
And then we're ready to go:
from diffusers import ControlNetModel, StableDiffusionXLControlNetPipeline, AutoencoderKL
from diffusers.utils import load_image
from PIL import Image
import torch
import numpy as np
import cv2
prompt = "aerial view, a futuristic research complex in a bright foggy jungle, hard lighting"
negative_prompt = 'low quality, bad quality, sketches'
image = load_image("https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd_controlnet/hf-logo.png")
controlnet_conditioning_scale = 0.5
controlnet = ControlNetModel.from_pretrained(
"TheMistoAI/MistoLine",
torch_dtype=torch.float16,
variant="fp16",
)
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
controlnet=controlnet,
vae=vae,
torch_dtype=torch.float16,
)
pipe.enable_model_cpu_offload()
image = np.array(image)
image = cv2.Canny(image, 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
image = Image.fromarray(image)
images = pipe(
prompt, negative_prompt=negative_prompt, image=image, controlnet_conditioning_scale=controlnet_conditioning_scale,
).images
images[0].save(f"hug_lab.png")
链接：https://pan.baidu.com/s/1DbZWmGJ40Uzr3Iz9RNBG_w?pwd=8mzs
提取码：8mzs
@misc{
title={Adding Conditional Control to Text-to-Image Diffusion Models},
author={Lvmin Zhang, Anyi Rao, Maneesh Agrawala},
year={2023},
eprint={2302.05543},
archivePrefix={arXiv},
primaryClass={cs.CV}
}