FLUX.1 dev ControlNet Union Pro
by Shakker-Labs
Open source · 6k downloads · 619 likes
3.5
(619 reviews)ImageAPI & LocalAbout
FLUX.1 dev ControlNet Union Pro is an advanced image generation model that combines the power of the FLUX.1-dev model with an enhanced version of ControlNet, enabling precise control over image creation. It supports seven different control modes, such as edge processing, depth, poses, or image quality, offering greater flexibility to tailor results to specific needs. This model is particularly useful for artists, designers, or developers looking to integrate visual constraints into their generations, such as composition, details, or styles. What sets it apart is its "Pro" version, trained with more data and steps, ensuring better accuracy and consistency in the results. It can also be combined with other ControlNets for even more sophisticated effects.
Documentation
FLUX.1-dev-ControlNet-Union-Pro
This repository contains a unified ControlNet for FLUX.1-dev model jointly released by researchers from InstantX Team and Shakker Labs.
Model Cards
- This checkpoint is a Pro version of FLUX.1-dev-Controlnet-Union trained with more steps and datasets.
- This model supports 7 control modes, including canny (0), tile (1), depth (2), blur (3), pose (4), gray (5), low quality (6).
- The recommended controlnet_conditioning_scale is 0.3-0.8.
- This model can be jointly used with other ControlNets.
Showcases
Inference
Please install diffusers from the source, as the PR has not been included in currently released version yet.
Multi-Controls Inference
Python
import torch
from diffusers.utils import load_image
from diffusers import FluxControlNetPipeline, FluxControlNetModel
from diffusers.models import FluxMultiControlNetModel
base_model = 'black-forest-labs/FLUX.1-dev'
controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro'
controlnet_union = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
controlnet = FluxMultiControlNetModel([controlnet_union]) # we always recommend loading via FluxMultiControlNetModel
pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)
pipe.to("cuda")
prompt = 'A bohemian-style female travel blogger with sun-kissed skin and messy beach waves.'
control_image_depth = load_image("https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro/resolve/main/assets/depth.jpg")
control_mode_depth = 2
control_image_canny = load_image("https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro/resolve/main/assets/canny.jpg")
control_mode_canny = 0
width, height = control_image_depth.size
image = pipe(
prompt,
control_image=[control_image_depth, control_image_canny],
control_mode=[control_mode_depth, control_mode_canny],
width=width,
height=height,
controlnet_conditioning_scale=[0.2, 0.4],
num_inference_steps=24,
guidance_scale=3.5,
generator=torch.manual_seed(42),
).images[0]
We also support loading multiple ControlNets as before, you can load as
Python
from diffusers import FluxControlNetModel
from diffusers.models import FluxMultiControlNetModel
controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro'
controlnet_union = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
controlnet_model_depth = 'Shakker-Labs/FLUX.1-dev-Controlnet-Depth'
controlnet_depth = FluxControlNetModel.from_pretrained(controlnet_model_depth, torch_dtype=torch.bfloat16)
controlnet = FluxMultiControlNetModel([controlnet_union, controlnet_depth])
# set mode to None for other ControlNets
control_mode=[2, None]
Resources
- InstantX/FLUX.1-dev-Controlnet-Canny
- Shakker-Labs/FLUX.1-dev-ControlNet-Depth
- Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro
Acknowledgements
This project is trained by InstantX Team and sponsored by Shakker AI. The original idea is inspired by xinsir/controlnet-union-sdxl-1.0. All copyright reserved.
Capabilities & Tags
diffuserssafetensorsText-to-ImageControlNetDiffusersFlux.1-devimage-generationStable Diffusiontext-to-imageen
Links & Resources