Image Control - ControlNet XL Union

What it does

ControlNet XL Union is an "all-in-one" ControlNet adapter for Stable Diffusion XL (SDXL). Instead of loading a separate ControlNet for every conditioning type (one for canny, one for depth, one for pose, etc.), this single model handles 6 different conditioning modes with the same network parameters. You give it a control image of the appropriate type, a canny edge map, a depth map, a pose skeleton, a HED soft-edge map, a surface normal map, or a semantic segmentation mask, and a text prompt, and SDXL generates a new image that follows the structure of the control image and the content/style of the text prompt. It's the workhorse for any SDXL-based image-to-image workflow that needs structural control but where you want flexibility to switch between control types without swapping models. Built by xinsir and trained on 10M+ carefully captioned images, with the same parameter count as the original ControlNet — no increase in compute or memory vs. running a single-condition ControlNet.

Problem it solves

• One model for many conditioning types – Switch between canny, depth, openpose, hed, normal, and segmentation without loading separate adapters — saves VRAM and pipeline complexity
• Composition-locked generation – Keep the structure, geometry, or pose of a reference photo while changing everything else (style, lighting, materials, era)
• Pose-driven character generation – Use openpose skeletons to place a character in any pose, then style with text
• Depth-guided scene restyling – Restyle a scene while preserving its 3D layout
• Semantic layout control – Use segmentation masks to lock in what goes where (sky, road, person) and let the prompt fill in how it looks
• Sketch-to-image – Turn line art (canny / HED / scribble-like maps) into rendered images
• Multi-condition fusion (built-in) – The model was trained to combine conditions, so you can stack control signals (e.g., openpose + depth) without manual tuning
• Stable foundation for the UI's control nodes – This is the model behind any "use a control image" SDXL flow in the studio

Input/Output

• Input:
• Control image matching the selected conditioning type:
• canny - edge maps from Annotator-CannyEdge
• depth - depth maps from Annotator-DepthAnything
• openpose - pose skeletons
• hed - soft edge / HED maps
• normal - surface normal maps
• segment - semantic segmentation masks
• Text prompt describing the desired image (style, content, mood)

• Options:
• Conditioning type (canny / depth / openpose / hed / normal / segment) — must match the control image you supply
• Strength of the ControlNet guidance (0.0 – 2.0, default 0.8)
• 0.0 = ignore the control image · 2.0 = strictly follow it
• 0.7 – 1.0 works well for most inputs; reduce if the control is too rigid
• Number of diffusion steps (default 25)
• 20 – 30 gives good quality; more steps add marginal quality at linear time cost
• Random seed (default 1)
• 1 picks a new random seed each run; set a fixed integer to reproduce the same image
• Output image width (512 / 768 / 1024) — the control image is resized to match
• Output image height (512 / 768 / 1024) — the control image is resized to match
• Advanced — Negative prompt (leave empty for no negative guidance; e.g. "blurry, low quality, watermark, text")
• Output: Generated image
• Up to 1024 × 1024 (native SDXL; bucket training supports any aspect ratio)
• Style and content follow the text prompt; structure follows the control image

Accuracy & Speed

Technical Details

Compliance & Provenance

For more on how we classify models and mark outputs, see our AI Policy.

Model Source

• HuggingFace: https://huggingface.co/xinsir/controlnet-union-sdxl-1.0
• GitHub: https://github.com/xinsir6/ControlNetPlus
• Paper: arXiv:2302.05543
• License: apache-2.0