Conditioning and ControlNet in generative AI

Module 12 · Generative AI · 18 concepts

In depth / 15 lines

Conditioning and ControlNet, in more detail

Conditioning means everything the model is told before it generates, and text is only one channel.

Text is excellent for subject and style and nearly useless for spatial arrangement.

Describing a person standing three-quarters on with their left arm raised rarely produces that pose.

ControlNet solves this by adding a structural input alongside the prompt.

The input is a spatial map the same size as the image: edges, depth, segmentation, or a pose skeleton.

Generation is then required to respect that map while the prompt supplies everything else.

The architecture is the interesting part, because it avoids retraining the base model.

A trainable copy of the encoder is made, and the original weights are frozen.

The copy processes the control signal and its output is added back into the frozen network.

The connecting layers start at zero, so at the beginning of training the model behaves exactly as before.

That zero initialisation is why a ControlNet can be trained on a modest dataset without damaging the base.

Several can be stacked, combining a depth map for layout with a pose for the figure.

In production this is usually the difference between a demo and a usable tool.

Advertising, product imagery and game assets all need the composition fixed in advance.

The same idea appears elsewhere as adapters and reference conditioning, all trading freedom for control.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Conditioning and ControlNet

Three places a condition can enter

\boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, \; t, \; c)

A diffusion model's noise predictor takes the noisy image, the timestep and the condition. How $c$ is injected decides what kind of control it gives.

\text{cross-attention} : Q \text{ from image patches}, \; K, V \text{ from text tokens}

Text enters through cross-attention in every block, so each spatial position can read whichever words matter to it.

\text{adaptive norm} : \mathbf{h} \leftarrow \gamma(c) \, \mathrm{LN}(\mathbf{h}) + \beta(c)

Scalars such as the timestep or a class label enter by modulating the normalisation, which is cheap and global.

Spatial conditions, such as an edge map or a pose, fit neither. They have the same shape as the image and must be respected position by position.

Classifier-free guidance

\tilde{\boldsymbol{\epsilon}} = \boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, \varnothing) + s \big( \boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, c) - \boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, \varnothing) \big)

$\varnothing$ : the empty condition, trained by dropping the prompt on about 10 percent of examples.
$s$ : the guidance scale.

The difference between the conditional and unconditional predictions is the direction the prompt is pulling, and guidance walks further along it than the model asked.

s = 1 : \text{no guidance}, \qquad s \approx 7 : \text{typical}, \qquad s > 15 : \text{saturated and broken}

It costs a second forward pass per step, which is why guidance doubles generation time unless the two passes are batched together.

ControlNet

\mathbf{y} = F(\mathbf{x}; \Theta) + \mathcal{Z}\big( F(\mathbf{x} + \mathcal{Z}(\mathbf{c}); \Theta_{\mathrm{copy}}) \big)

$\Theta$ : the frozen weights of the original model.
$\Theta_{\mathrm{copy}}$ : a trainable copy of the encoder half.
$\mathcal{Z}$ : a 1 by 1 convolution initialised to zero.

The encoder blocks are duplicated, the copy is trained on the spatial condition, and its output is added back into the frozen network's skip connections.

\mathcal{Z} = 0 \implies \text{output identical to the base model}

The zero convolutions are the reason this can be trained at all.

At step one the branch contributes nothing, so a randomly initialised addition cannot destroy a model that took a month to train.

The base model is never updated, so one copy of it serves every control type, and a ControlNet is a file you add rather than a model you replace.

One conditioned generation, in order

Encode the prompt to text embeddings.
Prepare the spatial condition, such as running an edge detector on a reference image.
At each denoising step, run the frozen network with the ControlNet branch adding into it.
Run it again with an empty prompt, and combine the two by guidance.
Take the solver step and continue.

What each control is for

Canny edges: keep the exact outlines, change everything else.
Depth: keep the three-dimensional layout, change the content.
Pose: keep the skeleton, change the person and the scene.
Segmentation: keep the region boundaries and what each region is.
Scribble: keep a loose composition drawn by hand.

Each carries its own strength dial, and the failure at high values is the same in every case: the output satisfies the control and stops obeying the prompt.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What does ControlNet let you specify that a text prompt cannot?

Generative AI

18 concepts in this module, then a graded test with every answer explained.

Study this module →