A plain diffusion model, as covered so far, generates unconditionally — random samples from whatever distribution it learned, with no control over content. Conditioning lets you steer generation toward specific desired content, such as matching a text description.
Injecting Conditioning Into the U-Net
The most common approach for text conditioning uses cross-attention — the exact mechanism from Cross-Attention — inserted at multiple layers within the U-Net. The U-Net's own spatial features act as the query; the text embedding (typically produced by a separate pretrained text encoder, like CLIP's text encoder) provides the keys and values. This lets every spatial location in the image being generated selectively attend to whichever words in the text prompt are most relevant to that specific region.
Classifier-Free Guidance
A widely-used technique for strengthening how closely generation follows the conditioning: train the model to sometimes generate unconditionally (by randomly dropping the conditioning signal during a fraction of training steps), then at generation time, compute both a conditional and unconditional prediction and combine them:
\(c\) is the conditioning (e.g. text embedding), \(\varnothing\) represents no conditioning, and \(w>1\) is the guidance scale. This effectively exaggerates the difference the conditioning makes, pushing generation to follow the prompt more strongly than the raw conditional prediction alone would — the "guidance scale" setting many text-to-image tools expose directly to users.
Diagram
The text embedding is injected via cross-attention throughout the U-Net, steering every denoising step toward matching the prompt.
Code
from diffusers import StableDiffusionPipeline
import torch
pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
image = pipe(
"a serene Japanese garden in autumn",
guidance_scale=7.5, # the classifier-free guidance scale -- higher = follows the prompt more strongly
num_inference_steps=50
).images[0]Common Mistakes
- Setting the guidance scale extremely high expecting proportionally better prompt adherence with no downside — excessively high guidance scales can produce over-saturated, artifact-heavy, or visually unnatural images, not just "more accurate" ones.
- Assuming conditioning only works via text — the same cross-attention injection pattern generalizes to other conditioning signals (class labels, sketches, depth maps), directly echoing the conditional GAN idea from Conditional GAN.
Interview Relevance
Q: "How does classifier-free guidance strengthen a diffusion model's adherence to a text prompt?" By training the model to also generate unconditionally (with conditioning randomly dropped some fraction of the time), then at generation time, computing both a conditional and unconditional noise prediction and extrapolating away from the unconditional one toward the conditional one, scaled by a guidance factor \(w>1\). This exaggerates the effect of the conditioning signal, producing outputs that follow the prompt more strongly than the raw conditional prediction alone.
Practice Question
Why does the U-Net's spatial features serve as the query in text-conditioning cross-attention, rather than the text embedding?
