Diffusion models emerged as a direct response to GAN training's real, practical instability (mode collapse, adversarial balancing issues, as covered in GAN) — offering a fundamentally different, far more stable training recipe, at the cost of much slower generation.
The Core Idea, Before Any Formulas
Diffusion models learn to generate data by learning to reverse a gradual noising process: start with real data, slowly corrupt it into pure random noise over many small steps, then train a network to reverse this — undoing the noise, one small step at a time — until pure noise can be transformed back into a realistic new sample.
Why This Training Is So Much More Stable Than a GAN's
| GAN | Diffusion Model | |
|---|---|---|
| Training setup | Two networks competing adversarially against a moving target | One network, trained with a simple, well-defined, fixed regression objective (predict the noise that was added) |
| Common failure modes | Mode collapse, oscillating/non-converging dynamics | Much less prone to these — a straightforward supervised-style loss, similar in spirit to Mean Squared Error |
| Sample generation speed | Fast — one forward pass through the generator | Slow — requires many sequential denoising steps |
| Sample diversity | Can suffer from mode collapse, reduced diversity | Generally strong sample diversity, closely tracking the true data distribution |
The Fundamental Tradeoff
Diffusion models trade GAN's fast, single-pass generation for a much more stable, reliable training process — but pay for it with generation that requires many sequential steps (historically hundreds to a thousand, though modern techniques have reduced this substantially). This tradeoff — stability and quality at the cost of generation speed — is the central practical consideration that shaped the entire family of techniques covered in this category.
Code — A First Glimpse
from diffusers import StableDiffusionPipeline
import torch
pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
image = pipe("a photograph of a mountain lake at sunset", num_inference_steps=50).images[0]
# 50 SEQUENTIAL denoising steps -- directly reflecting diffusion's iterative generation processCommon Mistakes
- Assuming diffusion models eliminated GANs entirely — GANs remain useful in domains where fast, single-pass generation matters more than diffusion's stability/quality advantages, though diffusion has become dominant for high-quality image generation specifically.
- Underestimating diffusion's generation-speed cost — this is a real, significant practical tradeoff, and much of the applied research around diffusion models (covered later in this category) is specifically about reducing the number of required denoising steps.
Interview Relevance
Q: "Why did diffusion models become the dominant approach for high-quality image generation, despite being slower than GANs?" Diffusion models train with a simple, stable, well-defined objective (predicting added noise), largely avoiding GAN training's characteristic instabilities like mode collapse and adversarial imbalance. This stability, combined with strong sample quality and diversity, outweighed the generation-speed disadvantage for many applications, especially as techniques for accelerating diffusion sampling matured.
Practice Question
Why does a GAN generate an image faster than a diffusion model, at a structural level?
