Latent diffusion makes one crucial efficiency change: instead of running the entire, expensive diffusion process directly on full-resolution pixel images, first compress images into a much smaller latent space, and run diffusion there instead — this single idea is what makes Stable Diffusion practical to run on consumer hardware.
The Core Idea
The Encoder and Decoder here are exactly a VAE, from Variational Autoencoder — trained separately (and typically kept frozen during diffusion training), compressing images into a smaller spatial latent representation while preserving the perceptually important information needed to reconstruct a visually similar image.
Why This Matters — The Compute Savings
Recall the quadratic attention cost from Context Window — the same principle applies here to spatial resolution: a diffusion process running directly on, say, 512×512 pixel images is vastly more expensive (in both compute and memory) than one running on a much smaller, compressed latent representation — commonly reduced 8x or more per spatial dimension (e.g. down to a 64×64 latent grid). Since the entire iterative denoising loop (potentially dozens to hundreds of steps) runs in this smaller space, the total compute savings across the whole generation process are enormous.
Diagram
Encoding and decoding happen once each, at the start and end; the expensive, many-step diffusion loop runs entirely in the much smaller latent space.
Code
from diffusers import AutoencoderKL, UNet2DConditionModel
import torch
vae = AutoencoderKL.from_pretrained("runwayml/stable-diffusion-v1-5", subfolder="vae")
unet = UNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5", subfolder="unet")
image = torch.rand(1, 3, 512, 512) # a full-resolution image
with torch.no_grad():
latent = vae.encode(image).latent_dist.sample()
print(latent.shape) # (1, 4, 64, 64) -- a MUCH smaller representation the diffusion process actually works on
# The U-Net (noise predictor) operates entirely on this 64x64 latent, not the 512x512 imageCommon Mistakes
- Assuming the VAE encoder/decoder are trained jointly with the diffusion U-Net — they're typically trained separately, beforehand, and then kept frozen while the diffusion process is trained purely in the resulting fixed latent space.
- Confusing "latent diffusion" with an entirely different generative technique — it's the exact same forward/reverse diffusion process from earlier notes in this category, just relocated to run on compressed latents instead of raw pixels.
Interview Relevance
Q: "Why does running diffusion in a compressed latent space, rather than directly on pixels, matter so much practically?" The entire iterative denoising process — potentially dozens to hundreds of sequential steps — has to run once per generated image. Running it on a much smaller latent representation (commonly 8x smaller per spatial dimension) instead of full-resolution pixels dramatically reduces the compute and memory cost of every single one of those steps, making high-quality image generation practical on far more modest hardware.
Practice Question
If an image is compressed from 512×512 pixels to a 64×64 latent grid, roughly how much smaller (in total spatial elements) is the representation the diffusion process actually operates on?
