Diffusion Forward Process

The forward process is the (fixed, not learned) half of a diffusion model: gradually adding small amounts of Gaussian noise to real data over many steps, until nothing recognizable remains — pure noise.

Formula — One Step

At each step \(t\), a small amount of noise (controlled by \(\beta_t\), a small value like 0.0001 to 0.02, following a predetermined schedule) is added to the previous step's (slightly noisier) image. \(\beta_t\) typically increases gradually across steps — smaller noise additions early on, larger later.

The Closed-Form Shortcut — Jumping Directly to Any Step

Repeatedly applying the one-step formula \(t\) times would be tedious to implement directly. Fortunately, the math simplifies beautifully: because each step adds independent Gaussian noise, there's a closed-form way to jump directly from the original clean image \(\mathbf{x}_0\) to any noisy step \(\mathbf{x}_t\) in one computation:

\(\bar\alpha_t = \prod_{s=1}^t (1-\beta_s)\) is the cumulative product of "keep" factors up to step \(t\). This single formula is a direct application of the same reparameterization trick from Variational Autoencoder — expressing a random variable as a deterministic function of a fixed-distribution noise term.

Diagram

Code

import torch

def forward_diffusion(x0, t, alpha_bar):
    epsilon = torch.randn_like(x0)
    sqrt_alpha_bar_t = alpha_bar[t] ** 0.5
    sqrt_one_minus_alpha_bar_t = (1 - alpha_bar[t]) ** 0.5
    x_t = sqrt_alpha_bar_t * x0 + sqrt_one_minus_alpha_bar_t * epsilon
    return x_t, epsilon   # return the noise too -- this IS the training target for the reverse process

T = 1000
betas = torch.linspace(0.0001, 0.02, T)
alphas = 1 - betas
alpha_bar = torch.cumprod(alphas, dim=0)

x0 = torch.rand(1, 3, 64, 64)   # a clean image
x_500, noise_added = forward_diffusion(x0, t=500, alpha_bar=alpha_bar)

Why the Forward Process Needs No Training

Notice this entire process is completely fixed — a predetermined noise schedule, with no learnable parameters at all. This is deliberate: the forward process's simplicity and fixed nature is exactly what makes generating training data trivial (jump to any noise level instantly via the closed-form formula) and what leaves the entire learning burden for the much harder reverse process, covered next.

Common Mistakes

• Assuming the forward process is learned — it's entirely fixed by the chosen noise schedule (\(\beta_t\) values); only the reverse process (next note) involves a trained neural network.
• Forgetting that the closed-form shortcut requires the cumulative product \(\bar\alpha_t\), not just the single-step \(\alpha_t\) — using the wrong one produces an image noised to the wrong degree.

Interview Relevance

Q: "Why is diffusion's forward (noising) process not learned, while the reverse (denoising) process is?" The forward process is a simple, well-understood mathematical operation — gradually adding Gaussian noise according to a fixed schedule — that requires no learning to define or execute correctly. The genuinely hard problem, and the one requiring a trained neural network, is the reverse direction: given a noisy image, figuring out how to remove noise to recover something realistic, which has no simple closed-form solution.

Practice Question

Why does the closed-form shortcut formula let you generate a training example at noise step 750 without first computing steps 1 through 749 individually?

Related DL Notes

• Diffusion Reverse Process
• Variational Autoencoder
• Probability Distributions