The Generative Adversarial Network (GAN) takes a radically different approach to generative modeling than autoencoders: two networks — a Generator and a Discriminator — trained simultaneously against each other in a competitive game, each pushing the other to improve.
The Two Players
| Network | Job |
|---|---|
| Generator (\(G\)) | Takes random noise \(\mathbf{z}\) and tries to produce a fake sample realistic enough to fool the Discriminator |
| Discriminator (\(D\)) | Takes a sample (real or generated) and tries to correctly classify it as real or fake |
The Minimax Formula
\(D\) wants to maximize this — correctly assigning high probability to real data (\(D(x)\) near 1) and low probability to generated fakes (\(D(G(z))\) near 0). \(G\) wants to minimize the same expression — producing fakes convincing enough that \(D(G(z))\) is close to 1, fooling the discriminator into thinking they're real.
The Adversarial Training Loop
- Sample a batch of real data and a batch of generated fakes (from current \(G\)).
- Update \(D\) to better distinguish real from fake (a standard binary classification training step — see Binary Cross-Entropy).
- Update \(G\) to produce fakes that fool the now-updated \(D\) more effectively.
- Repeat — each network continuously adapts to the other's latest improvements.
Diagram — The Adversarial Loop
The generator tries to fool the discriminator; the discriminator tries not to be fooled — both continuously improving against a moving target.
Code
import torch
import torch.nn as nn
generator = nn.Sequential(nn.Linear(100, 256), nn.ReLU(), nn.Linear(256, 784), nn.Tanh())
discriminator = nn.Sequential(nn.Linear(784, 256), nn.LeakyReLU(0.2), nn.Linear(256, 1), nn.Sigmoid())
loss_fn = nn.BCELoss()
g_optimizer = torch.optim.Adam(generator.parameters(), lr=0.0002)
d_optimizer = torch.optim.Adam(discriminator.parameters(), lr=0.0002)
real_images = torch.rand(32, 784)
z = torch.randn(32, 100)
fake_images = generator(z)
# Discriminator step
d_optimizer.zero_grad()
real_loss = loss_fn(discriminator(real_images), torch.ones(32, 1))
fake_loss = loss_fn(discriminator(fake_images.detach()), torch.zeros(32, 1))
(real_loss + fake_loss).backward()
d_optimizer.step()
# Generator step -- wants the discriminator to output 1 (real) for its fakes
g_optimizer.zero_grad()
g_loss = loss_fn(discriminator(fake_images), torch.ones(32, 1))
g_loss.backward()
g_optimizer.step()Why GAN Training Is Notoriously Unstable
Unlike ordinary supervised training toward a fixed target, both networks are chasing a constantly moving target (each other) — this can lead to real practical problems: mode collapse (the generator finds a small set of outputs that reliably fool the discriminator and stops producing genuine variety), or oscillating, non-converging training dynamics where neither network settles into a stable equilibrium.
Common Mistakes
- Training the discriminator far more than the generator (or vice versa) without balance — a discriminator that becomes too strong too quickly provides the generator with an unhelpfully weak, uninformative gradient signal (since \(D\) confidently rejects everything the generator produces).
- Forgetting to
.detach()the generated fakes when computing the discriminator's loss — without it, the discriminator's backward pass would also compute (unwanted, wasted) gradients through the generator.
Interview Relevance
Q: "What is 'mode collapse' in GAN training, and why does it happen?" Mode collapse is when the generator learns to produce only a narrow range of outputs (sometimes nearly identical ones) that reliably fool the current discriminator, rather than capturing the full diversity of the real data distribution. It happens because the generator's objective is purely to fool the discriminator — if a small set of convincing fakes achieves that goal, there's no direct pressure in the adversarial objective forcing genuine output diversity.
Practice Question
In the GAN minimax formula, what does it mean, in terms of \(D(G(z))\), for the generator to be "successfully fooling" the discriminator?
