Generative Adversarial Networks (GANs)

Understand how Generative Adversarial Networks (GANs) use two competing neural networks to generate realistic data through an iterative training process.

What it is

A GAN consists of two neural networks: the Generator and the Discriminator. The Generator creates synthetic data from random noise, aiming to fool the Discriminator. The Discriminator evaluates data, trying to distinguish between real samples and generated fakes. This setup creates a zero-sum game where both networks improve simultaneously until the Generator produces indistinguishable outputs.

Key terms include latent space (the input noise vector), adversarial loss (the objective function driving competition), and mode collapse (a failure state where the Generator produces limited variety).

Why it matters

• Data Augmentation: Generates synthetic images or tabular data to balance imbalanced datasets.
• Image Synthesis: Creates high-resolution images from low-resolution inputs or text descriptions.
• Anomaly Detection: Learns normal data distributions to identify outliers effectively.
• Style Transfer: Applies artistic styles to photographs by manipulating feature representations.

Syntax or steps

The training loop alternates between updating the Discriminator and the Generator. For each batch:

1. Sample real data and generate fake data using the current Generator.
2. Train the Discriminator to maximize the probability of correctly labeling real as 1 and fake as 0.
3. Train the Generator to minimize the Discriminator's ability to detect its fakes (maximizing the error on fake labels).

Example

import torch
import torch.nn as nn

# Define simple networks for MNIST-like data
class Generator(nn.Module):
    def __init__(self, latent_dim=100):
        super().__init__()
        self.model = nn.Sequential(
            nn.Linear(latent_dim, 256),
            nn.ReLU(),
            nn.Linear(256, 784), # Output size for 28x28 image
            nn.Tanh()
        )
    def forward(self, z):
        return self.model(z).view(-1, 1, 28, 28)

class Discriminator(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(
            nn.Flatten(),
            nn.Linear(784, 256),
            nn.LeakyReLU(0.2),
            nn.Linear(256, 1),
            nn.Sigmoid()
        )
    def forward(self, img):
        return self.model(img)

# Training step logic (pseudo-code structure)
def train_step(gen, disc, optimizer_g, optimizer_d, real_batch, latent_dim=100):
    # 1. Train Discriminator
    optimizer_d.zero_grad()
    b_size = real_batch.size(0)
    
    # Real data
    label_real = torch.ones(b_size, 1)
    pred_real = disc(real_batch)
    loss_d_real = nn.BCELoss()(pred_real, label_real)
    
    # Fake data
    z = torch.randn(b_size, latent_dim)
    fake_batch = gen(z)
    label_fake = torch.zeros(b_size, 1)
    pred_fake = disc(fake_batch.detach())
    loss_d_fake = nn.BCELoss()(pred_fake, label_fake)
    
    loss_d = (loss_d_real + loss_d_fake) / 2
    loss_d.backward()
    optimizer_d.step()

    # 2. Train Generator
    optimizer_g.zero_grad()
    z = torch.randn(b_size, latent_dim)
    fake_batch = gen(z)
    pred_fake = disc(fake_batch)
    loss_g = nn.BCELoss()(pred_fake, label_real) # Want discriminator to think it's real
    loss_g.backward()
    optimizer_g.step()
    
    return loss_d.item(), loss_g.item()

This code defines the architecture and the core adversarial update rule. Note that detach() prevents gradients from flowing back into the Generator during Discriminator updates, ensuring independent optimization.

Common mistakes

• Unstable Training: If one network learns too fast, the other fails. Fix by balancing learning rates or using techniques like spectral normalization.
• Mode Collapse: The Generator finds one "good" output and repeats it. Fix by adding diversity penalties or using mini-batch discrimination.
• Incorrect Labels: Using hard 0/1 labels can cause vanishing gradients. Fix by using label smoothing (e.g., 0.9 instead of 1.0).
• Forgetting Detach: Not detaching fake images during Discriminator training corrupts the Generator's weights prematurely.

When to use it

Use GANs when visual realism is paramount and you have resources to tune stability. Use VAEs if probabilistic interpretation and smooth latent spaces are more important than pixel-perfect sharpness.

Practice

Guided Exercise: Modify the train_step function to implement label smoothing by changing label_real to torch.full((b_size, 1), 0.9).

Challenge: Add a gradient penalty term to the Discriminator loss to stabilize training further. Hint: Calculate the gradient of the Discriminator output with respect to interpolated inputs between real and fake batches.

Quick check

Q: Why must we call detach() on the fake images before passing them to the Discriminator?

A: To prevent the Discriminator's loss calculation from updating the Generator's weights. We want to optimize the Discriminator only against fixed Generator outputs in that specific step.

Summary

GANs leverage adversarial competition between a Generator and Discriminator to produce high-quality synthetic data. Success depends on carefully balancing their training dynamics to avoid instability and mode collapse.