ResNet (2015) solved the exact problem that had limited how deep any CNN could practically go — vanishing gradients across many stacked layers — with one deceptively simple architectural addition: the residual (skip) connection.
The Problem It Solved
Before ResNet, simply stacking more layers eventually made networks harder to train, not easier — beyond a certain depth, accuracy would actually get worse, even on the training set (ruling out overfitting as the explanation). This was a direct manifestation of the vanishing gradient problem from Vanishing Gradient Problem: gradients shrinking multiplicatively across dozens or hundreds of layers, leaving early layers with almost no usable training signal.
Key Innovation: The Residual Connection
Instead of a block of layers learning the full desired output directly, ResNet has each block learn only the residual — the difference between its input and its desired output, \(F(\mathbf{x})\) — and then adds the original input \(\mathbf{x}\) back in directly. This "skip connection" provides an additive path for gradients to flow backward, completely bypassing the multiplicative chain-rule shrinkage that causes vanishing gradients — as flagged already in Vanishing Gradient Problem's solutions table.
Diagram
The input skips directly around the convolutional block and is added to its output — providing an unimpeded, additive gradient path back to earlier layers.
Code
import torch
import torch.nn as nn
class ResidualBlock(nn.Module):
def __init__(self, channels):
super().__init__()
self.conv1 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
self.conv2 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
self.relu = nn.ReLU()
def forward(self, x):
identity = x
out = self.relu(self.conv1(x))
out = self.conv2(out)
out += identity # the residual/skip connection
return self.relu(out)Advantages and Limitations
| Advantages | Limitations |
|---|---|
| Enabled training networks with over 100 (even 1,000+) layers, previously impractical | Requires matching shapes for the addition — a projection (1×1 convolution) is needed when a block changes channel count or spatial size |
| The residual connection idea proved so effective it's now used far beyond CNNs, including inside every Transformer block | Very deep ResNet variants still require substantial compute |
Use Cases
ResNet variants (ResNet-18, 34, 50, 101, 152) remain extremely widely used today — as classification backbones, as pretrained feature extractors for transfer learning, and as the architectural inspiration for the residual connections used throughout modern Transformer architectures (see the Transformers category).
Common Mistakes
- Assuming residual connections only matter for very deep networks — even moderately deep networks often train faster and more reliably with them, since they generally improve gradient flow regardless of exact depth.
- Forgetting that a residual connection requires the input and the block's output to have matching shapes — architectures typically insert a small projection (often a 1×1 convolution) on the skip path whenever a block changes the number of channels or spatial resolution.
Interview Relevance
Q: "How do residual connections solve the vanishing gradient problem in very deep networks?" By adding the block's input directly to its output, residual connections provide an additive path for gradients to flow backward, bypassing the multiplicative chain-rule product that causes vanishing gradients in a purely sequential stack of layers. Even if a block's own gradient contribution shrinks toward zero, the identity path still carries gradient signal back to earlier layers essentially undiminished.
Practice Question
Why does a residual block require its convolutional path's output to match the shape of its input exactly, and what's the typical fix when a block needs to change the number of channels?
