The Fully Convolutional Network (FCN) was the architecture that made modern semantic segmentation practical — by replacing a classification CNN's fully-connected layers with convolutional ones, preserving spatial structure all the way to the output.
The Problem It Solved
Standard classification CNNs deliberately discard spatial information via flattening or Global Average Pooling before their final fully-connected layers (see Flattening) — exactly the opposite of what segmentation needs, which is a full-resolution, per-pixel output. FCN asked: what if the network stayed fully convolutional throughout, never discarding spatial structure at all?
Key Innovation: Replacing FC Layers with Convolutions, Then Upsampling
FCN takes a standard classification backbone (like VGG) and replaces its fully-connected layers with equivalent convolutional layers, keeping the network's output spatial (though at a much lower resolution than the original input, due to the pooling/stride operations along the way). It then upsamples this low-resolution output back to the original input's spatial size — commonly via transposed convolution (sometimes loosely called "deconvolution"), a learnable operation that increases spatial resolution, in some sense the reverse of standard convolution's typical size reduction.
Diagram
The network downsamples via standard convolution/pooling, then upsamples back to the original resolution to produce a per-pixel output map.
Code — A Simplified Sketch
import torch.nn as nn
class SimpleFCN(nn.Module):
def __init__(self, num_classes):
super().__init__()
self.encoder = nn.Sequential(
nn.Conv2d(3, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(64, 128, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
)
self.upsample = nn.ConvTranspose2d(128, num_classes, kernel_size=4, stride=4)
def forward(self, x):
features = self.encoder(x) # downsampled spatially
return self.upsample(features) # upsampled back to (roughly) the original sizeAdvantages and Limitations
| Advantages | Limitations |
|---|---|
| First architecture to make dense, per-pixel prediction practical using a standard CNN backbone | Direct upsampling from a heavily downsampled feature map can lose fine spatial detail |
| Adaptable to any classification backbone by simply replacing its final layers | Later architectures (like U-Net, next note) improved on FCN's detail-preservation with skip connections |
Common Mistakes
- Assuming any upsampling method (like simple interpolation) is equivalent to FCN's learnable transposed convolution — a learnable upsampling operation can recover more useful detail than a purely fixed, non-learned resizing method.
Interview Relevance
Q: "What was FCN's key architectural change compared to a standard classification CNN?" It replaced the classification network's fully-connected layers with convolutional ones, keeping the entire network spatial throughout, and added a learnable upsampling step (transposed convolution) to restore the heavily downsampled feature map back to the original input's resolution — producing a genuine per-pixel output map instead of a single whole-image label.
Practice Question
Why does FCN need an upsampling step at all, rather than simply keeping every layer at the original input resolution throughout the network?
