Image segmentation pushes computer vision to its finest level of spatial detail: instead of a whole-image label (classification) or a box (detection), it assigns a class label to every individual pixel in the image.
Why Pixel-Level Prediction Is a Different Kind of Problem
A classification network collapses spatial information down to one label (via pooling/GAP, see Global Average Pooling). Segmentation needs the exact opposite: a full-resolution output map, the same spatial size as the input, with a class prediction at every position. This requires architectures specifically designed to preserve (or later restore) spatial resolution, rather than discarding it — the direct motivation behind the FCN and U-Net architectures covered later in this category.
Three Flavors, Introduced Here
| Flavor | What It Distinguishes |
|---|---|
| Semantic segmentation | Every pixel gets a class label, but individual object instances of the same class aren't distinguished from each other |
| Instance segmentation | Every pixel gets both a class label AND a specific object instance ID, distinguishing separate objects of the same class |
| Panoptic segmentation | Unifies both — assigns instance IDs to countable "things" (objects) while still labeling uncountable "stuff" (sky, road, grass) semantically |
Each of these gets its own dedicated note next, but the shared foundation is exactly this idea: a per-pixel prediction task, evaluated with metrics like IoU and Dice (see Dice Score) rather than whole-image accuracy.
Code — The Output Shape Difference
import torch
# Classification output: one label per whole image
classification_output_shape = (1, 10) # (batch, num_classes)
# Segmentation output: one label PER PIXEL
segmentation_output_shape = (1, 10, 224, 224) # (batch, num_classes, H, W)
# argmax over dim=1 gives the predicted class for every individual pixelCommon Mistakes
- Applying standard classification evaluation metrics (like whole-image accuracy) to a segmentation task — pixel-level metrics like IoU and Dice are the appropriate tools here, not whole-image classification metrics.
- Confusing which segmentation flavor a specific task actually needs — the choice between semantic, instance, and panoptic segmentation has real architectural consequences, covered in the next three notes.
Interview Relevance
Q: "Why can't a standard image classification CNN, as-is, perform image segmentation?" Classification architectures deliberately collapse spatial information into a single whole-image prediction, typically via pooling or Global Average Pooling. Segmentation needs the opposite — a full-resolution, per-pixel output — which requires architectures specifically designed to preserve or restore spatial detail throughout the network, rather than discarding it for a single final classification.
Practice Question
An image of a street scene contains three cars. Would semantic segmentation alone be able to tell you there are three separate cars? Why or why not?
