This closing comparison note resolves a question nearly every beginner hits when building their first classifier: sigmoid or softmax for the output layer? The answer depends entirely on whether classes are mutually exclusive.
Side-by-Side Comparison
| Sigmoid | Softmax | |
|---|---|---|
| Applied to | Each output neuron independently | The whole output vector jointly |
| Output interpretation | \(P(\text{this specific class is present})\), independent per class | \(P(\text{class } i)\) across all classes, forced to sum to 1 |
| Outputs sum to 1? | No — each is independent | Yes — always exactly 1 |
| Correct for | Binary classification, or multi-label classification (multiple non-exclusive classes can be simultaneously "true") | Multi-class classification with mutually exclusive classes (exactly one class is correct) |
| Paired loss | Binary cross-entropy | Categorical cross-entropy |
Worked Example — Why the Choice Matters
An image-tagging model that can output any combination of "outdoor," "daytime," "has-people" (these can all be simultaneously true — not mutually exclusive) should use independent sigmoid outputs, one per tag: \(P(\text{outdoor})=0.9\), \(P(\text{daytime})=0.8\), \(P(\text{has-people})=0.3\) — these don't need to sum to anything in particular.
A digit classifier that outputs exactly one of {0,1,...,9} (mutually exclusive — an image is only ever one digit) should use softmax: \(P(0)=0.02, P(1)=0.85, \ldots\) — forced to sum to 1, since exactly one answer is correct.
The Special Case: Binary Classification
For exactly 2 mutually exclusive classes, a single sigmoid output (representing \(P(\text{class}=1)\), with \(P(\text{class}=0)=1-P(\text{class}=1)\) implied) is mathematically equivalent to a 2-output softmax — using softmax with 2 outputs for binary classification is technically valid but redundant; a single sigmoid neuron is the simpler, standard choice.
Code
import torch
import torch.nn as nn
# Multi-label: independent sigmoid per tag
multi_label_logits = torch.tensor([2.0, 1.5, -0.5]) # outdoor, daytime, has-people
print(torch.sigmoid(multi_label_logits))
# each interpreted independently -- no constraint that they sum to 1
# Multi-class, mutually exclusive: softmax across all classes
multi_class_logits = torch.tensor([0.1, 3.2, 0.5, -1.0]) # e.g. 4 possible digit classes
print(torch.softmax(multi_class_logits, dim=0))
# sums to exactly 1.0 -- exactly one class is "the" answerCommon Mistakes
- Using softmax for a multi-label problem — forcing probabilities to sum to 1 when multiple labels can genuinely co-occur artificially suppresses valid simultaneous predictions (predicting one tag strongly necessarily lowers every other tag's probability, which is wrong when tags are independent).
- Using independent sigmoids for a genuinely mutually-exclusive multi-class problem — this doesn't enforce that exactly one class "wins," and the resulting scores don't form a coherent probability distribution over the classes.
Interview Relevance
Q: "You're building a model to detect which of several diseases (assume a patient can have more than one simultaneously) are present in a medical scan. Sigmoid or softmax for the output layer?" Sigmoid, applied independently to each disease's output neuron — since diseases can co-occur (not mutually exclusive), softmax's forced-sum-to-1 constraint would incorrectly suppress the ability to predict multiple diseases confidently at once.
Key Takeaways — Activation Functions
- Step is historical only; it has zero gradient almost everywhere and cannot be used with backpropagation.
- Sigmoid and tanh saturate for large \(|z|\), causing vanishing gradients in deep networks — they're now mostly reserved for output layers (sigmoid) and gated cells (both).
- ReLU is the modern hidden-layer default: cheap, non-saturating for positive inputs, but can "die." Leaky ReLU, PReLU and ELU each address dying neurons differently — fixed slope, learned slope, and smooth exponential floor respectively.
- GELU and Swish are smooth, non-monotonic ReLU alternatives that dominate in Transformers specifically.
- Softmax is for mutually-exclusive multi-class outputs; sigmoid (applied per-class) is for binary or multi-label outputs.
Next: Loss Functions covers every major loss formula — building directly on the MLE derivations from Probability & Statistics for DL and the activation choices from this category — with worked numerical examples for each.
Practice Question
A weather model predicts, for a given day, the probability it will be sunny, rainy, cloudy or snowy — assume exactly one of these four labels is correct per day. Which activation function belongs on the output layer, and what loss function pairs with it?
