Cross-Entropy

Cross-entropy measures the difference between two probability distributions — specifically, how well a model's predicted distribution matches the true distribution of labels. It is, without exaggeration, the single most widely used loss function in all of deep learning classification.

Formula

\(p\) is the true distribution (for a labeled example, this is 1 for the correct class and 0 for every other class), and \(q\) is the model's predicted distribution (its softmax output). Because \(p\) is zero everywhere except the correct class \(y\), the sum collapses to a single term:

This is the loss for one example: the negative log of the probability the model assigned to the correct class. This exactly matches the negative log-likelihood derived in Maximum Likelihood Estimation.

Numerical Example

A 3-class classifier predicts \(q = [0.7, 0.2, 0.1]\) for classes [cat, dog, bird]. The true label is "cat" (\(p=[1,0,0]\)).

If instead the model had predicted \(q=[0.1, 0.2, 0.7]\) (confidently wrong), the loss would be \(-\log(0.1) \approx 2.303\) — much higher, correctly penalizing the confidently incorrect prediction far more heavily than a merely uncertain one.

Loss vs Confidence — Why the Curve Matters

Code

import torch
import torch.nn as nn

loss_fn = nn.CrossEntropyLoss()   # expects raw logits, NOT softmax probabilities -- it applies softmax internally

logits = torch.tensor([[2.0, 0.5, -1.0]])   # raw scores for [cat, dog, bird]
true_label = torch.tensor([0])                # index 0 = "cat" is correct

loss = loss_fn(logits, true_label)
print(loss.item())   # a single scalar loss value

Binary Cross-Entropy — The Two-Class Special Case

For binary classification, this simplified form is used with a sigmoid output instead of softmax — covered in full in the Loss Functions category next.

Common Mistakes

• Applying softmax manually before passing predictions to PyTorch's nn.CrossEntropyLoss — it already applies log_softmax internally for numerical stability; applying softmax twice silently produces a wrong (usually much smaller) gradient signal.
• Forgetting cross-entropy is not symmetric: \(H(p,q) \ne H(q,p)\) in general — the order matters, and deep learning always uses the true labels as \(p\) and the model's predictions as \(q\).

Interview Relevance

Q: "Why does cross-entropy loss penalize a confidently wrong prediction so much more than mean squared error would?" Cross-entropy is \(-\log q(y)\), which grows toward infinity as the predicted probability for the correct class approaches zero — the logarithm's shape makes the penalty explode for confidently wrong predictions. MSE on probabilities grows only quadratically, penalizing the same mistake far less severely, which is part of why cross-entropy trains classifiers faster and more reliably.

Practice Question

A binary classifier predicts \(\hat y = 0.05\) for an example whose true label is 1 (positive). Compute the binary cross-entropy loss for this single example, and explain why it's a large value.

Related DL Notes

• Entropy
• KL Divergence
• Binary Cross-Entropy