Categorical cross-entropy generalizes binary cross-entropy to any number of mutually exclusive classes — the standard loss for multi-class classification, always paired with a softmax output.
Formula
\(y_{i,c}\) is 1 if example \(i\)'s true class is \(c\), and 0 otherwise (a one-hot encoded label vector), \(\hat{y}_{i,c}\) is the model's predicted probability for class \(c\) (from softmax). Because \(y_{i,c}=0\) for every class except the true one, this sum collapses to a single term per example: \(-\log(\hat{y}_{i,y_i})\) — the negative log of the predicted probability assigned to the correct class, exactly matching Cross-Entropy's general form.
Numerical Example
3-class example, true class is "cat" (one-hot: \([1,0,0]\)), model predicts \(\hat{\mathbf{y}}=[0.7, 0.2, 0.1]\):
Two Equivalent Label Formats
| Format | Example (3 classes, true class = index 0) | PyTorch |
|---|---|---|
| One-hot encoded | \([1, 0, 0]\) | Used with manual formula computation |
| Integer class index (sparse) | \(0\) | nn.CrossEntropyLoss expects this format directly |
PyTorch's nn.CrossEntropyLoss is designed around the sparse (integer index) format specifically — it never expects a one-hot vector. This exact distinction is the subject of the next note.
Code
import torch
import torch.nn as nn
logits = torch.tensor([[2.0, 0.5, -1.0]]) # raw scores, softmax applied internally
true_class_index = torch.tensor([0]) # integer index, NOT one-hot
loss_fn = nn.CrossEntropyLoss()
print(loss_fn(logits, true_class_index))
# Manual version using one-hot labels, for comparison
import torch.nn.functional as F
probs = F.softmax(logits, dim=1)
one_hot = torch.tensor([[1.0, 0.0, 0.0]])
manual_loss = -(one_hot * torch.log(probs)).sum(dim=1).mean()
print(manual_loss) # matches the built-in lossCommon Mistakes
- Passing one-hot encoded labels to
nn.CrossEntropyLoss— it expects integer class indices, not one-hot vectors; passing the wrong format either errors or silently produces incorrect gradients. - Applying softmax manually before
nn.CrossEntropyLoss(this exact mistake has been flagged repeatedly across this hub because it's genuinely the most common practical bug beginners hit).
Interview Relevance
Q: "What label format does PyTorch's CrossEntropyLoss expect, and why?" Integer class indices (e.g. 2 for the third class), not one-hot encoded vectors. This is a memory and computation efficiency choice — since only one class is ever "hot" in a one-hot vector, storing just its index is equivalent information with far less memory for large numbers of classes, and PyTorch's implementation is built around this sparse representation directly.
Practice Question
A 4-class model predicts \(\hat{\mathbf{y}}=[0.1, 0.1, 0.6, 0.2]\) for an example whose true class is index 2. Compute the categorical cross-entropy loss for this single example.
