Binary cross-entropy (BCE) is cross-entropy specialized for the two-class case — the standard loss for any binary classification network, always paired with a sigmoid output.
Formula
\(y_i \in \{0,1\}\) is the true label, \(\hat{y}_i \in (0,1)\) is the model's predicted probability (from sigmoid). Only one of the two terms is ever "active" for a given example: if \(y_i=1\), the second term vanishes (\(1-y_i=0\)) and the loss is \(-\log(\hat y_i)\); if \(y_i=0\), the first term vanishes and the loss is \(-\log(1-\hat y_i)\) — exactly the general cross-entropy formula from Cross-Entropy, specialized to 2 classes.
Numerical Example
Three examples: \(y=1, \hat y=0.9\) → \(-\log(0.9)\approx0.105\). \(y=0, \hat y=0.2\) → \(-\log(1-0.2)=-\log(0.8)\approx0.223\). \(y=1, \hat y=0.3\) → \(-\log(0.3)\approx1.204\) (a confidently wrong-ish prediction, penalized much more heavily).
The Critical Implementation Detail: BCELoss vs BCEWithLogitsLoss
| PyTorch Loss | Expects as Input | Applies Sigmoid Internally? |
|---|---|---|
nn.BCELoss | Already-sigmoided probabilities in (0,1) | No — you must apply torch.sigmoid() yourself first |
nn.BCEWithLogitsLoss | Raw logits (any real number) | Yes — combines sigmoid + BCE in one numerically stable operation |
BCEWithLogitsLoss is generally preferred: computing \(\log(\sigma(z))\) directly (rather than computing \(\sigma(z)\) first, then taking its log separately) avoids numerical instability for very negative or very positive logits — the same numerical-stability principle behind softmax's max-subtraction trick from Softmax Function.
Code
import torch
import torch.nn as nn
# Correct: BCEWithLogitsLoss expects raw logits, applies sigmoid internally
logits = torch.tensor([2.0, -1.5, 0.5])
labels = torch.tensor([1.0, 0.0, 1.0])
loss_fn = nn.BCEWithLogitsLoss()
print(loss_fn(logits, labels))
# Equivalent but less numerically stable: manual sigmoid + BCELoss
probs = torch.sigmoid(logits)
loss_fn_manual = nn.BCELoss()
print(loss_fn_manual(probs, labels)) # same result, computed less safelyCommon Mistakes
- Applying sigmoid manually and then using
BCEWithLogitsLoss— this double-applies sigmoid, exactly the same class of bug flagged for softmax +CrossEntropyLoss. - Using
BCELossdirectly on raw logits (forgetting the sigmoid entirely) — this passes values outside (0,1) into a log-based formula that expects probabilities, producing nonsensical (often NaN) losses.
Interview Relevance
Q: "Why does PyTorch recommend BCEWithLogitsLoss over manually applying sigmoid then BCELoss?" Numerical stability — computing the log of a sigmoid output separately can lose precision or produce NaN for extreme logit values, while BCEWithLogitsLoss combines the sigmoid and log-loss computation into a single, more numerically stable operation (similar to how CrossEntropyLoss combines softmax and log internally).
Practice Question
For a single example with true label \(y=0\) and predicted probability \(\hat y = 0.05\), compute the binary cross-entropy loss. Is this a small or large loss value, and does that match your intuition for a confident, correct prediction?
