Once forward propagation produces a prediction, something has to measure how wrong it is — that's the job of the loss function. This note introduces the concept in general terms; the full Loss Functions category later covers every specific formula in depth.
What a Loss Function Must Do
A loss function \(L(y_{\text{true}}, y_{\text{pred}})\) takes the true label and the model's prediction and returns a single non-negative number: 0 (or near 0) for a perfect prediction, larger values for worse predictions. This single number is what backpropagation and gradient descent work to minimize.
Two Loss Families, Introduced Here
| Task | Typical Loss | Intuition |
|---|---|---|
| Regression | Mean Squared Error: \((y_{\text{true}}-y_{\text{pred}})^2\) | Penalizes the squared distance between predicted and true numeric values |
| Classification | Cross-Entropy: \(-\log(P(\text{correct class}))\) | Penalizes how far the predicted probability for the correct class is from 1 (see Cross-Entropy) |
Numerical Example
Regression: true value \(y=10\), predicted \(\hat y=8\). MSE loss \(=(10-8)^2=4\). Classification: true class is "cat," model predicts \(P(\text{cat})=0.6\). Cross-entropy loss \(=-\log(0.6)\approx0.511\).
Loss on One Example vs Loss Over a Batch
In practice, training computes the loss averaged across a whole batch of examples at once (see Expected Value for why this average is itself an estimate of the true expected loss), not just one example at a time — this average is what a single call to loss.backward() differentiates.
Code
import torch
import torch.nn as nn
# Regression example
mse_loss = nn.MSELoss()
y_true = torch.tensor([10.0])
y_pred = torch.tensor([8.0])
print(mse_loss(y_pred, y_true)) # tensor(4.)
# Classification example
ce_loss = nn.CrossEntropyLoss()
logits = torch.tensor([[1.5, 0.2, -0.5]]) # raw scores, softmax applied internally
true_label = torch.tensor([0])
print(ce_loss(logits, true_label))Loss vs Evaluation Metric — A Reminder
As already flagged in Components of a Deep Learning System, the loss function must be differentiable (gradient descent needs to compute its derivative), while the evaluation metrics you actually report (accuracy, F1, etc.) don't need to be, and often measure something subtly different. A model can have a lower cross-entropy loss while having the same accuracy as another — the loss captures confidence calibration that raw accuracy ignores.
Common Mistakes
- Choosing a loss function that doesn't match the task's output distribution assumption — e.g. using MSE for a classification task, which doesn't penalize confidently wrong predictions nearly as sharply as cross-entropy does (see the derivation in Maximum Likelihood Estimation).
- Watching only the loss curve and assuming a decreasing loss automatically means the model is getting better at the task in every sense that matters — always also track a task-relevant evaluation metric.
Interview Relevance
Q: "Why not just use accuracy directly as the loss function for training a classifier?" Accuracy is not differentiable — it's a step function based on whether the predicted class matches the true class, with zero gradient almost everywhere. Gradient descent needs a smooth, differentiable loss (like cross-entropy) that provides a useful gradient signal even for predictions that are "close but not quite correct."
Practice Question
A regression model predicts \(\hat y = 15\) when the true value is \(y=12\). Compute both the MSE loss and the MAE (mean absolute error, \(|y-\hat y|\)) for this single example. Which penalizes this specific error more heavily?
