Once training completes, evaluation answers the question defined all the way back in DL Problem Definition: did the model actually achieve the success criteria set upfront?
Choosing the Right Metrics for the Task
| Task Type | Appropriate Metrics |
|---|---|
| Balanced classification | Accuracy, F1 score |
| Imbalanced classification | Precision, Recall, F1, PR-AUC — accuracy alone can be badly misleading (see Bayes' Theorem's disease-test example) |
| Regression | MAE, RMSE, R² — chosen based on how outliers should be weighted |
| Object detection | mAP |
| Segmentation | IoU, Dice score |
| Language generation | Perplexity, BLEU, ROUGE (each with real limitations, best combined with human evaluation) |
Every one of these metrics is covered in full mathematical depth in the Evaluation Metrics category — this note is specifically about applying the right one, chosen based on the problem definition, not defaulting to whichever metric is most familiar or easiest to compute.
Evaluating Against the Held-Out Test Set — Exactly Once
# The test set is touched ONLY here, at the very end, after all model/hyperparameter
# decisions have already been finalized using validation performance alone
final_test_metrics = evaluate(best_model, test_loader)
print(final_test_metrics)
# This number is the honest, final answer to "how well does this model actually work"Comparing Against Baselines
A model's raw metric value means little in isolation — reporting it alongside the simple baseline established in Model Selection, and ideally against any existing solution the new model is meant to replace, gives a genuinely meaningful sense of whether the added complexity and cost were actually worth it.
Common Mistakes
- Reporting only accuracy for a severely imbalanced classification task — as covered extensively in this hub, this can be dramatically misleading, sometimes making a model that never predicts the minority class at all appear deceptively strong.
- Evaluating on the test set multiple times during development to "check progress" — this exact behavior silently converts the test set into a second validation set, undermining its purpose as an honest final estimate.
Interview Relevance
Q: "Why is it important to compare a new deep learning model's performance against a simple baseline, not just report its metric value in isolation?" A metric value alone doesn't reveal whether the added complexity and cost of the deep learning solution actually provided meaningful benefit — a model that's only marginally better than a much simpler, cheaper baseline may not justify its added engineering, compute, and maintenance cost. Comparing against a clear baseline gives a genuinely meaningful sense of the model's real value, not just an isolated number.
Practice Question
For a highly imbalanced fraud detection task (1% fraud rate), why would reporting only accuracy be actively misleading, and what metrics would you report instead?
