Implementation exercises for model evaluation — computing classification metrics from scratch, and interpreting evaluation results to diagnose model behavior.
🟢 Problem 1: Compute a confusion matrix and derived metrics from scratch
Task: Given predicted and true labels for a binary classification task, compute the confusion matrix and derive accuracy, precision, recall, and F1 score without using any library function.
def compute_metrics(y_true, y_pred):
tp = sum((yt == 1 and yp == 1) for yt, yp in zip(y_true, y_pred))
tn = sum((yt == 0 and yp == 0) for yt, yp in zip(y_true, y_pred))
fp = sum((yt == 0 and yp == 1) for yt, yp in zip(y_true, y_pred))
fn = sum((yt == 1 and yp == 0) for yt, yp in zip(y_true, y_pred))
accuracy = (tp + tn) / (tp + tn + fp + fn)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
return {"TP": tp, "TN": tn, "FP": fp, "FN": fn,
"accuracy": accuracy, "precision": precision, "recall": recall, "f1": f1}
y_true = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
y_pred = [1, 0, 0, 1, 0, 1, 1, 0, 1, 0]
metrics = compute_metrics(y_true, y_pred)
for k, v in metrics.items():
print(f"{k}: {v}")Hint if stuck: Verify your implementation against sklearn.metrics.precision_score, recall_score, and f1_score on the same y_true/y_pred — they should match exactly.
🟡 Problem 2: Implement ROC curve computation from scratch
Task: Given predicted probabilities (not hard labels) and true labels, compute the true positive rate and false positive rate at several threshold values, and plot the resulting ROC curve.
def compute_roc_points(y_true, y_scores, thresholds):
points = []
for t in thresholds:
y_pred = [1 if s >= t else 0 for s in y_scores]
tp = sum((yt == 1 and yp == 1) for yt, yp in zip(y_true, y_pred))
fn = sum((yt == 1 and yp == 0) for yt, yp in zip(y_true, y_pred))
fp = sum((yt == 0 and yp == 1) for yt, yp in zip(y_true, y_pred))
tn = sum((yt == 0 and yp == 0) for yt, yp in zip(y_true, y_pred))
tpr = tp / (tp + fn) if (tp + fn) > 0 else 0
fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
points.append((fpr, tpr))
return points
y_scores = [0.9, 0.1, 0.4, 0.8, 0.3, 0.7, 0.6, 0.2, 0.85, 0.15]
thresholds = np.linspace(0, 1, 20)
roc_points = compute_roc_points(y_true, y_scores, thresholds)
fprs, tprs = zip(*roc_points)
plt.plot(fprs, tprs, marker='o')
plt.plot([0, 1], [0, 1], linestyle='--', color='gray') # random-guess baseline
plt.xlabel('False Positive Rate'); plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.show()🔴 Problem 3: Implement k-fold cross-validation from scratch
Task: Implement 5-fold cross-validation manually (without sklearn.model_selection.KFold) — split data into 5 folds, train on 4, evaluate on the held-out 1, repeat for each fold, and report the mean and standard deviation of validation accuracy.
def k_fold_cross_validation(X, y, k=5, train_fn=None, eval_fn=None):
n = len(X)
indices = np.random.permutation(n)
fold_size = n // k
fold_scores = []
for fold in range(k):
val_indices = indices[fold * fold_size:(fold + 1) * fold_size]
train_indices = np.setdiff1d(indices, val_indices)
X_train, y_train = X[train_indices], y[train_indices]
X_val, y_val = X[val_indices], y[val_indices]
model = train_fn(X_train, y_train)
score = eval_fn(model, X_val, y_val)
fold_scores.append(score)
print(f"Fold {fold}: accuracy = {score:.4f}")
print(f"Mean: {np.mean(fold_scores):.4f} +/- {np.std(fold_scores):.4f}")
return fold_scoresHint if stuck: The key correctness check: across all 5 folds, every single example should appear in the validation set exactly once, and never in both train and validation for the same fold — verify this explicitly with a small test case before trusting the implementation on real data.
🟡 Problem 4: Diagnose a model's behavior purely from its confusion matrix
Task: Given the confusion matrix below for a 3-class classifier, identify which class the model struggles with most, and what specific error pattern it makes.
# Predicted: Cat Dog Bird
# Actual Cat: 85 10 5
# Actual Dog: 8 88 4
# Actual Bird: 3 35 62
# Analyze: which class has the lowest recall? Which two classes get confused most often?
confusion = np.array([[85, 10, 5], [8, 88, 4], [3, 35, 62]])
class_names = ['Cat', 'Dog', 'Bird']
for i, name in enumerate(class_names):
recall = confusion[i, i] / confusion[i].sum()
print(f"{name} recall: {recall:.3f}")
# Expected finding: Bird has the lowest recall (62/100 = 0.62), and it's most
# frequently confused with Dog (35 misclassifications) -- far more than with
# Cat (3) -- suggesting Bird and Dog share features the model conflates,
# worth investigating via error analysis on the specific misclassified examples