ML monitoring continuously tracks a deployed model's health — not just whether the service is up, but whether its predictions are still trustworthy — since a model can fail silently in ways a standard uptime check would never catch.
What to Actually Monitor
| Category | What to Track | Why |
|---|---|---|
| System health | Latency, error rate, uptime | Standard software monitoring, still necessary |
| Input data | Feature distributions, missing value rates | Catches data drift and upstream pipeline bugs |
| Predictions | Output distribution, class balance of predictions | A sudden shift can signal a broken model or a genuine real-world change |
| Performance | Accuracy/F1/RMSE, whenever true labels eventually become available | The most direct signal of model drift |
Minimal Monitoring Implementation
import numpy as np
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("model_monitor")
def predict_with_monitoring(model, features, reference_stats):
# Log basic input statistics for drift comparison later
logger.info(f"Input feature means: {features.mean(axis=0)}")
prediction = model.predict(features)
probability = model.predict_proba(features)[:, 1]
# Flag low-confidence predictions for review
if probability.mean() < 0.55 and probability.mean() > 0.45:
logger.warning("Batch has unusually low average confidence -- possible drift")
return prediction, probabilityThe Delayed-Label Problem
Unlike system latency (known instantly), a model's true accuracy often can't be measured until real outcomes are known — a loan default might not be confirmed for months, a churn prediction's accuracy only resolves once you see whether the customer actually left. This delay is exactly why monitoring input and prediction distributions matters as an early warning signal — they're available immediately, well before ground-truth labels catch up.
Setting Alerting Thresholds
# A simple threshold-based alert -- more sophisticated systems use
# statistical drift tests (see Data Drift) instead of a fixed cutoff
baseline_mean_confidence = 0.78 # recorded from validation data
def check_confidence_drift(current_batch_confidence, threshold=0.10):
drift = abs(current_batch_confidence - baseline_mean_confidence)
if drift > threshold:
send_alert(f"Confidence drift detected: {drift:.3f} change from baseline")
check_confidence_drift(current_batch_confidence=0.61) # triggers -- 0.17 change > 0.10 thresholdPractical Use Cases
- Every production model, without exception — monitoring is what turns "deployed" into "reliably operated"
- Early detection of upstream data pipeline bugs, which often show up first as input distribution anomalies
Common Mistakes
- Only monitoring system-level metrics (uptime, latency) and assuming that's sufficient — a model can be perfectly "up" while making increasingly wrong predictions.
- Waiting for ground-truth labels before checking for any problems, when input/prediction distribution monitoring can surface issues much earlier.
Interview Relevance
Q: "Why isn't uptime monitoring enough for a production ML model?" A model can be fully "up" and responding instantly while its predictions have become systematically wrong due to drift or an upstream data bug — uptime monitoring only catches the service failing outright, not the model quietly making bad predictions while functioning normally.
Practice Question
Your model's true labels (loan defaults) aren't known for 6 months after prediction. Propose two monitoring signals you could check well before that delay resolves.
