This note expands on the preview from DL Monitoring into the full practice of ML monitoring — the systems and metrics that keep a deployed model's health visible on an ongoing basis.
The Layers of Production ML Monitoring
| Layer | What's Tracked | Detail Covered In |
|---|---|---|
| Data quality | Input data distribution shifts, missing value rates, schema violations | Data Drift |
| Prediction behavior | Output distribution shifts, confidence score trends | Concept Drift |
| Model accuracy | Direct performance metrics, once ground-truth labels become available | Model Drift |
| System health | Latency, throughput, error rates, resource utilization | Inference Latency, GPU Utilization |
Code — A Basic Monitoring Dashboard Structure
import logging
import time
class ModelMonitor:
def __init__(self):
self.prediction_log = []
def log_prediction(self, input_features, prediction, confidence, latency_ms):
self.prediction_log.append({
'timestamp': time.time(),
'input_features': input_features,
'prediction': prediction,
'confidence': confidence,
'latency_ms': latency_ms,
})
def compute_recent_stats(self, window_minutes=60):
cutoff = time.time() - window_minutes * 60
recent = [p for p in self.prediction_log if p['timestamp'] > cutoff]
return {
'count': len(recent),
'avg_confidence': sum(p['confidence'] for p in recent) / max(len(recent), 1),
'avg_latency_ms': sum(p['latency_ms'] for p in recent) / max(len(recent), 1),
'p99_latency_ms': sorted(p['latency_ms'] for p in recent)[int(len(recent) * 0.99)] if recent else 0,
}In a real production system, this kind of logging typically feeds into a dedicated monitoring/observability platform (e.g. Grafana, Prometheus, or a specialized ML monitoring tool), with automated alerting when metrics cross concerning thresholds — rather than requiring someone to manually check a dashboard.
Alerting — Making Monitoring Actionable
Monitoring data is only valuable if it leads to action — configuring automated alerts (e.g. "notify the on-call engineer if p99 latency exceeds 500ms for 5 consecutive minutes," or "flag if average prediction confidence drops more than 10% week-over-week") turns passive dashboards into an active safety net that catches problems quickly, rather than relying on someone noticing a slow decline by chance.
Common Mistakes
- Building monitoring dashboards without any automated alerting — a dashboard nobody is actively watching provides little practical protection against a real production problem.
- Monitoring only system health (latency, uptime) while ignoring model-specific signals (confidence trends, prediction distribution) — a model can be "healthy" from an infrastructure standpoint while its actual predictions quietly degrade in quality.
Interview Relevance
Q: "Why is monitoring a deployed model's prediction confidence and output distribution valuable, in addition to standard system health metrics like latency and uptime?" System health metrics confirm the service is running and responsive, but say nothing about whether the model's actual predictions remain accurate and reliable — a model can be perfectly "healthy" from an infrastructure standpoint while data or concept drift silently degrades its real-world prediction quality. Tracking prediction-specific signals (confidence trends, output distribution shifts) provides earlier, more direct visibility into this kind of degradation, often well before it would show up in a system health dashboard.
Practice Question
Why does monitoring provide more value when paired with automated alerting, rather than existing only as a dashboard someone has to remember to check?
