Model drift is the umbrella term for a deployed model's predictions gradually becoming less accurate over time — the general phenomenon of "model degradation," caused by either the input data or the underlying relationship it was trained on changing.
Two Distinct Causes, Both Called "Drift"
| Cause | What Changes | Example |
|---|---|---|
| Data Drift (covariate shift) | The distribution of input features | Average customer age shifts upward over a year |
| Concept Drift | The actual relationship between features and target | What predicted "high spender" pre-pandemic no longer predicts it post-pandemic |
Both ultimately cause the same symptom — declining real-world accuracy, "model degradation" — but they call for different diagnoses: data drift is detectable by comparing input distributions alone (no labels needed); concept drift requires comparing actual model accuracy against ground truth over time, since the inputs might look perfectly normal while what they mean has changed.
Detecting Drift — Population Stability Index (PSI)
\(Expected\%_i\) is a feature's proportion in bin \(i\) during training; \(Actual\%_i\) is its proportion in the same bin in current production data.
Worked Example
An "income" feature, bucketed into 3 bins:
| Bin | Training % | Production % |
|---|---|---|
| Low | 30% | 15% |
| Medium | 50% | 45% |
| High | 20% | 40% |
import numpy as np
expected = np.array([0.30, 0.50, 0.20])
actual = np.array([0.15, 0.45, 0.40])
psi = np.sum((actual - expected) * np.log(actual / expected))
print(round(psi, 3)) # 0.248Reading the PSI Value
| PSI | Interpretation |
|---|---|
| < 0.10 | No significant shift |
| 0.10 – 0.25 | Moderate shift — worth monitoring closely |
| > 0.25 | Significant shift — investigate and likely retrain |
The worked example's PSI (0.248) sits right at the moderate/significant boundary — exactly the kind of borderline result that warrants a closer look rather than an automatic alarm or automatic dismissal.
Practical Use Cases
- Automated drift alerts that trigger a closer investigation or a retraining pipeline — see ML Retraining
- Explaining a mysterious production accuracy decline to stakeholders with concrete, quantified evidence
Common Mistakes
- Using the term "drift" without specifying whether it's data drift or concept drift — the appropriate response differs (retraining on fresh data often fixes data drift; concept drift may require rethinking features or the modeling approach entirely).
- Only checking for drift after a business metric has already visibly declined, instead of proactively monitoring input distributions continuously.
Interview Relevance
Q: "What's the difference between data drift and concept drift, and why does the distinction matter?" Data drift is a shift in input feature distributions (detectable without labels, via something like PSI); concept drift is a change in the actual relationship between features and target (requires comparing predictions against ground truth) — the distinction matters because the fix differs: data drift often just needs retraining on recent data, while concept drift may mean the model's fundamental assumptions no longer hold.
Practice Question
A feature's PSI is 0.05 (low), but the model's accuracy against ground truth has clearly dropped. What kind of drift does this combination suggest?
