Data drift occurs when the statistical distribution of a model's input data shifts over time, away from the distribution it was originally trained on — one of the most common causes of silent production performance degradation.
Why Data Drift Happens
The real world changes: user demographics shift, seasonal patterns emerge, upstream data sources change their format or content, external events shift behavior patterns entirely. None of this requires any change to the model's code or weights, but it changes what the model actually encounters after deployment — potentially quite different from what it learned during training.
Code — A Statistical Drift Detection Check
from scipy import stats
import numpy as np
def detect_drift(training_feature, production_feature, threshold=0.05):
# Kolmogorov-Smirnov test: checks whether two samples come from the
# same underlying distribution
statistic, p_value = stats.ks_2samp(training_feature, production_feature)
if p_value < threshold:
print(f"WARNING: significant distribution shift detected (p={p_value:.4f})")
return True
return False
# Example: checking whether a numerical feature's production distribution
# has drifted meaningfully from its training-time distribution
drifted = detect_drift(training_data['user_age'], recent_production_data['user_age'])A small p-value here indicates the two distributions are statistically unlikely to be the same — a signal (not definitive proof of harm, but a genuine warning worth investigating) that the production data has shifted meaningfully from what the model was trained on.
Data Drift vs Concept Drift — A Key Distinction
| Data Drift | Concept Drift | |
|---|---|---|
| What changes | The distribution of input features \(P(X)\) | The relationship between inputs and the correct output \(P(Y|X)\) |
| Example | User age distribution shifts younger over time | What counts as "spam" evolves as spammers change tactics — the same input pattern that once meant "not spam" now does |
This distinction, expanded fully in Concept Drift, matters because the two require somewhat different detection approaches — data drift can often be detected purely from input data, while concept drift may require ground-truth labels to detect directly.
Common Mistakes
- Detecting data drift but not investigating whether it actually harms model performance — not every input distribution shift meaningfully degrades predictions; investigating the actual performance impact (when ground truth is available) avoids over-reacting to benign shifts.
- Monitoring only a single, easily-measured feature for drift while ignoring others that may be equally or more predictive — comprehensive monitoring should cover the features the model actually relies on most heavily.
Interview Relevance
Q: "What is data drift, and why can it degrade a deployed model's performance even though the model's code and weights never change?" Data drift is a shift in the statistical distribution of input data a model encounters in production, away from the distribution it was originally trained on — caused by real-world changes like shifting user demographics, seasonal patterns, or upstream data source changes. A model's learned parameters are static once training finishes and reflect the training data's distribution; when production data drifts away from that distribution, the model is effectively being applied outside the range of conditions it learned to handle well, which can silently degrade its real-world accuracy.
Practice Question
A drift detection check flags a significant shift in a feature's distribution, but the model's accuracy (measured against available ground truth) hasn't actually changed. What would you conclude, and what would you check next?
