This note ties together every earlier reference to leak-safe preprocessing — showing precisely, mechanically, how a scikit-learn Pipeline prevents data leakage during cross-validation, in a way manual preprocessing structurally cannot.
The Leak, Made Concrete — Manual Preprocessing
# WRONG: fit the scaler once, on the full training set, BEFORE cross-validation
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # sees ALL of X_train, including every CV fold's "test" portion
model = LogisticRegression()
scores = cross_val_score(model, X_train_scaled, y_train, cv=5)
# Each fold's "test" data was already seen by the scaler when it computed
# its mean/std from the FULL X_train -- a subtle but real leakEven though the model is retrained per fold correctly, the scaler was fit once on the entire training set — meaning every fold's held-out portion already influenced the scaling statistics applied to it. This is a smaller leak than fitting on the test set outright, but it's still real, and it still inflates cross-validated scores somewhat.
The Fix — Pipeline Refits Every Step Per Fold
# RIGHT: wrap the scaler and model together
from sklearn.pipeline import Pipeline
pipeline = Pipeline([
("scaler", StandardScaler()),
("model", LogisticRegression()),
])
scores = cross_val_score(pipeline, X_train, y_train, cv=5)
# cross_val_score calls pipeline.fit() SEPARATELY for each fold's training portion --
# the scaler is refit from scratch each time, using ONLY that fold's training dataWhen a Pipeline is passed to cross_val_score or GridSearchCV, every step — including preprocessing — gets refit independently within each fold, using only that fold's training partition. The held-out fold never influences any fitted statistic used to transform it.
Why This Matters More Than It Looks Like It Should
| Leak Severity | Scenario |
|---|---|
| Severe | Fitting a scaler/encoder on the full dataset (train + test) before any split at all |
| Moderate but real | Fitting on the full training set before cross-validation folds (the example above) |
| Prevented entirely | Using a Pipeline inside cross-validation, refitting every step per fold |
The "moderate" leak is easy to introduce accidentally and easy to miss in a code review — a Pipeline eliminates the entire category of mistake structurally, rather than relying on the developer remembering the correct manual order every single time.
This Applies to Feature Selection and Target Encoding Too
# Feature selection needs the SAME discipline
from sklearn.feature_selection import SelectKBest, f_classif
pipeline = Pipeline([
("select", SelectKBest(f_classif, k=10)), # refit per fold -- which features are "best" can change per fold
("model", LogisticRegression()),
])
# WITHOUT the pipeline, selecting features once on the full training set before
# cross-validation leaks information about which features looked good on data
# that later becomes each fold's held-out portionPractical Use Cases
- Any cross-validated evaluation or hyperparameter search involving fitted preprocessing, feature selection, or target encoding
- Explaining to a team why "just fit the scaler once at the top of the script" is a subtle but real bug, not a harmless shortcut
Common Mistakes
- Fitting any preprocessing or feature-selection step once, outside a Pipeline, before running cross-validation.
- Assuming this leak is negligible because it "only" involves the training set, not the test set — it still inflates cross-validated scores and can mislead hyperparameter selection.
Interview Relevance
Q: "Is it still data leakage if you fit a scaler on the full training set before cross-validation, but never touch the actual test set?" Yes — each cross-validation fold's held-out portion is drawn from that training set, so fitting the scaler on all of it beforehand means every fold's "unseen" data already influenced the scaling statistics applied to it; it's a smaller leak than touching the true test set, but it's still real and still inflates reported CV performance.
Practice Question
A teammate runs SelectKBest once on the full training set, then passes the reduced feature set into cross_val_score. Explain the leak this introduces and how a Pipeline would fix it.
