A preprocessing pipeline chains every cleaning, encoding and scaling step into a single object — so the exact same transformations fit on training data are guaranteed to apply, in the same order, to new data at prediction time.
Why Not Just Write Preprocessing Steps as Separate Lines?
Manually applying preprocessing step by step works until you deploy the model — then you must remember to apply the identical sequence, with the identical fitted parameters (the training set's mean, the training set's categories), to every new prediction request. A pipeline makes that automatic instead of a manual, error-prone reimplementation.
A Realistic Pipeline with Mixed Column Types
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
numeric_features = ["age", "income"]
categorical_features = ["city", "payment_method"]
numeric_pipeline = Pipeline([
("imputer", SimpleImputer(strategy="median")),
("scaler", StandardScaler()),
])
categorical_pipeline = Pipeline([
("imputer", SimpleImputer(strategy="most_frequent")),
("encoder", OneHotEncoder(handle_unknown="ignore")),
])
preprocessor = ColumnTransformer([
("num", numeric_pipeline, numeric_features),
("cat", categorical_pipeline, categorical_features),
])
full_pipeline = Pipeline([
("preprocessing", preprocessor),
("model", LogisticRegression(max_iter=1000)),
])
full_pipeline.fit(X_train, y_train) # fits imputers, scaler, encoder AND the model together
predictions = full_pipeline.predict(X_test) # applies the exact same fitted steps to new dataExpected behavior: calling .fit() once trains every step in the correct order; calling .predict() on new data automatically applies the same fitted imputers, scaler and encoder — no manual reapplication, and no risk of accidentally fitting on test data.
Why This Directly Prevents Data Leakage
Inside cross-validation, a Pipeline refits its preprocessing steps within each fold, using only that fold's training data — this is exactly the discipline that manual preprocessing tends to get wrong. See Data Leakage and Pipeline & Data Leakage.
Practical Use Cases
- Any production ML system — pipelines are how preprocessing logic ships alongside the model
- Cross-validation and hyperparameter tuning, where preprocessing must be refit per fold
- Serializing an entire preprocessing+model workflow with
joblibas a single deployable object
Common Mistakes
- Preprocessing data manually outside the pipeline "just this once," then forgetting to replicate that exact step in production.
- Not using
ColumnTransformerwhen numeric and categorical columns need different treatment — applying a scaler to a categorical column, or an encoder to a numeric one, by accident.
Interview Relevance
Q: "Why use a scikit-learn Pipeline instead of preprocessing manually?" Reproducibility and leakage prevention — a pipeline guarantees the exact fitted transformations from training are applied consistently to new data, and refits correctly within each cross-validation fold instead of leaking test-fold information.
Practice Question
You have numeric columns needing standardization and categorical columns needing one-hot encoding. Sketch the ColumnTransformer structure you'd use to handle both in a single pipeline.
