Beyond built-in transformers like StandardScaler or OneHotEncoder, real feature engineering often needs custom logic — this note covers wrapping that custom logic into a proper, pipeline-compatible transformer.
Quick Custom Steps — FunctionTransformer
from sklearn.preprocessing import FunctionTransformer
from sklearn.pipeline import Pipeline
import numpy as np
log_transformer = FunctionTransformer(np.log1p) # wraps a plain function as a pipeline step
pipeline = Pipeline([
("log_transform", log_transformer),
("scaler", StandardScaler()),
("model", LogisticRegression()),
])
pipeline.fit(X_train, y_train)FunctionTransformer is the fastest way to drop a simple, stateless function (like a log transform) into a pipeline — no custom class needed.
Custom Transformers With Learned State
For feature engineering that needs to learn something from the training data (like a custom aggregation or a target-encoding-style lookup), write a proper transformer class implementing scikit-learn's fit/transform interface:
from sklearn.base import BaseEstimator, TransformerMixin
import pandas as pd
class RatioFeatureAdder(BaseEstimator, TransformerMixin):
def __init__(self, numerator_col, denominator_col):
self.numerator_col = numerator_col
self.denominator_col = denominator_col
def fit(self, X, y=None):
return self # nothing to learn from data for this simple example
def transform(self, X):
X = X.copy()
X["ratio_feature"] = X[self.numerator_col] / X[self.denominator_col].clip(lower=1)
return X
pipeline = Pipeline([
("add_ratio", RatioFeatureAdder("total_debt", "annual_income")),
("scaler", StandardScaler()),
("model", LogisticRegression()),
])Inheriting from BaseEstimator and TransformerMixin gives the class scikit-learn-compatible behavior for free — fit_transform(), compatibility with GridSearchCV's step__param tuning of the constructor arguments, and correct behavior inside a larger pipeline.
A Transformer That Actually Learns From Data
class FrequencyEncoder(BaseEstimator, TransformerMixin):
def fit(self, X, y=None):
self.freq_map_ = X.value_counts(normalize=True).to_dict() # LEARNED from training data
return self
def transform(self, X):
return X.map(self.freq_map_).fillna(0).to_frame()
# Fit on train only, apply consistently to test -- exactly the leak-prevention
# discipline a pipeline enforces automatically
encoder = FrequencyEncoder()
encoder.fit(X_train["city"])
X_test_encoded = encoder.transform(X_test["city"])This is the same fit/transform discipline covered in Data Leakage — anything "learned" from data (like this frequency map) belongs in fit(), applied later via transform(), never computed fresh on test data.
Practical Use Cases
- Domain-specific feature engineering (ratios, date-time extraction, custom aggregations) that no built-in transformer covers
- Making custom feature logic reusable, testable, and compatible with grid search and cross-validation
Common Mistakes
- Writing feature engineering as loose, standalone functions instead of proper transformer classes — this loses pipeline compatibility, cross-validation-safe refitting, and grid search integration.
- Learning something from data inside
transform()instead offit()— this silently recomputes on every call, including on test data, defeating the entire purpose of the fit/transform split.
Interview Relevance
Q: "How would you add a custom, dataset-specific feature engineering step to a scikit-learn pipeline?" Write a class inheriting from BaseEstimator and TransformerMixin, implementing fit() (for anything that needs to be learned from training data) and transform() (applying that learned logic) — this makes the custom step behave exactly like any built-in transformer within a Pipeline.
Practice Question
Sketch a custom transformer class that adds a "days_since_signup" feature from a "signup_date" column, computed relative to a fixed reference date passed to the constructor.
