Feature scaling puts numeric features on a comparable range. Without it, a feature measured in the thousands (income) can completely dominate a feature measured in single digits (years of experience) — not because it's more important, but purely because of its scale.
Before / After, Visually
After scaling, both features occupy a comparable numeric range — neither dominates a distance or gradient calculation purely by scale.
Which Models Actually Need It
| Sensitive to Scale | Not Sensitive to Scale |
|---|---|
| KNN, K-Means (distance-based) | Decision Trees, Random Forest (threshold-based splits) |
| SVM (margin-based) | Gradient Boosting / XGBoost (also threshold-based) |
| Linear/Logistic Regression, Neural Networks (gradient-based) | Naive Bayes (probability-based, not distance-based) |
The Three Main Techniques
- Standardization — rescales to mean 0, standard deviation 1; the default choice for most models
- Normalization (Min-Max) — rescales into a fixed [0, 1] range; useful when you need bounded values
- Robust Scaling — uses median and IQR instead of mean and standard deviation; the right choice when outliers are present
Common Mistakes
- Scaling before splitting into train/test — leaks test-set statistics into the scaler. Always
fitthe scaler on training data only. - Scaling features for tree-based models unnecessarily — harmless, but wasted effort; trees split on thresholds, not distances or gradients.
- Forgetting to scale the target variable too, when the algorithm requires it (rare, but relevant for some regularized regression setups).
Interview Relevance
Q: "Does Random Forest need feature scaling?" No — tree-based models split on a single feature's threshold at a time (e.g. "income > 50000"), so the relative scale between different features never affects the split decision, unlike distance- or gradient-based models.
Practice Question
You're training a KNN classifier on features "age" (18–70) and "income" (20,000–500,000) without scaling. Explain what will go wrong and why.