Hyperparameter tuning searches for the settings — like k in KNN, max_depth in a tree, or C in an SVM — that make a model perform best, since these can't be learned from data the way a model's actual parameters are.
The Three Main Search Strategies
| Strategy | How It Searches | Full Note |
|---|---|---|
| Grid Search | Exhaustively tries every combination in a specified grid | Thorough, but expensive as dimensions grow |
| Random Search | Randomly samples combinations from specified distributions | Often more efficient in high-dimensional search spaces |
| Bayesian Optimization | Uses past results to intelligently choose the next combination to try | Most sample-efficient, more complex to set up |
Why Tuning Matters — A Concrete Reminder
An untuned model isn't automatically "safe" — a default SVM's regularization strength, a default KNN's \(k=5\), or a default Random Forest's tree depth are just one point in a much larger space of possible settings, usually not the best one for a specific dataset. See the recurring example throughout this hub: choosing k in KNN, tuning C in SVM, and tuning tree depth are all instances of this exact same tuning problem.
Minimal Working Example
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
param_grid = {"C": [0.1, 1, 10], "gamma": [0.01, 0.1]}
search = GridSearchCV(SVC(), param_grid, cv=5, scoring="accuracy")
search.fit(X_train, y_train)
print(search.best_params_)
print(search.best_score_)Practical Use Cases
- Squeezing meaningful additional accuracy out of an already-reasonable model
- Systematically comparing candidate settings instead of manual, ad hoc trial and error
Common Mistakes
- Tuning hyperparameters using the test set instead of cross-validation on the training set — see Cross-Validation for Hyperparameter Tuning.
- Tuning far more hyperparameters simultaneously than necessary, wasting compute on dimensions that barely affect performance.
Interview Relevance
Q: "Why can't hyperparameters be learned the same way model parameters are?" Model parameters (like regression coefficients) are learned by optimizing a differentiable loss function directly from training data; hyperparameters (like tree depth or k) control the model's structure or the training process itself, and evaluating a specific hyperparameter choice requires actually training and validating a model — it's a search problem, not a direct optimization one.
Practice Question
You have limited compute budget and 6 hyperparameters to tune, each with a wide range of possible values. Which search strategy would you reach for first, and why?