Random search samples hyperparameter combinations randomly from specified distributions, instead of exhaustively trying every grid combination — counterintuitively, it's often more efficient than grid search at finding good settings, for a genuine mathematical reason.
Why Random Search Often Beats Grid Search
With the same 9-point budget, grid search only tests 3 unique values per hyperparameter axis; random search can test up to 9 — meaningfully better coverage when one hyperparameter matters far more than another.
In practice, most hyperparameters matter far less than a handful of "important" ones — a well-known finding from Bergstra & Bengio's research on this topic. Random search's uneven, scattered sampling explores the important dimensions more thoroughly for the same total budget, since it never wastes samples repeating the same value along an unimportant dimension.
Python Implementation
from sklearn.model_selection import RandomizedSearchCV
from sklearn.svm import SVC
from scipy.stats import uniform, loguniform
param_distributions = {
"C": loguniform(1e-2, 1e2), # sample C on a log scale, not linearly
"gamma": loguniform(1e-3, 1e0),
}
search = RandomizedSearchCV(
SVC(), param_distributions, n_iter=30, cv=5, scoring="accuracy", random_state=42
)
search.fit(X_train, y_train)
print(search.best_params_)Notice n_iter=30 directly controls the budget — unlike grid search, adding another hyperparameter to tune doesn't multiply the cost; you simply search the same fixed number of random combinations across a larger space.
Why Log-Uniform Distributions Matter for Some Hyperparameters
Hyperparameters like \(C\), \(\gamma\), or a learning rate often matter more in terms of order of magnitude (0.001 vs 0.01 vs 0.1) than exact value — sampling uniformly on a linear scale would oversample large values and undersample the small end. loguniform instead samples evenly across orders of magnitude, matching how these hyperparameters actually behave.
Practical Use Cases
- Larger search spaces (4+ hyperparameters) where grid search's exhaustive cost is impractical
- A good default first-pass search strategy before considering the added complexity of Bayesian optimization
Common Mistakes
- Sampling scale-sensitive hyperparameters (like regularization strength) from a linear uniform distribution instead of log-uniform.
- Setting
n_itertoo low for the size of the search space, under-exploring it.
Interview Relevance
Q: "Why can random search outperform grid search with the same computational budget?" When only a few hyperparameters actually matter much (a common real-world pattern), grid search wastes budget repeating unimportant hyperparameter values across many grid cells, while random search's independent sampling per dimension tests far more distinct values along each axis, exploring the truly important dimensions more thoroughly.
Practice Question
Explain, using the diagram's logic, why testing a learning rate hyperparameter with random search (9 samples) would likely find a better value than grid search (9 points across a 3x3 grid), assuming learning rate is the dominant factor.
