This note ties the entire Hyperparameter Tuning category into the project lifecycle specifically — when tuning should actually happen, and how to allocate limited compute across it.
When to Tune — After a Working Baseline, Not Before
Hyperparameter tuning is most valuable and most efficient after a working baseline model, correctly implemented and reasonably performing, already exists — tuning hyperparameters against a fundamentally broken pipeline (from a bug, not a hyperparameter issue) wastes significant compute chasing improvements that a simple implementation fix would have delivered instantly and for free.
The Recommended Order
- Get a correct, working implementation first (per Model Training's "overfit a tiny batch" sanity check).
- Establish a reasonable baseline using sensible hyperparameter defaults (per the individual tuning notes: Learning Rate Tuning, Optimizer Selection, etc.).
- Tune the highest-impact hyperparameters first — learning rate, then architecture size/depth, then regularization strength.
- Only invest in systematic/automated search (Random Search, Optuna) once manual, intuition-guided tuning has been reasonably exhausted, or when the remaining performance gap genuinely justifies the additional compute cost.
Allocating a Limited Tuning Budget
# A practical compute-budgeting approach
total_compute_budget = 100 # e.g. 100 GPU-hours available for this project
allocation = {
"initial implementation debugging": 5,
"baseline establishment": 10,
"manual hyperparameter tuning": 20,
"automated search (Optuna, etc.)": 40,
"final model training (best config, full data)": 20,
"buffer for unexpected issues": 5,
}
# Planning this explicitly, rather than tuning indefinitely until compute runs out
# unplanned, keeps a project on track to actually finishCommon Mistakes
- Beginning extensive hyperparameter search before confirming the underlying implementation is actually correct — tuning can't fix a genuine bug, and time spent doing so before verifying correctness is largely wasted.
- Allocating tuning compute without any explicit plan or budget, tuning indefinitely until the project simply runs out of time or compute — planning the budget upfront (as in the code example) keeps a project on a realistic, deliverable timeline.
Interview Relevance
Q: "Why should hyperparameter tuning generally happen after establishing a correct, working baseline implementation, rather than in parallel with initial development?" Tuning hyperparameters against a fundamentally buggy implementation wastes compute chasing performance that a simple bug fix would deliver instantly and for free — worse, poor results from a broken implementation can be misdiagnosed as a hyperparameter or modeling problem, sending debugging effort in the wrong direction entirely. Confirming correctness first (e.g. via the tiny-batch overfitting sanity check) ensures tuning effort is actually spent productively.
Key Takeaways — DL Project Development (Part 1)
- Problem definition and success metrics should be settled before any modeling begins, not decided retroactively after seeing results.
- Data collection, exploration, cleaning, preprocessing, and augmentation each address a specific, distinct concern — and must be applied consistently across train/validation/test splits without leaking information between them.
- Model selection should start from a simple baseline and scale complexity only as justified by demonstrated need, informed by data size and deployment constraints.
- Training should be verified correct on a small scale before a full, expensive run; hyperparameter tuning is most valuable once a correct, working baseline already exists.
Practice Question
Why is it more efficient to fix implementation bugs before starting hyperparameter tuning, rather than tuning hyperparameters in parallel with debugging?
