This distinction trips up nearly every deep learning beginner at least once: parameters are learned automatically by training; hyperparameters are chosen by you, before training even starts, and never change during it.
The Distinction
| Parameters | Hyperparameters | |
|---|---|---|
| Who sets them | The training algorithm (gradient descent) | You, the practitioner, before training |
| When they change | Every training step | Fixed during a single training run; you might change them between different runs |
| Examples | Weights, biases | Learning rate, number of layers, batch size, number of epochs, dropout rate, choice of optimizer |
| How they're found | Backpropagation + gradient descent | Manual tuning, grid/random search, Bayesian optimization (see Hyperparameter Tuning category) |
A Concrete List — Sorting the Two Apart
For a typical MLP training run, ask "does this value get updated by optimizer.step()?" If yes, it's a parameter. If it's set once in your code before the training loop starts and never touched by the optimizer, it's a hyperparameter:
- Parameters: every entry of every weight matrix, every bias value.
- Hyperparameters: learning rate, number of hidden layers, neurons per layer, batch size, number of epochs, choice of activation function, choice of optimizer, dropout probability, weight decay strength.
Code — Seeing the Split Directly
import torch.nn as nn
import torch.optim as optim
# Hyperparameters -- chosen by you, fixed for this run
learning_rate = 0.001
hidden_size = 128
num_epochs = 20
batch_size = 32
# The model's parameters are created based on hyperparameter choices,
# but the parameter VALUES themselves are learned during training
model = nn.Sequential(
nn.Linear(784, hidden_size), nn.ReLU(),
nn.Linear(hidden_size, 10)
)
# The optimizer updates model.parameters() -- and ONLY those -- every step
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
print(sum(p.numel() for p in model.parameters())) # total learnable PARAMETERS
# hidden_size, learning_rate, num_epochs, batch_size are HYPERPARAMETERS -- not counted hereWhy the Distinction Matters in Practice
You cannot use gradient descent to find good hyperparameters directly (this is why they're called "hyper" — they sit one level above the parameters gradient descent optimizes). Finding good hyperparameters is instead its own search problem, covered fully in the Hyperparameter Tuning category — and getting them wrong (e.g. too high a learning rate) can prevent the parameters from ever converging to good values at all, regardless of how much training data or time you have.
Common Mistakes
- Trying to "learn" a hyperparameter like the number of layers via backpropagation — it's not a differentiable, continuous quantity gradient descent can act on directly; that's precisely why separate search techniques (grid search, Bayesian optimization, or specialized methods like Neural Architecture Search) exist.
- Confusing "hyperparameter tuning" with "training" — tuning typically means running multiple full training runs with different hyperparameter settings and comparing validation performance, not a single run.
Interview Relevance
Q: "Is the number of neurons in a hidden layer a parameter or a hyperparameter?" Hyperparameter — it's an architectural choice fixed before training starts. The actual weight values connecting to and from that layer's neurons are the parameters, learned during training. The count of neurons determines the shape of those parameter tensors, but the count itself isn't learned by gradient descent.
Practice Question
Classify each of the following as a parameter or a hyperparameter: (a) dropout rate, (b) a specific weight in the output layer, (c) number of training epochs, (d) a specific bias value in a hidden layer.
