PReLU (Parametric ReLU) takes Leaky ReLU's idea one step further: instead of fixing the negative slope \(\alpha\) as a hyperparameter you choose, PReLU makes \(\alpha\) a learnable parameter — the network figures out the best slope for itself during training.
Formula
Structurally identical to Leaky ReLU — the only difference is that \(\alpha\) is now updated by gradient descent, just like any other weight, rather than fixed in advance.
How \(\alpha\) Gets Learned
Backpropagation computes this gradient just like any other parameter's gradient, and \(\alpha\) is updated via the same gradient descent rule from Gradient Descent. Each channel (in a CNN) or each layer can even have its own separate, independently-learned \(\alpha\), giving the network fine-grained control over how it handles negative activations in different parts of the architecture.
Code
import torch.nn as nn
import torch
layer = nn.PReLU(num_parameters=1, init=0.25) # init is the starting value; it will be updated during training
x = torch.tensor([-2.0, 0.0, 2.0])
print(layer(x))
print(list(layer.parameters())) # alpha is now a genuine, trainable nn.ParameterTradeoffs vs Leaky ReLU
| Leaky ReLU | PReLU | |
|---|---|---|
| \(\alpha\) | Fixed hyperparameter (e.g. 0.01) | Learned parameter, adapted to the data |
| Extra parameters | None | One (or more) additional learnable parameter(s) per layer/channel |
| Overfitting risk | Lower (fewer parameters) | Slightly higher, especially on small datasets |
| Flexibility | Same slope everywhere | Can adapt differently per channel/layer |
Common Mistakes
- Using PReLU by default on small datasets — the extra learnable parameters add a small overfitting risk that isn't always worth it compared to simpler alternatives (ReLU, Leaky ReLU) when data is limited.
- Assuming PReLU is strictly "more advanced, therefore better" — it's a genuine tradeoff (flexibility vs. extra parameters/overfitting risk), and empirical testing on your specific task is the only way to know which wins.
Interview Relevance
Q: "What's the difference between Leaky ReLU and PReLU?" They share the exact same formula, but Leaky ReLU's negative-slope parameter \(\alpha\) is a fixed hyperparameter chosen before training, while PReLU treats \(\alpha\) as a learnable parameter, updated via backpropagation and gradient descent alongside the network's weights — letting the model discover a task-appropriate slope automatically.
Practice Question
If a PReLU neuron's learned \(\alpha\) converges to a value very close to 0, what does the neuron's behavior become nearly equivalent to?
