The step function is the original activation function — the one Rosenblatt's Perceptron used in 1958. It's rarely used in modern networks, but understanding exactly why it was abandoned explains why every activation function since has been designed around one specific requirement: being differentiable.
Formula
| Property | Value |
|---|---|
| Range | \(\{0, 1\}\) — only two possible outputs |
| Derivative | 0 everywhere except at \(z=0\), where it's undefined (a discontinuous jump) |
| Shape | A flat line at 0, jumping instantly to a flat line at 1 |
Graph
A hard jump from 0 to 1 at z=0 — flat everywhere else, meaning zero gradient almost everywhere.
The Fatal Flaw: Zero Gradient
Backpropagation (see the Backpropagation category) needs \(\frac{\partial \phi}{\partial z}\) at every point to propagate error signals backward through a network. The step function's derivative is 0 almost everywhere — meaning gradient descent gets no information about which direction to adjust weights, since a tiny nudge to \(z\) almost never changes the output at all. This single property makes the step function unusable for training multi-layer networks with gradient-based methods, even though it works fine for the historical single-layer Perceptron (whose learning rule, from Perceptron Learning Algorithm, doesn't rely on calculus-based gradients at all).
Code
import numpy as np
def step(z):
return np.where(z >= 0, 1, 0)
z = np.array([-2, -0.1, 0, 0.1, 2])
print(step(z)) # [0 0 1 1 1]Where It's Used Today
Almost nowhere in modern deep learning — its role is now purely historical and educational, marking the starting point that every subsequent activation function (starting with sigmoid, next) improved on by becoming smooth and differentiable.
Common Mistakes
- Assuming the step function is simply "an older, simpler ReLU" — the two behave completely differently for gradient-based training; ReLU has a well-defined, useful gradient for positive inputs, while the step function's gradient is uselessly zero almost everywhere.
Interview Relevance
Q: "Why can't the step function be used as an activation function in a modern, gradient-trained neural network?" Its derivative is zero everywhere except at a single discontinuous point, so backpropagation receives no usable gradient signal — weight updates would essentially never happen. This is exactly why every practical activation function since has been designed to be smooth (or at least piecewise-differentiable with non-zero gradient somewhere useful).
Practice Question
Why did the original Perceptron Learning Algorithm not need the step function to be differentiable, while a modern multi-layer network trained with backpropagation does?
