Sigmoid Function

The sigmoid function squashes any real number into the range (0, 1), making its output directly interpretable as a probability. It was the dominant hidden-layer activation function in early deep learning, before its serious drawbacks (covered below) led to ReLU becoming the default.

Formula

Derivative

A convenient property: the derivative is expressible directly in terms of the function's own output — no need to recompute \(z\) during backpropagation, just reuse \(\sigma(z)\) from the forward pass.

Graph

Numerical Example

The Vanishing Gradient Problem — Sigmoid's Fatal Flaw

Look at the graph: for large \(|z|\) (either very positive or very negative), the curve is nearly flat — the function saturates. Where the curve is flat, \(\sigma'(z)\) is close to 0. Since backpropagation multiplies gradients together layer by layer (via the chain rule), stacking many sigmoid layers means multiplying together many numbers that are each at most 0.25 and often much smaller — the product shrinks exponentially with depth. This is the concrete mechanism behind the vanishing gradient problem, and it's the single biggest reason sigmoid fell out of favor for hidden layers in deep networks.

Not Zero-Centered — A Second Drawback

Sigmoid's output is always positive (between 0 and 1), never negative. This means gradients flowing back through a sigmoid-activated layer tend to all push in the same direction for a given neuron's weights, which can make optimization less efficient (a less severe issue than vanishing gradients, but still a real one that tanh, next, was designed to fix).

Code

import numpy as np
import torch

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

print(sigmoid(np.array([-2, 0, 2])))   # [0.119 0.5 0.881]

z = torch.tensor([-2.0, 0.0, 2.0])
print(torch.sigmoid(z))

Where It's Still Used Today

• Binary classification output layers — where you specifically want a probability-like output in (0,1).
• Gates inside LSTM and GRU cells — where a 0-to-1 "how much to let through" signal is exactly what's needed (covered in the LSTM & GRU category).

It's rarely used for hidden layers in modern feedforward or convolutional networks — that role has been taken over by ReLU and its variants.

Common Mistakes

• Using sigmoid throughout a deep network's hidden layers "because it looks like a natural probability curve" — this reintroduces vanishing gradients, exactly the problem that limited pre-2012 deep networks.
• Forgetting sigmoid's output range makes it wrong for multi-class classification — for that, softmax (covered later in this category) is needed instead.

Interview Relevance

Q: "Why did ReLU replace sigmoid as the default activation for hidden layers in deep networks?" Sigmoid saturates for large positive or negative inputs, making its derivative close to zero in those regions. Backpropagating through many sigmoid layers multiplies many such small gradients together, causing the vanishing gradient problem in deep networks. ReLU's derivative is either 0 or exactly 1, with no saturation on the positive side, which avoids this compounding shrinkage.

Practice Question

Compute \(\sigma(5)\) and \(\sigma'(5)\). What does the small value of \(\sigma'(5)\) tell you about how much gradient signal would pass through this neuron during backpropagation if it were operating in this region?

Related DL Notes

• Tanh Function
• Vanishing Gradient Problem
• Sigmoid vs Tanh