Backward Pass

The backward pass is where backpropagation actually computes gradients — starting from the loss and working toward the input, one layer at a time, using the values cached during the forward pass.

The Core Recursive Formula

Define \(\boldsymbol\delta^{(l)}\) as the gradient of the loss with respect to layer \(l\)'s pre-activation, \(\frac{\partial L}{\partial \mathbf{z}^{(l)}}\) — often called the layer's "error signal." Backpropagation computes it recursively, from the last layer backward:

\(\odot\) denotes element-wise multiplication (see Matrix Multiplication). The transpose \(\big(\mathbf{W}^{(l+1)}\big)^\top\) is exactly what routes the next layer's error signal backward to this layer — the same transpose relationship flagged in Matrix Transpose.

Reading the Recursive Step

To get layer \(l\)'s error signal, take the next layer's error signal \(\boldsymbol\delta^{(l+1)}\), route it backward through that next layer's weights (via the transpose), and then scale it by how sensitive this layer's own activation function was to its input (\(\phi'^{(l)}(\mathbf{z}^{(l)})\), using the cached pre-activation from Forward Pass). This is precisely the recursive, layer-by-layer application of the chain rule from Chain Rule in Backpropagation.

Diagram

Code — Manual Backward Pass for One Layer

import torch

def sigmoid_derivative(z):
    s = torch.sigmoid(z)
    return s * (1 - s)

# Given: delta from the next layer, and this layer's own weight matrix and cached z
def backward_step(delta_next, W_next, z_this):
    return (W_next.T @ delta_next) * sigmoid_derivative(z_this)

delta_next = torch.tensor([0.3])
W_next = torch.tensor([[0.5, -0.2]])
z_this = torch.tensor([0.8, -0.5])
delta_this = backward_step(delta_next, W_next, z_this)
print(delta_this)

Common Mistakes

• Forgetting the element-wise multiplication by \(\phi'(\mathbf{z}^{(l)})\) — the transpose-routed signal alone only accounts for the linear part of the layer; the activation function's own local sensitivity must also be folded in.
• Mixing up which weight matrix belongs in the recursive step — it's always the next layer's weights (\(\mathbf{W}^{(l+1)}\)), routing the next layer's error signal back to this layer, not this layer's own weights.

Interview Relevance

Q: "In the backward pass recursion, why does the next layer's weight matrix need to be transposed?" The forward pass computes \(\mathbf{z}^{(l+1)} = \mathbf{W}^{(l+1)}\mathbf{a}^{(l)}\); by the chain rule, the gradient flowing backward from \(\mathbf{z}^{(l+1)}\) to \(\mathbf{a}^{(l)}\) involves this same weight matrix but in transposed form, since gradients flow in the opposite direction to the forward computation's shape — exactly the general relationship established in Matrix Transpose.

Practice Question

If layer 2's error signal is \(\boldsymbol\delta^{(2)}=[0.4]\), \(\mathbf{W}^{(2)}=[[0.6, -0.1]]\), and layer 1's pre-activation is \(\mathbf{z}^{(1)}=[0.3, -0.2]\) with ReLU activation, compute \(\boldsymbol\delta^{(1)}\). (Recall ReLU's derivative is 1 for positive inputs, 0 for negative.)

Related DL Notes

• Chain Rule in Backpropagation
• Gradient Calculation
• Forward Pass