This note applies the chain rule from Chain Rule specifically to the structure of a multi-layer neural network — showing exactly how a gradient signal travels backward through layer after layer to reach a weight buried deep inside the network.
A Network as a Chain of Functions
An \(L\)-layer network is literally a composition of functions: \(\hat y = f^{(L)}(f^{(L-1)}(\cdots f^{(1)}(\mathbf{x})\cdots))\). To find how the loss \(L\) depends on a weight \(w\) buried in layer \(l\), the chain rule must pass through every layer between \(l\) and the output:
Each factor is a "local" derivative — how one layer's output changes with respect to its own input or its own weights, which is exactly the Jacobian introduced in Jacobian.
The Key Insight: Local Gradients, Computed Once, Reused
Notice that the product for layer \(l-1\)'s weights shares every term except the very last factor with layer \(l\)'s product. Backpropagation exploits this directly: it computes the "gradient flowing into" each layer's output once, moving backward from the loss, and every layer only ever needs to know two things — its own local derivative with respect to its inputs, and its own local derivative with respect to its weights. Neither layer needs any awareness of what's happening in any other layer.
Diagram — The Chain, Layer by Layer
Each layer only ever computes its own local derivative; the chain rule stitches these local pieces into the full gradient for any weight, however deep in the network.
Code
import torch
x = torch.tensor(1.0, requires_grad=True)
a1 = x * 2 # layer 1 (toy: multiply by 2)
a2 = a1 ** 2 # layer 2 (toy: square)
loss = (a2 - 3) ** 2 # loss
loss.backward()
print(x.grad) # PyTorch applied the exact chain-rule product across all three "layers" automaticallyCommon Mistakes
- Trying to derive a deep network's full gradient by writing out the entire chain-rule product by hand for every weight — this is exactly the tedious, error-prone, and unnecessary work backpropagation's local-gradient reuse eliminates; frameworks handle it automatically via the computational graph.
- Forgetting that a variable feeding into multiple downstream paths (common in architectures with skip connections) must have its gradient contributions from every path summed, not just taken from one — this was flagged already in Computational Graphs.
Interview Relevance
Q: "Why does each layer in backpropagation only need to know its own local derivative, not the whole network's structure?" Because the chain rule's product structure means the gradient for any weight decomposes into a chain of per-layer local derivatives. Backpropagation computes the "gradient flowing backward" once per layer and passes it to the previous layer, so each layer only ever needs to combine that incoming signal with its own local derivative — no layer needs global knowledge of the network.
Practice Question
For a 5-layer network, how many "local derivative" factors would the chain rule need to multiply together to compute the gradient for a weight in layer 1?
