Forward propagation is the process of pushing an input through every layer of a network, in order, to produce a final prediction. It's called "forward" to distinguish it from backpropagation, which runs in the opposite direction.
The General Formula, Layer by Layer
Starting from the input \(\mathbf{a}^{(0)}=\mathbf{x}\), each layer \(l\) computes its pre-activation \(\mathbf{z}^{(l)}\) (a matrix multiplication plus bias) and then its activation \(\mathbf{a}^{(l)}\) (applying the layer's non-linearity). The final layer's activation is the network's prediction, \(\hat{\mathbf{y}} = \mathbf{a}^{(L)}\) for an \(L\)-layer network.
Full Numerical Walkthrough — A Tiny 2-Layer Network
Input \(\mathbf{x}=[1, 2]\). Layer 1: \(\mathbf{W}^{(1)} = \begin{bmatrix}0.1 & 0.2\\0.3 & 0.4\end{bmatrix}\), \(\mathbf{b}^{(1)}=[0.1, 0.1]\), ReLU activation.
Layer 2 (output): \(\mathbf{W}^{(2)}=[0.5, -0.5]\), \(b^{(2)}=0\), sigmoid activation.
The network's final prediction for this input is approximately 0.426.
Code — The Same Computation in PyTorch
import torch
import torch.nn as nn
class TinyNet(nn.Module):
def __init__(self):
super().__init__()
self.layer1 = nn.Linear(2, 2)
self.layer2 = nn.Linear(2, 1)
def forward(self, x):
a1 = torch.relu(self.layer1(x)) # layer 1: weighted sum + ReLU
y_hat = torch.sigmoid(self.layer2(a1)) # layer 2: weighted sum + sigmoid
return y_hat
model = TinyNet()
x = torch.tensor([[1.0, 2.0]])
prediction = model(x) # this single call performs the entire forward pass
print(prediction)Note that model(x) (equivalently, model.forward(x)) is exactly this note's formula, executed automatically — every "forward pass" you'll write for the rest of this hub follows this same layer-by-layer pattern, however many layers or however specialized (convolutional, attention-based) they are.
Common Mistakes
- Applying an activation function to the very last output when the loss function already expects raw logits — e.g. applying sigmoid before
nn.BCEWithLogitsLoss, which applies sigmoid internally for numerical stability. - Forgetting that forward propagation, by itself, does not involve any learning — it only computes a prediction from the network's current weights; learning happens afterward, in the backward pass and weight update.
Interview Relevance
Q: "Walk through what happens during a neural network's forward pass." The input is passed through each layer in sequence: at each layer, the previous layer's output is multiplied by that layer's weight matrix, a bias is added, and a non-linear activation function is applied — the result becomes the input to the next layer. This repeats until the final (output) layer, whose activation is the network's prediction.
Practice Question
For a 2-layer network with \(\mathbf{W}^{(1)}=[[1,0],[0,1]]\), \(\mathbf{b}^{(1)}=[0,0]\), ReLU, then \(\mathbf{W}^{(2)}=[1,1]\), \(b^{(2)}=0\), no output activation — compute the forward pass output for input \(\mathbf{x}=[-1, 3]\).
