Transformer Encoder

This note zooms into one single Transformer encoder layer — the repeated building block that, stacked \(N\) times, forms the full encoder stack from Transformer Architecture.

The Structure of One Encoder Layer

1. Multi-head self-attention over the input
2. Add the attention output back to the original input (a residual connection), then layer-normalize
3. A position-wise feed-forward network
4. Add the feed-forward output back to its input (another residual connection), then layer-normalize again

This exact pattern — sublayer, then add-and-normalize — repeats twice per encoder layer, and the whole layer repeats \(N\) times (commonly 6 or 12 in published architectures) to form the full encoder stack.

Why Self-Attention Here, Not Cross-Attention

The encoder's job is purely to build a rich representation of the input sequence — every position needs to gather relevant context from every other position within that same input. This is exactly self-attention's job (see Self-Attention), not cross-attention, since there's no separate second sequence involved yet at this stage — cross-attention only enters the picture in the decoder, where it lets the decoder look back at this encoder's finished output.

Diagram — One Encoder Layer

Code

import torch
import torch.nn as nn

encoder_layer = nn.TransformerEncoderLayer(
    d_model=512, nhead=8, dim_feedforward=2048, batch_first=True
)
encoder = nn.TransformerEncoder(encoder_layer, num_layers=6)   # 6 stacked identical layers

x = torch.randn(1, 10, 512)   # 10 tokens, 512-dim embeddings
encoder_output = encoder(x)
print(encoder_output.shape)   # (1, 10, 512) -- same shape as input; every position now carries
                                # rich context gathered from every other position

Why the Output Shape Matches the Input Shape

Every layer in the encoder preserves the input's shape exactly — this is by design, since it allows the same layer structure to be stacked \(N\) times without any dimension mismatches, and it means the final encoder output has one vector per input token, each vector now enriched with context gathered from the entire sequence via self-attention.

Common Mistakes

• Assuming information only flows "forward" through the stack in a strictly hierarchical way, like a CNN's spatial feature hierarchy — each encoder layer's self-attention can relate any two positions directly, at every layer, not just progressively wider receptive fields the way stacked convolutions do.
• Forgetting that all \(N\) encoder layers have independently learned weights — they share the same structure, but each layer learns its own distinct parameters.

Interview Relevance

Q: "What are the two main sublayers inside one Transformer encoder layer, and what role does each play?" Multi-head self-attention, which lets every position gather relevant context from every other position in the input; and a position-wise feed-forward network, which applies an additional non-linear transformation independently to each position. Both sublayers are wrapped with a residual connection and layer normalization.

Practice Question

If an encoder has 6 layers and each attends over the full input sequence, does layer 3 only see information gathered by layer 2, or can it directly relate any two original input positions on its own?

Related DL Notes

• Transformer Architecture
• Transformer Decoder
• Residual Connections