Positional Encoding

Positional encoding solves the exact problem flagged in Self-Attention: raw self-attention is order-agnostic — it has no built-in notion of which token came first, second, or last. Positional encoding injects that missing information directly.

The Sinusoidal Formula

\(pos\) is the token's position in the sequence (0, 1, 2, ...); \(i\) indexes the embedding dimension pairs. Every even dimension uses sine, every odd dimension uses cosine, each at a different frequency determined by \(i\). This produces a unique vector for every position, which is simply added to the token's regular embedding before being fed into the encoder/decoder stack.

Why Sine and Cosine, Specifically

Numerical Example

With \(d_{\text{model}}=4\), position 0: \(PE(0,0)=\sin(0)=0\), \(PE(0,1)=\cos(0)=1\), \(PE(0,2)=\sin(0)=0\), \(PE(0,3)=\cos(0)=1\) — position 0 always produces \([0,1,0,1,\ldots]\) regardless of \(d_{\text{model}}\), since \(\sin(0)=0\) and \(\cos(0)=1\) for every frequency. Position 1 produces different, non-trivial values at each dimension, since the frequencies differ per dimension pair.

Diagram

Code

import torch
import math

def positional_encoding(seq_len, d_model):
    pe = torch.zeros(seq_len, d_model)
    position = torch.arange(seq_len).unsqueeze(1).float()
    div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    return pe

pe = positional_encoding(seq_len=10, d_model=16)
print(pe.shape)   # (10, 16) -- one positional vector per sequence position

# Usage: simply ADD this to the token embeddings before feeding into the Transformer
token_embeddings = torch.randn(10, 16)
input_to_transformer = token_embeddings + pe

Learned Positional Embeddings — A Common Alternative

Some later Transformer-based models (including many modern LLMs) instead use a learned positional embedding — a trainable lookup table, one vector per position, updated via backpropagation like any other embedding — rather than the fixed sinusoidal formula. This trades the sinusoidal approach's extrapolation-friendly mathematical structure for potentially better task-specific fit, at the cost of being limited to whatever maximum sequence length was seen during training.

Common Mistakes

• Forgetting to add positional encoding at all — without it, a Transformer genuinely cannot distinguish "the cat sat on the mat" from "the mat sat on the cat," since self-attention alone has no notion of order.
• Concatenating positional encoding instead of adding it — the standard approach adds the positional vector directly to the token embedding (both must have the same dimension \(d_{\text{model}}\)); some variants do concatenate, but addition is the original and most common design.

Interview Relevance

Q: "Why does the Transformer need positional encoding at all, when RNNs never needed anything like it?" An RNN processes tokens strictly in order, one at a time — position is implicit in the sequential processing itself. Self-attention, by design, treats all positions symmetrically and in parallel, with no inherent notion of order — permuting the input tokens would just permute the corresponding outputs, preserving the same relationships. Positional encoding explicitly injects position information into each token's representation to restore this missing order-awareness.

Practice Question

Why is positional encoding typically added to the token embedding rather than processed as a completely separate input stream?

Related DL Notes

• Self-Attention
• Transformer Architecture
• Positional Embeddings (LLMs)