Building on Positional Encoding, this note covers the two most common alternatives used in modern LLMs: learned positional embeddings, and the now-dominant Rotary Positional Embeddings (RoPE).
Learned Positional Embeddings
Instead of the fixed sinusoidal formula, some models simply use a trainable lookup table — one learnable vector per position, updated via backpropagation exactly like a token embedding. Simple and effective, but fundamentally limited to whatever maximum sequence length was seen during training — a position never encountered in training has no learned embedding to fall back on.
RoPE — Rotary Positional Embeddings
RoPE takes a fundamentally different approach: instead of adding a positional vector to the token embedding, it rotates the query and key vectors (from Query, Key, Value) by an angle proportional to their position, directly within the attention computation itself.
\(R_{\Theta,m}\) is a rotation matrix parameterized by position \(m\). The key mathematical property this achieves: the dot product between a rotated query at position \(m\) and a rotated key at position \(n\) ends up depending only on their relative distance \(m-n\), not their absolute positions — giving the model a naturally relative sense of position "baked into" every attention score computation.
Why RoPE Became the Dominant Choice
| Property | Sinusoidal (Original) | Learned | RoPE |
|---|---|---|---|
| Added where | Once, at input embeddings | Once, at input embeddings | Directly inside every attention computation |
| Relative position awareness | Indirect | Indirect | Direct — built into the dot product mathematically |
| Extrapolation to unseen lengths | Some, in principle | Poor — no representation for unseen positions | Better empirically, a major practical advantage |
Code — Conceptual RoPE Application
import torch
def apply_rope(q, position, theta_base=10000):
d = q.shape[-1]
freqs = 1.0 / (theta_base ** (torch.arange(0, d, 2).float() / d))
angles = position * freqs
cos, sin = torch.cos(angles), torch.sin(angles)
q1, q2 = q[..., 0::2], q[..., 1::2]
rotated = torch.stack([q1 * cos - q2 * sin, q1 * sin + q2 * cos], dim=-1)
return rotated.flatten(-2)
q = torch.randn(8) # a single query vector, 8-dim
rotated_q = apply_rope(q, position=5)
print(rotated_q.shape) # (8,) -- same shape, rotated by an angle tied to position 5Common Mistakes
- Assuming RoPE adds a separate positional vector like sinusoidal encoding does — it works entirely differently, by rotating the existing query/key vectors, with no separate positional embedding tensor added anywhere.
- Assuming RoPE fully solves context-length extrapolation — it empirically helps significantly more than learned embeddings, but models still generally perform best within (or near) their originally trained context length, and specialized extension techniques exist for pushing well beyond it.
Interview Relevance
Q: "Why do most modern LLMs use RoPE instead of the original sinusoidal positional encoding?" RoPE encodes position directly into the attention mechanism by rotating query and key vectors, such that the resulting attention scores depend mathematically only on relative position between tokens, not absolute position. This gives models a more natural, direct sense of relative distance and has been found empirically to generalize better to sequence lengths beyond what was seen during training, compared to both the original sinusoidal and simple learned positional embedding approaches.
Practice Question
Why does a learned positional embedding table fundamentally struggle with sequences longer than any seen during training, in a way that both sinusoidal encoding and RoPE are better equipped to handle?
