Transformer interview questions covering self-attention, multi-head attention, positional encoding, and the overall architecture — with fully explained answers.
Q1. Why did Transformers largely replace RNNs for sequence modeling?
RNNs process sequences step by step, which is inherently sequential and can't be parallelized across time steps during training — this makes training slow, and RNNs also struggle with very long-range dependencies due to the vanishing gradient problem. Transformers replace recurrence with self-attention, which lets every position attend directly to every other position in a single step — this is fully parallelizable across the sequence during training (dramatically faster on modern hardware) and gives every pair of positions a direct connection regardless of distance, avoiding the long-range dependency problem RNNs face.
Q2. Explain self-attention using the Query/Key/Value analogy.
Each token produces three vectors: a Query (what this token is "looking for"), a Key (what this token "offers" to others looking), and a Value (the actual content to be passed along if attended to). Attention scores are computed by comparing each Query against every Key (via dot product), producing a relevance score for every pair of positions; these scores are turned into weights (via softmax), and the output for each position is a weighted sum of all Values, weighted by how relevant each other position's Key was to this position's Query.
Q3. Why do we scale attention scores by \(\sqrt{d_k}\)?
\[ \text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V \]where \(d_k\) is the dimension of the key vectors. Without scaling, dot products between Q and K vectors can grow large in magnitude as \(d_k\) increases, pushing the softmax into regions with extremely small gradients (since softmax saturates for large input magnitudes) — this would slow or stall learning. Dividing by \(\sqrt{d_k}\) keeps the dot products in a numerically well-behaved range regardless of dimension, keeping gradients healthy.
Q4. What is the purpose of multi-head attention instead of a single attention head?
A single attention head computes one specific type of relevance/relationship between tokens. Multi-head attention runs several attention computations in parallel, each with its own learned projections, allowing the model to jointly attend to information from different representation subspaces — one head might learn to track syntactic relationships, another might track coreference, another something else entirely. The outputs from all heads are concatenated and linearly projected, giving the model a richer, more expressive combined representation than any single head could provide alone.
Q5. Why do Transformers need positional encoding?
Self-attention itself is permutation-invariant — it has no inherent notion of token order, since it computes relevance based purely on content (Q/K/V), not position. Without any positional signal, "the cat sat on the mat" and "mat the on sat cat the" would produce identical self-attention computations. Positional encoding injects information about each token's position into its embedding before it enters the attention layers, giving the model the ability to use order information.
Q6. What's the difference between the encoder and decoder stacks in the original Transformer?
The encoder processes the full input sequence with unrestricted (bidirectional) self-attention — every position can attend to every other position, since the full input is available at once. The decoder generates the output sequence autoregressively, using masked self-attention (so each position can only attend to earlier positions, not future ones it hasn't generated yet) plus a cross-attention layer that attends to the encoder's output, letting the decoder incorporate information from the full input sequence while generating each output token.
Q7. What is masked self-attention, and why is it needed in the decoder?
Masked self-attention prevents each position from attending to positions after it in the sequence, by setting those attention scores to \(-\infty\) before the softmax (so they become exactly zero after). This is necessary in the decoder because generation is autoregressive — at inference time, the model genuinely doesn't have access to tokens it hasn't generated yet, so training must enforce this same constraint to avoid the model learning to "cheat" by looking at future tokens it won't actually have access to at inference time.
Q8. Why are residual connections and layer normalization important in Transformers?
Transformers are typically very deep (many stacked encoder/decoder blocks), and without residual connections, gradients would struggle to propagate cleanly through so many layers, echoing the same optimization difficulty ResNet's skip connections address in CNNs. Layer normalization stabilizes the scale of activations at each layer, keeping training numerically stable and allowing effective use of higher learning rates — together, these two components are essential to making very deep Transformer stacks trainable at all.
Q9. What is the computational complexity of self-attention with respect to sequence length, and why does it matter?
Self-attention computes a relevance score between every pair of tokens, giving it \(O(n^2)\) computational and memory cost in sequence length \(n\). This matters because doubling the input length quadruples the compute and memory needed for attention — this quadratic scaling is the core technical challenge addressed by long-context model techniques like sparse attention and FlashAttention, covered in the Modern AI category.
