RNN and LSTM interview questions covering sequential modeling, the vanishing gradient problem, gating mechanisms, and BPTT — with fully explained answers.
Q1. Why do we need RNNs instead of a plain feedforward network for sequence data?
A feedforward network has no notion of order or memory between inputs — each input is processed entirely independently. Sequential data (text, time series, audio) has meaning that depends on order and context from previous elements. RNNs maintain a hidden state that's updated at each time step and carried forward, giving the network a form of memory across the sequence — the same weights are reused (shared) at every time step, which also makes RNNs able to handle variable-length sequences naturally, unlike a fixed-input-size feedforward network.
Q2. Explain the vanishing gradient problem specific to RNNs.
During backpropagation through time, gradients are propagated backward through every time step, involving repeated multiplication by the same weight matrix (and activation derivatives) at each step. If these repeated factors are consistently less than 1 in magnitude, the gradient shrinks exponentially as it's propagated back across many time steps — meaning the network essentially can't learn dependencies that span long distances in the sequence, since the gradient signal from a distant time step has vanished by the time it reaches earlier weights.
Q3. How does LSTM solve the vanishing gradient problem RNNs have?
LSTM introduces a separate cell state that flows through time with only minor, gated linear modifications (not repeated matrix multiplication and squashing through an activation like the RNN hidden state). This near-additive update path lets gradients flow backward through many time steps largely unimpeded, rather than shrinking exponentially — the gating mechanism (forget, input, output gates) explicitly controls what information is kept, added, or output at each step, giving the network a learnable way to preserve important information over long sequences.
Q4. Walk through what each of the three LSTM gates does.
The forget gate decides what fraction of the existing cell state to keep versus discard, based on the current input and previous hidden state. The input gate decides what new information from the current input should be added to the cell state. The output gate decides what part of the (updated) cell state should be exposed as the current hidden state/output. Together, these three gates give the LSTM fine-grained, learned control over what to remember, what to forget, and what to expose at each time step.
Q5. What's the difference between GRU and LSTM? When would you choose one over the other?
GRU simplifies LSTM's architecture — it merges the cell state and hidden state into one, and uses only two gates (update and reset) instead of three, giving it fewer parameters. In practice, the two often perform comparably on many tasks, so GRU is sometimes preferred when compute/memory efficiency matters and the simpler architecture is sufficient, while LSTM's extra gating capacity can be advantageous for more complex sequence modeling tasks — but this isn't a universal rule, and empirical comparison on the specific task is the most reliable way to decide.
Q6. What is Backpropagation Through Time (BPTT)?
BPTT is backpropagation applied to a "unrolled" RNN — the recurrent network is conceptually unrolled into a chain of copies (one per time step) sharing the same weights, and gradients are computed by applying the chain rule backward through this entire unrolled chain, back to the first time step. Because the same weights are shared across all time steps, the gradient with respect to each weight is the sum of its contributions from every time step in the unrolled chain.
Q7. What's the difference between many-to-one and many-to-many RNN architectures? Give examples.
Many-to-one takes a full sequence as input and produces a single output at the end — e.g. sentiment classification of a sentence (many words in, one label out). Many-to-many produces an output at every time step (e.g. part-of-speech tagging, where each word gets its own tag) or, in the encoder-decoder variant, produces a full output sequence after consuming the full input sequence (e.g. machine translation).
Q8. Why do bidirectional RNNs exist, and when are they NOT appropriate?
A standard RNN only has access to past context when producing an output at a given time step. A bidirectional RNN runs two RNNs — one forward, one backward over the sequence — and combines both, giving each time step's output access to both past and future context, which helps for tasks like named entity recognition where surrounding context (including what comes later) is informative. Bidirectional RNNs are NOT appropriate for real-time, streaming, or autoregressive generation tasks, where future context genuinely isn't available yet at inference time (e.g. generating text one token at a time).
Q9. Explain teacher forcing — what problem does it solve, and what downside does it introduce?
During sequence-to-sequence training, teacher forcing feeds the ground-truth previous token as input to the decoder at each step (rather than the model's own, possibly incorrect, previous prediction) — this significantly stabilizes and speeds up training, since early in training the model's own predictions would be mostly wrong and would compound errors across the sequence. The downside is exposure bias: at inference time, the model must rely on its own (sometimes imperfect) predictions as input, a distribution it never actually trained on, which can cause errors to compound in ways training never exposed the model to.
