PyTorch interview questions covering autograd, the training loop mechanics, and common implementation pitfalls — with fully explained answers, including code.
Q1. What is autograd, and how does PyTorch build the computational graph?
Autograd is PyTorch's automatic differentiation engine. As operations are performed on tensors with requires_grad=True, PyTorch dynamically builds a computational graph tracking every operation performed — this is a "define-by-run" approach, meaning the graph is built fresh on each forward pass, rather than defined statically upfront. When .backward() is called on a scalar output (typically the loss), autograd traverses this graph backward, applying the chain rule at each node to compute gradients for every tensor with requires_grad=True.
Q2. What's the difference between .backward() and optimizer.step()?
loss.backward() # computes gradients and stores them in each parameter's .grad attribute
optimizer.step() # uses those stored .grad values to actually UPDATE the weights.backward() only computes and stores gradients — it doesn't change any weights. optimizer.step() is a separate call that reads the gradients already stored in .grad and applies the actual weight update according to the chosen optimization algorithm (SGD, Adam, etc.). Both steps are required; forgetting either one means training silently does nothing useful.
Q3. Why do we call optimizer.zero_grad() before each backward pass?
PyTorch accumulates (adds to) gradients in .grad by default rather than overwriting them — this is intentional, since it supports use cases like gradient accumulation across multiple mini-batches. But for the standard training loop, this means gradients from the previous step would incorrectly add onto the current step's gradients if not cleared first. optimizer.zero_grad() resets all gradients to zero before each new backward pass, ensuring each step's gradient reflects only that step's batch.
Q4. What's the difference between model.eval() and torch.no_grad()?
model.eval() changes the behavior of specific layers — Dropout stops randomly zeroing activations, and BatchNorm uses its stored running statistics instead of computing batch statistics — but gradients are still tracked unless separately disabled. torch.no_grad() is a context manager that disables gradient tracking entirely for anything inside it, saving memory and computation during inference, but doesn't change any layer's behavior. For inference, both are typically used together: model.eval() for correct layer behavior, torch.no_grad() for efficiency.
Q5. Explain the purpose of the DataLoader and Dataset classes.
class MyDataset(torch.utils.data.Dataset):
def __len__(self): return len(self.data)
def __getitem__(self, idx): return self.data[idx], self.labels[idx]
loader = torch.utils.data.DataLoader(MyDataset(), batch_size=32, shuffle=True, num_workers=4)Dataset defines how to access individual examples (via __getitem__) and how many exist (via __len__) — it abstracts away the data source's specifics. DataLoader wraps a Dataset and handles batching, shuffling, and parallel data loading (via num_workers) automatically, so the training loop can simply iterate over ready-made batches rather than manually managing indexing and batching logic.
Q6. What's the difference between a tensor's .detach() and setting requires_grad_(False)?
.detach() creates a new tensor that shares the same underlying data but is disconnected from the computational graph — the original tensor is unaffected, and further operations on the detached copy won't be tracked. Calling .requires_grad_(False) modifies the tensor in place, directly changing whether gradients are tracked for that specific tensor going forward, without creating a new tensor. .detach() is more common when you want a "snapshot" value without affecting the original tensor's graph tracking.
Q7. How would you correctly move a model and data to GPU?
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
for x_batch, y_batch in train_loader:
x_batch, y_batch = x_batch.to(device), y_batch.to(device) # move EVERY batch too
output = model(x_batch)Both the model's parameters and every batch of data must be moved to the same device — a very common bug is moving the model to GPU once but forgetting that each new batch from the DataLoader still starts on CPU and needs to be moved explicitly inside the loop, which produces a "tensors on different devices" runtime error.
Q8. What's the difference between using nn.Module versus a plain function to define a model?
nn.Module automatically tracks all sub-layers and their parameters (via .parameters()), integrates with PyTorch's serialization (state_dict), and provides hooks for behavior like switching between train/eval mode consistently across the whole model. A plain function can technically compute a forward pass but has none of this built-in bookkeeping — you'd need to manually track parameters, handle saving/loading, and manage mode switching yourself, which is why nn.Module is the standard way to define any real model in PyTorch.
Q9. Why might you get a "CUDA out of memory" error, and what are common fixes?
This happens when the total memory required (model parameters, gradients, optimizer state, and activations for the current batch) exceeds the GPU's available memory. Common fixes: reduce batch size, use mixed precision training (roughly halves activation/gradient memory), use gradient checkpointing (trades compute for memory by recomputing activations during backward instead of storing them), or use gradient accumulation to simulate a larger effective batch size without the memory cost of one large batch — all covered in depth in the Memory Optimization note.
