Computational Graphs (PyTorch)

import torch

def dynamic_forward(x, use_extra_layer):
    y = x * 2
    if use_extra_layer:   # a genuine Python conditional -- the graph SHAPE depends on this
        y = y + 1
    return y

x = torch.tensor(3.0, requires_grad=True)
result = dynamic_forward(x, use_extra_layer=True)   # the graph includes the +1 step
result.backward()
print(x.grad)

Because the graph is built dynamically, ordinary Python control flow (if-statements, loops with variable length) works naturally — the graph simply reflects whatever operations actually executed on this specific forward pass, which is exactly what makes variable-length sequence processing (RNNs) and conditional architectures straightforward to implement.

Inspecting the Graph

x = torch.tensor(2.0, requires_grad=True)
y = x ** 2
z = y * 3

print(z.grad_fn)          # <MulBackward0> -- the operation that produced z
print(z.grad_fn.next_functions)   # references to the operations that produced z's inputs

grad_fn is literally a pointer into the recorded graph — following it backward traces exactly the same computation history the backward pass will walk through.

retain_graph — When You Need the Graph Twice

y = x ** 2
y.backward(retain_graph=True)   # normally, the graph is FREED after backward() to save memory
y.backward()                     # without retain_graph=True above, this second call would ERROR

By default, PyTorch frees the computational graph immediately after .backward() to save memory — calling .backward() a second time on the same graph without retain_graph=True raises an error, since the graph it needs no longer exists.

Common Mistakes

• Calling .backward() twice on the same computation without retain_graph=True and being surprised by the resulting error — this is expected behavior, since the graph is freed by default after the first call.
• Assuming the graph persists across training iterations — a fresh graph is built on every single forward pass; nothing carries over automatically between iterations unless you explicitly retain it.

Interview Relevance

Q: "Why does PyTorch's dynamic graph construction make it well-suited to models with variable control flow, like RNNs processing variable-length sequences?" Because the graph is built fresh on every forward pass, reflecting whatever operations actually executed, ordinary Python control flow (loops, conditionals) integrates naturally — a loop processing a variable number of sequence steps simply produces a graph with that many corresponding operations, with no need to predefine a fixed computational structure ahead of time.

Practice Question

Why does calling .backward() a second time on the same output, without retain_graph=True, raise an error by default?

Related DL Notes

• Computational Graphs
• Autograd
• nn.Module