Model Training

This note is the project-lifecycle checkpoint tying together the Training Deep Networks and PyTorch categories — the actual "run the training loop" stage, viewed from a practical project-management angle rather than pure mechanics.

The Training Stage Checklist

1. Establish a working, minimal training loop first — verify it runs end to end (even for just a few steps) before investing in a full training run.
2. Overfit a tiny subset of data intentionally (e.g. 10–20 examples) — a well-implemented model should be able to memorize this trivially; failure to do so signals a genuine implementation bug, not a modeling problem, before you waste a full run debugging the wrong thing.
3. Run the full training loop with checkpointing (Checkpointing) and early stopping (Early Stopping) enabled from the start.
4. Track both training and validation metrics throughout, watching for the overfitting/underfitting signatures from Overfitting and Underfitting.

The "Overfit a Tiny Batch" Sanity Check

# A genuinely valuable debugging technique before a full training run
tiny_batch = next(iter(train_loader))   # just one batch
x_tiny, y_tiny = tiny_batch

for step in range(200):   # train ONLY on this tiny batch, many steps
    optimizer.zero_grad()
    loss = loss_fn(model(x_tiny), y_tiny)
    loss.backward()
    optimizer.step()
    if step % 20 == 0:
        print(f"step {step}: loss={loss.item():.4f}")

# If loss doesn't approach ~0, something is likely broken in the model,
# loss function, or data pipeline -- BEFORE spending time on a full run

This technique catches a surprisingly large fraction of implementation bugs (a mismatched loss function, an architecture that can't actually learn the task, a data pipeline producing incorrect labels) cheaply and quickly, well before committing to a full, expensive training run.

Common Mistakes

• Launching a full, expensive training run without first verifying the pipeline works correctly on a small scale — debugging a broken pipeline after a multi-hour training run has already completed wastes far more time than catching it upfront.
• Not enabling checkpointing from the very start of a long training run — a crash partway through, without checkpoints, means losing all progress and starting over from scratch.

Interview Relevance

Q: "Why is 'overfitting a tiny batch of data' a valuable debugging step before launching a full training run?" A correctly implemented model, loss function, and data pipeline should be able to trivially memorize a very small number of examples (e.g. 10–20) — if the loss doesn't drop close to zero on this tiny batch after many training steps, something is genuinely broken in the implementation (wrong loss function, incorrect labels, a bug in the model architecture), not a modeling or hyperparameter issue. Catching this cheaply, on a tiny batch, avoids wasting a full, expensive training run debugging the wrong problem.

Practice Question

You run the "overfit a tiny batch" sanity check and the loss stays flat, never decreasing at all, even after 500 steps. What would you investigate first?

Related DL Notes

• Training Loop
• Checkpointing
• Model Evaluation