Pretraining is the massive, foundational training stage where an LLM learns language, facts, and reasoning patterns purely from next-token prediction over enormous amounts of raw, unlabeled text — the single most compute-intensive stage of an LLM's entire lifecycle.
The Setup
| Aspect | Typical Scale |
|---|---|
| Training data | Trillions of tokens — web pages, books, code, articles |
| Labels | None required — the "label" at every position is simply the actual next token in the raw text |
| Objective | Minimize next-token prediction cross-entropy loss (see Next-Token Prediction) |
| Compute | Often thousands of GPUs/TPUs running for weeks to months |
Why This Stage Alone Produces a Genuinely Capable Model
To get good at predicting the next word across a truly enormous, diverse corpus of human-written text, a model has to implicitly learn an enormous amount: grammar, facts about the world, reasoning patterns, coding conventions, and more — none of this is explicitly labeled or taught; it all emerges as a side effect of getting good at the single objective of predicting what word comes next, applied at massive scale. This mirrors the "the embedding is a byproduct" insight from Word2Vec, just at vastly greater scale and with much richer emergent behavior.
Scaling Laws — A Brief Note
Research (notably the "Chinchilla" scaling laws) found that model quality depends predictably on the balance between model size (parameters) and training data quantity (tokens) — for a fixed compute budget, there's a specific ratio of parameters to training tokens that tends to produce the best-performing model, and many earlier large models were found to be significantly undertrained relative to their parameter count (echoing the exact same lesson from RoBERTa, just at a much larger scale).
Code — The Core Training Loop, Conceptually
import torch
import torch.nn.functional as F
def pretraining_step(model, token_batch):
# token_batch: (batch_size, seq_len) -- raw tokenized text, no labels needed
inputs = token_batch[:, :-1] # every token except the last
targets = token_batch[:, 1:] # every token except the first -- i.e., "the next token"
logits = model(inputs) # (batch, seq_len-1, vocab_size)
loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)), targets.reshape(-1))
return loss
# The "labels" here are literally just the input sequence, shifted by one position --
# no human annotation required at any pointCommon Mistakes
- Assuming pretraining alone produces a helpful, instruction-following assistant — pretraining produces a raw text-continuation model; the additional stages covered in the rest of this category (SFT, alignment) are what shape it into something more directly useful and safe to interact with.
- Underestimating how much data quality (not just quantity) matters — training on low-quality, repetitive, or heavily duplicated web text can meaningfully hurt a model's final quality, which is why substantial effort typically goes into filtering and curating pretraining data.
Interview Relevance
Q: "How can an LLM learn facts, reasoning, and coding ability from a training objective as simple as 'predict the next word'?" To become genuinely good at predicting the next word across an enormous, diverse corpus of human-written text, a model has to implicitly capture a huge amount of underlying structure — grammar, world knowledge, reasoning patterns — since accurately predicting continuations in technical, factual, or logical text requires modeling that structure, even though none of it is explicitly labeled as a separate training signal.
Practice Question
Why is next-token prediction such a convenient pretraining objective specifically for scaling to enormous, diverse, unlabeled datasets?
