Pretraining is the first, most expensive stage of building an LLM — self-supervised learning on massive amounts of raw text, with no human-labeled examples required.
Why "Self-Supervised"
The training signal comes from the text itself — no human needs to label anything. Given "The cat sat on the ___", the correct answer ("mat") is right there in the original text. This is what makes it possible to train on trillions of tokens: the labels are free, generated automatically from existing text.
What Goes Into the Training Data
- Web pages (crawled and filtered for quality)
- Books and articles
- Source code repositories
- Reference material (encyclopedic content, documentation)
Data quality and filtering matter enormously — training on unfiltered, low-quality, or duplicated text measurably degrades the resulting model's capability, which is why data curation is a significant part of the pretraining effort, not an afterthought.
What Pretraining Produces: A "Base Model"
The output of pretraining is fluent and broadly knowledgeable, but not naturally conversational or instruction-following — see LLM Training for why the next stages are still necessary.
The Knowledge Cutoff
Whatever the pretraining data included up to a certain date becomes the model's effective "knowledge" — anything after that date, the model simply has no information about, unless it's given that information at inference time (see RAG). This is why models can confidently discuss events up to their cutoff and know nothing about anything after it.
Practical Use Case
Understanding pretraining explains a common support question: "why doesn't the model know about [recent event]?" — it simply wasn't in the training data, and no amount of clever prompting can retrieve information the model was never trained on.
Common Mistakes
- Assuming a model can be updated with new knowledge just by prompting it harder — new factual knowledge requires either retraining/fine-tuning or providing it as context (RAG), not clever prompting
- Underestimating how much data curation/filtering affects the final model — "more data" isn't automatically better if it's low quality
Interview Relevance
Q: "Why is pretraining called 'self-supervised'?" — the answer should center on the training labels coming automatically from the text itself (predicting the next token), requiring no human annotation.
Practice Question
Explain why a model pretrained only on English text would perform poorly translating into a language with almost no representation in its training data.