Parameters (also called weights) are the numeric values inside a neural network that get adjusted during training — collectively, they're what the model "learned." When you see a model described as "7B" or "70B," that's its parameter count, in billions.
What a Parameter Actually Is
Each transformer block contains large matrices of numbers used to transform token representations (in attention and feed-forward layers). Every individual number in every one of these matrices is one parameter. A model with 7 billion parameters has 7 billion individually-learned numeric values, all adjusted during training to collectively minimize next-token prediction error.
Simplified illustration of one tiny piece:
weight_matrix = [[0.023, -0.451, 0.109, ...],
[-0.302, 0.087, 0.664, ...],
...]
# A real model has billions of these values across many such matricesParameter Count and Model Naming
| Common Naming | Approximate Parameter Count |
|---|---|
| "7B" model | ~7 billion parameters |
| "13B" model | ~13 billion parameters |
| "70B" model | ~70 billion parameters |
Exact parameter counts for many commercial hosted models (like the largest closed models from major providers) are often not publicly disclosed — treat any specific number you see for those as an estimate or rumor unless the provider has officially confirmed it.
Weight Precision and File Size
Each parameter is stored as a number at some numeric precision — commonly 16-bit floating point for a full-precision model, or lower (8-bit, 4-bit) for a quantized version. Lower precision means a smaller file size and less memory/compute needed to run the model, usually at some small cost to output quality. This is why you'll see the same open-weight model offered in multiple sizes on disk despite having the same parameter count.
Practical Use Case
Parameter count is a rough, imperfect proxy for capability and cost — more parameters generally means better performance on complex tasks, but also higher inference cost and latency. See Model Size for the practical tradeoffs this creates when choosing a model.
Common Mistakes
- Assuming parameter count alone determines quality — training data quality, training technique, and architecture choices all matter significantly too; a well-trained smaller model can outperform a poorly-trained larger one on a given task
- Confusing "parameters" with "training data size" — these are different things: parameters are the model's learned internal values; training data is what it learned from
Interview Relevance
Q: "What does it mean when a model is described as having 7 billion parameters?" — the expected answer: 7 billion individual learned numeric values (weights) inside the model's neural network layers, adjusted during training.
Practice Question
Explain why a 4-bit quantized version of a model uses roughly a quarter of the memory of its 16-bit counterpart, in terms of what a "parameter" actually is.