Recall from Fine-Tuning that updating every parameter of a pretrained model can achieve the best possible task performance. At LLM scale, this note shows precisely why full fine-tuning becomes prohibitively expensive — the exact problem the rest of this category solves.
The Real Memory Cost, Precisely
| Component | Memory per Parameter |
|---|---|
| The parameter itself (FP16) | 2 bytes |
| Its gradient | 2 bytes |
| Adam's first moment \(m_t\) | 4 bytes (typically kept in FP32 for stability) |
| Adam's second moment \(v_t\) | 4 bytes |
| Total | ~12 bytes per parameter |
For a 7-billion-parameter model, full fine-tuning with Adam (see Adam Optimizer) requires roughly \(7\text{B} \times 12\text{ bytes} \approx 84\text{ GB}\) just for training state — before even accounting for activations, far exceeding what a single consumer or even many professional GPUs can hold.
Why Every Parameter Needs This Overhead
Full fine-tuning treats every single weight as trainable, so autograd must track gradients for all of them, and Adam must maintain its full moving-average state (\(m_t\), \(v_t\) from Adam Optimizer) for all of them too. None of this overhead is optional under full fine-tuning — it's a direct consequence of allowing every parameter to update.
Code — Seeing the Scale Directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
total_params = sum(p.numel() for p in model.parameters())
print(f"{total_params:,} parameters") # ~7,000,000,000
approx_gb_for_full_finetuning = total_params * 12 / (1024**3)
print(f"~{approx_gb_for_full_finetuning:.1f} GB just for training state") # ~78 GBThe Direct Motivation for the Rest of This Category
If only a small fraction of a model's parameters actually need to change to adapt it to a new task, the natural fix is training only those — leaving the vast majority frozen, with no gradient or optimizer state overhead at all. This is precisely the idea behind PEFT, the next note, and every specific technique covered for the rest of this category.
Common Mistakes
- Assuming full fine-tuning is always the "gold standard" worth pursuing regardless of cost — for many tasks, parameter-efficient methods achieve performance close enough to full fine-tuning that the massive resource savings make them the clearly better practical choice.
- Forgetting that inference-time memory (just the model weights) is far smaller than training-time memory — the 12-bytes-per-parameter figure applies specifically to training, not to running a fine-tuned model afterward.
Interview Relevance
Q: "Why does fine-tuning a 7B-parameter model require roughly 12x more memory per parameter than just storing the model itself?" Training requires not just the parameters (2 bytes each in FP16) but also their gradients (2 bytes) and, with Adam, two additional moving-average state tensors per parameter (4 bytes each) — totaling roughly 12 bytes of training state per parameter, dramatically more than the 2 bytes needed to simply store and run the model for inference.
Practice Question
Roughly how much training-state memory would full fine-tuning with Adam require for a 1.3-billion-parameter model?
