A model's vocabulary is the complete, fixed set of tokens it knows how to represent — every token id from 0 to vocab_size-1 maps to exactly one entry in this set, and this size choice ripples through the entire model's design.
The Vocabulary-Size Tradeoff
| Vocabulary Size | Effect |
|---|---|
| Small (e.g. a few thousand) | Small embedding/output layers (cheaper), but longer sequences (since more subword pieces are needed per word) |
| Large (e.g. 50,000+) | Shorter sequences (more whole words get their own token), but larger embedding table and output projection layer — more parameters and compute in exactly those two layers |
Modern LLMs commonly use vocabularies in the tens of thousands to over 100,000 tokens, balancing these two costs empirically.
How Vocabulary Size Directly Sizes Two Layers
Both the input embedding table (converting a token id into a vector) and the final output layer (converting a hidden vector into a probability distribution over every possible next token, as in Softmax Function) scale directly with vocabulary size — for a large vocabulary and modest \(d_{\text{model}}\), these two layers alone can represent a significant fraction of a model's total parameters.
Special Tokens
Beyond ordinary word/subword pieces, most vocabularies reserve a handful of special tokens with dedicated meanings: <pad> (padding shorter sequences to a uniform length within a batch), <unk> (a fallback for anything genuinely unrepresentable), <bos>/<eos> (beginning/end of sequence, used in the autoregressive generation loop from Seq2Seq Model), and task-specific tokens like BERT's [CLS] and [SEP].
Code
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
print(tokenizer.vocab_size) # 30522
print(tokenizer.cls_token, tokenizer.sep_token, tokenizer.pad_token)
# [CLS] [SEP] [PAD] -- special tokens reserved in the vocabularyCommon Mistakes
- Using two different pretrained components (e.g. a tokenizer from one model with the embedding layer of another) without checking their vocabularies match exactly — token ids are meaningless numbers without a specific vocabulary mapping them to actual sub-word pieces.
- Underestimating how much a large vocabulary contributes to total model size — for a small model, the embedding and output layers can dominate the parameter count if the vocabulary is disproportionately large.
Interview Relevance
Q: "How does vocabulary size affect a language model's total parameter count?" Both the input embedding table and the final output projection layer scale directly and proportionally with vocabulary size (each contributing \(\text{vocab\_size}\times d_{\text{model}}\) parameters). For models with modest hidden dimensions but large vocabularies, these two layers can represent a substantial fraction of the model's total parameters.
Practice Question
For a model with \(d_{\text{model}}=512\) and a vocabulary of 30,000 tokens, roughly how many parameters does the embedding layer alone contain?
