Tokenization splits raw text into the discrete units — tokens — a model actually processes. The choice of tokenization granularity is one of the most consequential early design decisions in any NLP pipeline.
Three Levels of Granularity
| Level | Example ("unhappiness") | Tradeoff |
|---|---|---|
| Word-level | ["unhappiness"] | Intuitive, but vocabulary grows huge, and any unseen word becomes an unusable "unknown" token |
| Character-level | ["u","n","h","a","p","p","i","n","e","s","s"] | Tiny vocabulary, no unknown-word problem, but sequences become very long and lose word-level structure |
| Subword-level | ["un", "happiness"] or ["un","happy","ness"] | The modern standard — balances vocabulary size against sequence length, and gracefully handles unseen words by falling back to familiar sub-pieces |
Why Subword Tokenization Won
Word-level tokenization has a fatal flaw: any word not seen during training (a typo, a rare technical term, a name) becomes an unknown token, losing all information. Subword algorithms — most commonly Byte-Pair Encoding (BPE) — solve this by learning a vocabulary of frequent sub-word pieces, so an unfamiliar word like "unhappiness" can still be represented by combining familiar pieces ("un" + "happiness"), even if the whole word was never seen during training.
How BPE Works, Conceptually
- Start with a vocabulary of individual characters.
- Count every pair of adjacent symbols across the training corpus.
- Merge the most frequent pair into a new single symbol, add it to the vocabulary.
- Repeat steps 2–3 for a fixed number of merges (determining the final vocabulary size).
This process naturally learns common prefixes, suffixes, and whole common words as single tokens, while keeping rare/unseen words representable via smaller, more common sub-pieces.
Code
# A modern subword tokenizer in practice (Hugging Face's tokenizers library)
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
tokens = tokenizer.tokenize("unhappiness is rare")
print(tokens)
# ['un', '##hap', '##pi', '##ness', 'is', 'rare'] -- an unfamiliar word gets split into familiar sub-piecesCommon Mistakes
- Assuming tokens always correspond to whole words — with subword tokenization, one word can become several tokens, and this directly affects sequence length, context window usage (covered in LLM Fundamentals), and even API pricing for commercial LLMs.
- Using a tokenizer trained on one language/domain for a very different one — a tokenizer trained mostly on English text will represent, say, dense technical code or a different language far less efficiently, producing many more tokens per unit of actual content.
Interview Relevance
Q: "Why do modern LLMs use subword tokenization instead of word-level tokenization?" Word-level tokenization can't represent any word not seen during training — it becomes an unusable "unknown" token, permanently losing information. Subword tokenization (typically BPE) learns a vocabulary of frequent sub-word pieces, so unfamiliar or rare words can still be represented by combining smaller, familiar pieces, while common whole words still get their own efficient single token.
Practice Question
Why does character-level tokenization avoid the "unknown word" problem entirely, and what's the cost of using it as your only tokenization strategy?
