Tokenization is the algorithm that splits raw text into tokens, and converts each token to a numeric ID the model can actually process. Most modern LLMs use a subword algorithm called Byte Pair Encoding (BPE) or a close variant.
The Two-Step Process
Step 1 — Split text into tokens:
"unbelievable" → ["un", "believ", "able"]
Step 2 — Map each token to a numeric ID using the model's vocabulary:
["un", "believ", "able"] → [403, 12891, 522]
The model only ever sees the numeric IDs — the text itself is
just a human-readable representation of what those IDs mean.How Byte Pair Encoding (BPE) Builds Its Vocabulary
BPE is trained (once, ahead of time, on a large text corpus) by repeatedly merging the most frequently occurring pair of adjacent symbols:
Start: every word split into individual characters
"lower" → l o w e r
"lowest" → l o w e s t
Find most frequent adjacent pair across the corpus, e.g. "l"+"o"
Merge into "lo": l o w e r → lo w e r
Repeat thousands of times, building up a vocabulary of
increasingly larger frequent chunks — common whole words end
up as single tokens; rare words get split into meaningful
sub-pieces.This is why common English words are usually a single token, while rare words, made-up words, and most non-English text get split into more pieces — the vocabulary reflects what was frequent in the tokenizer's training data.
Every Model Has Its Own Tokenizer
Different model families are trained with different tokenizers and vocabularies — the same text produces a different token count and different token boundaries depending on the model. This is why token counting has to be done per-model, not with a single universal formula (see Token Count).
Practical Example — Code Tokenizes Differently Than Prose
"def calculate_total(price, tax_rate):"
This might tokenize into something like:
["def", " calculate", "_total", "(price", ",", " tax", "_rate", "):"]
Code, especially with underscores and unusual naming, often
tokenizes less efficiently (more tokens per character) than
plain prose — worth accounting for when estimating cost for
code-heavy applications.Common Mistakes
- Assuming token boundaries always align with word boundaries — they frequently don't, especially for technical terms, code, and non-English text
- Using one model's token count as an estimate for a different model — different tokenizers can produce meaningfully different counts for identical text
Interview Relevance
"Explain how BPE tokenization works, briefly" — the merge-based vocabulary-building process above is the expected shape of the answer; you don't need to reproduce the exact algorithm from memory, just the core idea.
Practice Question
Explain why the word "ChatGPT" might tokenize into 2-3 tokens even though it's one "word," using what you now know about how BPE vocabularies are built.