The BLEU score (Bilingual Evaluation Understudy) evaluates machine-generated text — most commonly machine translation — by measuring how much its word sequences (n-grams) overlap with one or more human-written reference translations.
The Core Idea: N-gram Precision
BLEU checks what fraction of the generated text's n-grams (contiguous sequences of \(n\) words — "n-grams" for \(n=1\) are single words, \(n=2\) are word pairs, and so on) also appear in the reference text(s). This is computed for several values of \(n\) (typically 1 through 4) and combined:
\(p_n\) is the n-gram precision for order \(n\); \(w_n\) are weights (commonly equal, e.g. \(\frac{1}{4}\) each for \(n=1,2,3,4\)); \(\text{BP}\) is a brevity penalty.
Why the Brevity Penalty Is Needed
Without it, a very short generated translation (e.g. just one correct, common word) could achieve artificially high n-gram precision simply by being too short to include much that's wrong. The brevity penalty explicitly penalizes generated text that's shorter than the reference, discouraging this kind of gaming:
\(c\) is the generated text's length, \(r\) is the reference length. If the generated text is at least as long as the reference, no penalty; if it's shorter, the penalty grows the more it falls short.
Numerical Example (Simplified, Unigram-Only)
Reference: "the cat sat on the mat." Generated: "the cat sat on mat." Unigram overlap: "the," "cat," "sat," "on," "mat" all appear in the reference — 5 out of 5 generated words match (unigram precision \(p_1=1.0\)), but the generated sentence is missing a word (5 words vs 6), triggering a brevity penalty that pulls the final score down from this otherwise perfect-looking unigram match.
Code
from nltk.translate.bleu_score import sentence_bleu
reference = [["the", "cat", "sat", "on", "the", "mat"]]
candidate = ["the", "cat", "sat", "on", "mat"]
score = sentence_bleu(reference, candidate)
print(score)Known Limitations
- BLEU rewards exact word-level overlap and doesn't understand meaning — a translation using entirely different, but equally correct, phrasing/synonyms can score poorly.
- It compares against a fixed set of reference translations, but natural language often has many equally valid ways to express the same meaning, not captured by counting exact n-gram matches.
Common Mistakes
- Treating BLEU as a perfect proxy for translation quality — it correlates reasonably well with human judgment in aggregate over large test sets, but can diverge substantially from human judgment for individual sentences, especially ones using valid paraphrasing.
- Comparing BLEU scores computed with different numbers of reference translations, tokenization schemes, or n-gram orders — these implementation details can meaningfully shift the resulting score.
Interview Relevance
Q: "What's a key limitation of BLEU as a translation quality metric?" BLEU measures n-gram overlap with reference translations, but doesn't understand meaning — a correct translation that happens to use different (but equally valid) wording or synonyms than the reference can score poorly, even though it's a genuinely good translation. This is exactly why BLEU is typically used in aggregate over large test sets rather than trusted for judging any single translation in isolation.
Practice Question
Why does BLEU need a brevity penalty in addition to n-gram precision alone?
