The ROUGE score (Recall-Oriented Understudy for Gisting Evaluation) is BLEU's counterpart for text summarization — where the priority flips from "did the generated text avoid adding wrong words" (precision) to "did the generated text capture the important content from the reference" (recall).
ROUGE-N: N-gram Recall
Notice the denominator: unlike BLEU's precision (which divides by n-grams in the generated text), ROUGE-N divides by n-grams in the reference text — directly measuring how much of the reference's content the generated summary successfully captured, which is exactly the priority for summarization (where missing important content is a bigger concern than including a few extra words).
ROUGE-L: Longest Common Subsequence
ROUGE-L instead measures the length of the longest common subsequence (LCS) between generated and reference text — a sequence of words appearing in the same relative order in both texts, though not necessarily contiguously. This captures sentence-level structural similarity in a way that's more flexible than requiring exact contiguous n-gram matches.
Numerical Example (ROUGE-1, Unigram Recall)
Reference summary: "the company reported strong quarterly earnings." Generated summary: "company reported earnings." Unigram overlap: "company," "reported," "earnings" — 3 words. Reference has 6 words total.
In practice, ROUGE is usually reported alongside a corresponding precision and F1 score too, not recall alone — giving a fuller picture, similar to how classification metrics pair precision and recall (see Precision & Recall).
Code
from rouge_score import rouge_scorer
scorer = rouge_scorer.RougeScorer(['rouge1', 'rougeL'], use_stemmer=True)
scores = scorer.score(
"the company reported strong quarterly earnings",
"company reported earnings"
)
print(scores) # includes precision, recall, and F1 for each ROUGE variantBLEU vs ROUGE — Why the Emphasis Differs
| BLEU | ROUGE | |
|---|---|---|
| Primary emphasis | Precision (did the generated text avoid wrong content?) | Recall (did the generated text capture the reference's content?) |
| Typical task | Machine translation | Text summarization |
| Why the different emphasis | A wrong or extra word in a translation is a clear error | Missing important content in a summary is usually a bigger concern than including a few extra words |
Common Mistakes
- Using BLEU for summarization or ROUGE for translation without understanding why each metric emphasizes precision or recall differently — the choice of metric should match which type of error matters more for the specific task.
- Treating ROUGE (or BLEU) scores as capturing factual correctness — both are purely surface-level text overlap metrics and say nothing about whether a summary is factually accurate to the source document.
Interview Relevance
Q: "Why does ROUGE emphasize recall while BLEU emphasizes precision?" Summarization's key failure mode is missing important content from the source — recall directly measures how much of the reference's content the summary captured. Translation's key failure mode is producing incorrect or extraneous content — precision directly measures how much of the generated text is actually correct relative to the reference. Each metric's emphasis matches the error type most relevant to its target task.
Practice Question
Reference: "the stock market fell sharply today." Generated: "stock market fell today amid concerns." Compute ROUGE-1 recall (fraction of the reference's 6 words that appear in the generated text).
