BLEU and ROUGE in NLP, explained in 5 lines

Module 04 · NLP · 20 concepts

In depth / 15 lines

BLEU and ROUGE, in more detail

BLEU and ROUGE score generated text against one or more human-written references.

Both work by counting overlapping runs of words called n-grams.

BLEU is precision-flavoured, asking how much of what you produced appears in the reference.

That fits translation, where inventing extra content is the failure to avoid.

It adds a brevity penalty, or a two-word output would score perfectly.

ROUGE is recall-flavoured, asking how much of the reference appears in your output.

That fits summarisation, where the failure to avoid is leaving important content out.

ROUGE-1 counts single words, ROUGE-2 counts pairs, and ROUGE-L counts the longest shared run.

Both return a number between zero and one, and both are usually reported as percentages.

Their shared weakness is that they compare wording rather than meaning.

A summary that is correct in different words scores badly.

A summary that copies the reference and reverses one fact scores well.

Using several reference texts eases this, but references are expensive to write.

They are cheap, deterministic and reproducible, which is why they survived so long.

LLM-as-judge has largely replaced them for open-ended generation, at higher cost and lower reproducibility.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath BLEU and ROUGE

Two metrics, two directions

BLEU is a precision measure and ROUGE is a recall measure.

BLEU asks how much of what the model produced appears in the reference. ROUGE asks how much of the reference appears in what the model produced.

That is why BLEU belongs to translation, where inventing material is the failure, and ROUGE belongs to summarisation, where leaving material out is.

BLEU, written out

p_n = \frac{\sum_{\text{n-grams}} \min\big(C_{\mathrm{cand}}, \; C_{\mathrm{ref}}\big)}{\sum_{\text{n-grams}} C_{\mathrm{cand}}}

$C_{\mathrm{cand}}$ : how often an n-gram appears in the candidate.
$C_{\mathrm{ref}}$ : how often it appears in the reference.

The clip by the reference count is what stops a candidate of the word the repeated seven times from scoring a perfect precision.

\mathrm{BLEU} = \mathrm{BP} \cdot \exp\left( \sum_{n=1}^{4} w_n \log p_n \right)

Precisions for $n$ from 1 to 4 are combined by a geometric mean with equal weights, so a zero at any order sends the whole score to zero.

\mathrm{BP} = \begin{cases} 1 & c > r \ e^{1 - r/c} & c \leq r \end{cases}

The brevity penalty is the recall half. Without it, emitting one very likely word would score a perfect precision, and the penalty is the only thing punishing an output that is too short.

ROUGE, and its three common forms

\mathrm{ROUGE\text{-}N} = \frac{\sum_{\text{n-grams in ref}} C_{\mathrm{match}}}{\sum_{\text{n-grams in ref}} C_{\mathrm{ref}}}

ROUGE-N is n-gram recall, with ROUGE-1 and ROUGE-2 the two that are always reported.

\mathrm{ROUGE\text{-}L} = \frac{(1 + \beta^2) R_{\mathrm{lcs}} P_{\mathrm{lcs}}}{R_{\mathrm{lcs}} + \beta^2 P_{\mathrm{lcs}}}

ROUGE-L uses the longest common subsequence instead of fixed n-grams, so it rewards words in the right order without requiring them to be adjacent.

One evaluation, in order

Tokenise the candidate and every reference the same way, with the same casing rule.
Count the n-grams of each order.
Clip the candidate counts by the maximum over the references.
Compute the precisions or recalls, then combine them.
Report the tokeniser and the number of references alongside the number.

Step five is not a formality.

The same system scores several points differently under different tokenisers, which is why sacreBLEU exists and why a BLEU score without its configuration is not comparable to anything.

What both of them miss

Compare "The cat is on the mat" with "The mat has a cat on it".

A correct paraphrase that shares few n-grams scores badly, and a fluent sentence that shares many while reversing the meaning scores well.

Neither metric knows what a word means, so both are measuring surface overlap and calling it quality.

\text{BERTScore} : \max_{j} \; \mathbf{e}_i^{\top} \mathbf{e}_j

The learned alternatives match tokens by embedding similarity instead, which handles paraphrase and costs a forward pass per pair.

They remain useful because they are fast, deterministic and comparable across papers.

They also correlate well enough in aggregate, even though they are unreliable on any single sentence.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Why can a genuinely good summary still score poorly on ROUGE?

NLP

20 concepts in this module, then a graded test with every answer explained.

Study this module →