Retrieval Metrics in AI evaluation

Module 13 · AI Evaluation · 18 concepts

In depth / 15 lines

Retrieval Metrics, in more detail

Retrieval metrics measure whether search put the right documents in front of the model.

Recall at k asks whether a relevant document appears anywhere in the top k.

It is the metric that matters most for RAG, since a reranker can sort what was retrieved.

A chunk never retrieved is unrecoverable.

Precision at k asks what fraction of the top k are relevant.

It matters more when context space is tight, since irrelevant chunks crowd out good ones.

The two trade against each other as k grows.

Mean reciprocal rank rewards putting the first relevant result as high as possible.

It uses only the first hit, which suits questions with one right answer.

NDCG accounts for graded relevance and for position together.

It handles the case where some documents are highly relevant and others only partly.

It is the standard in search evaluation and needs graded judgements to compute.

All of these require knowing which documents are actually relevant.

Building that ground truth by hand is the real work, and there is no shortcut.

Fifty well-chosen questions with labelled correct chunks beats a thousand guessed ones.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Retrieval Metrics

Precision and recall, cut at k

P@k = \frac{|\text{relevant} \cap \text{top } k|}{k}, \qquad R@k = \frac{|\text{relevant} \cap \text{top } k|}{|\text{relevant}|}

Precision at 5 asks how many of the five results were useful. Recall at 5 asks how many of the useful documents made it into five.

k \text{ small} \implies R@k \text{ bounded by } \frac{k}{|\text{relevant}|}

Recall at a small $k$ has a ceiling below 1 whenever there are more relevant documents than slots, which is why the two are always read together.

Neither notices order. A relevant document at position 1 and at position 5 score identically, and for a system feeding a model that is a real omission.

MRR, for when there is one right answer

\mathrm{MRR} = \frac{1}{|Q|} \sum_{q=1}^{|Q|} \frac{1}{\mathrm{rank}_q}

$\mathrm{rank}_q$ : the position of the first relevant result for query $q$ .

First place scores 1, second scores 0.5, third 0.33, and a query with nothing relevant scores 0.

It is the right metric for question answering and navigational search, where the user wants one answer and will not read past it.

It ignores everything after the first hit, so a system that returns one good result and four terrible ones scores perfectly.

nDCG, which uses the order and the grades

\mathrm{DCG}@k = \sum_{i=1}^{k} \frac{2^{rel_i} - 1}{\log_2(i + 1)}

$rel_i$ : the graded relevance of the result at position $i$ , often 0 to 3.

The numerator rewards higher grades exponentially, so one excellent result outweighs several mediocre ones.

The denominator discounts by position, so the same document is worth less the further down it appears.

\mathrm{nDCG}@k = \frac{\mathrm{DCG}@k}{\mathrm{IDCG}@k}

Dividing by the best possible ordering normalises it to the range 0 to 1, which makes queries with different numbers of relevant documents comparable.

One evaluation, in order

Fix a query set that reflects real traffic, not invented questions.
Label the relevance of the pooled results from every system being compared.
Compute the metrics per query.
Average across queries, and report the spread as well as the mean.
Fix $k$ to what the downstream stage actually consumes.

Pooling matters. Labelling only one system's results and evaluating another against those labels counts its unlabelled correct answers as wrong.

Choosing which one to report

Measure the retrieval stage by recall at $k$ .

The first stage's job is not to miss anything, because nothing downstream can recover a document that was never returned.

Measure the rerank stage by nDCG at $k$ or by MRR.

The reranker's job is order, so it should be measured by a metric that knows about order.

For RAG, measure recall at $k$ , where $k$ is the context budget.

For retrieval feeding a model, recall at the number of chunks that fit is the number that predicts answer quality.

Precision matters too, because irrelevant context measurably degrades the generation.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Which retrieval metric usually matters most for a RAG system?

AI Evaluation

18 concepts in this module, then a graded test with every answer explained.

Study this module →