Sample Size in AI evaluation, explained in 5 lines

Module 13 · AI Evaluation · 18 concepts

In depth / 15 lines

Sample Size, in more detail

Sample size decides how much of your evaluation score is real and how much is noise.

It is the most ignored number in applied machine learning.

On fifty examples, a difference of a few per cent is almost certainly noise.

Model A at 84 per cent and model B at 80 per cent on fifty items is not a result.

That gap is two examples going the other way.

The smaller the set, the wider the range the true score could sit in.

A confidence interval expresses that, and reporting a bare number hides it.

At fifty examples, an 80 per cent score has an interval of roughly 68 to 89 per cent.

At five hundred it narrows to about 76 to 83 per cent.

So a small improvement needs a large test set to be visible at all.

Detecting a two-point difference reliably takes thousands of examples.

Comparing on the same examples helps, since paired comparison removes item difficulty from the noise.

Rare failures need proportionally more data, since a one per cent failure rate needs hundreds to observe.

Report the interval or at least the sample size alongside every score.

Comparing two models on a tiny test set is the most common way people fool themselves.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Sample Size

How wrong a measured rate can be

\mathrm{SE} = \sqrt{\frac{p(1-p)}{n}}

$p$ : the measured success rate.
$n$ : the number of examples.

An accuracy of 80 percent on 100 examples has a standard error of 4 points, so the true value is plausibly anywhere from 72 to 88.

p \pm 1.96 \, \mathrm{SE}

That interval is what a single number on an evaluation set hides, and it is why two models three points apart on 100 examples are not distinguishable.

n \times 4 \implies \mathrm{SE} \div 2

Precision improves with the square root, so halving the uncertainty costs four times the labelling.

Sizing an evaluation set

n = \frac{z^2 \, p(1-p)}{\varepsilon^2}

$\varepsilon$ : the margin of error you are willing to accept.
$z = 1.96$ for 95 percent confidence.

For a margin of 3 points around an expected 80 percent accuracy, that is 683 examples. For 1 point, it is 6,147.

p = 0.5 \implies \text{largest } n

The variance is highest at 50 percent, so using 0.5 gives a safe upper bound when the rate is unknown.

Comparing two models needs more than either alone

\mathrm{SE}_{\mathrm{diff}} = \sqrt{\frac{p_1(1-p_1)}{n_1} + \frac{p_2(1-p_2)}{n_2}}

The error of a difference combines both errors, so distinguishing two similar systems takes roughly twice the data of measuring one.

Running both models on the identical set removes the variance from which examples were drawn, and it usually cuts the required size by half or more.

b, \; c \text{ where they disagree}

McNemar's test uses only the disagreements: the cases one got right and the other wrong. Everything both got right or both got wrong carries no information about which is better.

One evaluation, in order

Decide the smallest difference that would change a decision.
Compute the $n$ needed to detect it, before labelling anything.
Evaluate both systems on the same examples.
Report the interval, not only the point estimate.
Test the difference on the disagreements rather than comparing two averages.

Where the arithmetic is optimistic

Correlated examples make the effective $n$ smaller than $n$ .

Ten questions about the same document are not ten independent samples, so a set built by expanding a few sources has less power than its size suggests.

Repeatedly checking an evaluation set and stopping when the result looks good inflates the false positive rate, exactly as it does in A and B testing.

Test 20 metrics at $p < 0.05$ and one false win is expected.

Reporting many metrics on the same set invites the same problem, and the honest correction is to decide the primary metric first and treat the rest as description.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Model A scores 82 percent and model B scores 79 percent on 50 examples. What can you conclude?

AI Evaluation

18 concepts in this module, then a graded test with every answer explained.

Study this module →