ROC-AUC in machine learning, explained in 5 lines

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

ROC-AUC, in more detail

ROC-AUC does not care about your threshold at all.

It asks a simpler question: does the model rank cases in the right order?

Take one real positive and one real negative at random.

AUC is the chance the model gave the positive one the higher score.

A perfect ranker gets 1.0 and random guessing gets 0.5.

Below 0.5 means it is ranking backwards, which is worth investigating.

Being threshold-free, it compares models fairly before you have tuned anything.

Its weakness shows when negatives vastly outnumber positives.

The false positive rate barely moves when the denominator is enormous.

So a model can look strong at 0.95 while being useless in practice.

Fraud detection is the classic case where exactly this happens.

It also treats the two kinds of mistake as equally important.

It says nothing about whether the probabilities themselves are sensible.

A model can rank perfectly and still be wildly overconfident.

For rare positives use PR-AUC, and keep ROC-AUC as the secondary number.

Want the full technical article? Wikipedia ↗

Deep dive

TPR = \frac{TP}{TP + FN} \qquad FPR = \frac{FP}{FP + TN}

$TPR$ : the true positive rate, which is recall: the fraction of real positives caught.
$FPR$ : the false positive rate: the fraction of real negatives wrongly flagged.

Sweep the threshold from high to low, plot $TPR$ against $FPR$ at every value, and that path is the ROC curve.

The area underneath it is the AUC, a single number between 0 and 1.

The area has an exact probabilistic reading, and it is far more useful than "area under a curve".

\mathrm{AUC} = P\big( s(x^{+}) > s(x^{-}) \big)

AUC is the probability that the model ranks a random positive above a random negative.

So 0.5 is a coin toss, 1.0 is a perfect ranking, and 0.5 is the floor rather than 0: a model scoring 0.2 is a good model with its sign flipped.

\mathrm{AUC} = \int_0^1 TPR \, d(FPR)

Because it integrates over every threshold, AUC measures the quality of the ranking rather than of any particular decision.

That makes it the right metric when you have not chosen an operating point yet, and the wrong one when you have.

It also says nothing about calibration: a model can rank perfectly and still output probabilities that are all wrong.

$FPR$ has $TN$ in its denominator, and under heavy imbalance $TN$ is enormous.

With 1000 negatives, going from 10 false positives to 50 moves $FPR$ from 0.01 to 0.05, which is barely visible on the plot.

But precision may have collapsed from 0.5 to 0.17 over the same move, because precision divides by the false positives directly.

That is why PR-AUC is the honest curve when positives are rare, and ROC-AUC flatters models on imbalanced data.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What does an AUC of 0.5 tell you?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.