TracksStudyAbout
Sign in
Start studying
Module 1 · Machine Learning Module contents
Module 1  ·  Card 43 / 55beginner

Accuracy

Accuracy is the fraction of predictions the model got right.

It is correct predictions divided by total predictions.

It is intuitive and works well when classes are roughly balanced.

It is badly misleading when they are not.

If 99% of emails are not spam, always guessing not spam scores 99% and catches nothing.

Back Next
Learn in depth ↓
Module 01 · Machine Learning · 55 concepts
In depth  /  15 lines

Accuracy, in more detail

Accuracy is the first metric everyone reaches for and often the wrong one.

It counts every correct answer equally, whatever kind it was.

That is fine when both classes turn up about as often as each other.

It falls apart the moment one class is rare.

A test for a disease that one in a thousand people have scores 99.9 percent by always saying no.

That model is useless and its accuracy looks excellent.

The number also hides which kind of mistake is being made.

Two models on the same data can share an accuracy and behave completely differently.

It says nothing about how confident the model was.

Being barely right and being certain both count the same.

It also depends on the threshold you happened to choose.

Move the threshold and accuracy moves, though the model has not changed at all.

Report it next to the class balance, or it means very little.

For rare events, precision, recall and PR-AUC tell you far more.

Use it as a sanity check, not as the thing you optimise.

Want the full technical article? Wikipedia ↗
Deep dive

What is underneath Accuracy

The formula

Accuracy=TP+TNTP+TN+FP+FN\mathrm{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}
  • TPTP: predicted positive, actually positive.
  • TNTN: predicted negative, actually negative.
  • FPFP: predicted positive, actually negative. A false alarm.
  • FNFN: predicted negative, actually positive. A miss.

The denominator is simply every prediction made, so accuracy is the fraction you got right.

Why it is the wrong number more often than it is the right one

Consider a disease that one person in a thousand has, and a model that always says "no".

Accuracy=0+9991000=99.9%\mathrm{Accuracy} = \frac{0 + 999}{1000} = 99.9\%

It has never been right about a single case that mattered, and it scores better than most real models.

The problem is that accuracy weights every error the same, and the two errors are almost never equally costly.

The baseline it must beat

Accuracytrivial=max⁡(PP+N,NP+N)\mathrm{Accuracy}_{\text{trivial}} = \max\left( \frac{P}{P+N}, \frac{N}{P+N} \right)

Predicting the majority class every time gives you this for free, so an accuracy figure quoted without it says nothing.

At 90% class imbalance, 90% accuracy is not a result. It is the absence of one.

When it is the right metric

  1. The classes are roughly balanced.
  2. A false positive and a false negative cost about the same.
  3. You care about overall correctness rather than about one class.

All three have to hold. If any one of them fails, reach for precision, recall or the pair together.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
99% of your emails are not spam. Your model labels everything not spam. What is its accuracy and what is the problem?
Concepts these five lines lean on
PrecisionPR-AUCRecallROC-AUC
Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close