Start studying
Module 7 · Data Cleaning Module contents
Module 7  ·  Card 9 / 13intermediate

Outlier Detection with Z-Scores

A z-score says how many standard deviations a value lies above or below the mean.

It is the value minus the mean, divided by the standard deviation.

For example, if deliveries average 30 minutes with a standard deviation of 5, a 50 minute delivery has a z-score of 4.

A common rule flags values with a z-score above 3 or below minus 3 as possible outliers.

The rule suits roughly bell-shaped data, and extreme values inflate the very standard deviation meant to catch them.

Learn in depth ↓
Module 07 · Data Cleaning · 13 concepts
In depth  /  15 lines

Outlier Detection with Z-Scores, in more detail

A raw value says little on its own, because 50 is extreme for delivery minutes and ordinary for exam marks.

The z-score rescales any value into standard deviations from the mean, so values from different columns become comparable.

That shared scale is what makes a single cut-off, such as 3, usable across many columns.

The steps are to work out the mean and the standard deviation, then subtract the mean from each value and divide.

A value equal to the mean has a z-score of 0, and values below the mean get negative scores.

If data follows a normal distribution, about 99.7% of values fall within 3 standard deviations of the mean.

So a z-score beyond 3 should appear only about 3 times in every 1,000 values from a normal distribution.

In a table of 1 million rows, that still means around 2,700 flags even when nothing is wrong.

The cut-off can be raised for large tables or lowered for small ones, depending on how many flags are useful.

The method has a weakness called masking: a few huge values raise the standard deviation and hide each other.

In a column of 10 values or fewer, no value can score above 3 at all, however extreme it is.

Robust versions replace the mean with the median and the standard deviation with a spread measure that ignores extremes.

In practice z-scores suit roughly bell-shaped measurements, such as sensor readings or test marks.

For skewed columns, taking logarithms first can make the shape closer to a bell before scoring.

The usual mistake is removing everything beyond 3 automatically, when the cut-off only says where to look.

Want the full technical article? Wikipedia ↗
Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
Delivery times average 30 minutes with a standard deviation of 5. Under the usual z-score rule, which delivery time is flagged?
Data Cleaning

13 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close