Start studying
Module 7 · Data Cleaning Module contents
Module 7  ·  Card 10 / 13intermediate

Outlier Detection with IQR

The IQR rule flags a value as an outlier when it lies far outside the middle half of the data.

The fences sit 1.5 times the interquartile range below the first quartile and above the third quartile.

For example, with quartiles of 20 and 40 minutes the IQR is 20, so the fences are minus 10 and 70 minutes.

Quartiles barely move when a few extreme values are added, so the fences stay put even in messy data.

That makes the IQR rule a better default than z-scores for skewed data such as incomes or delivery times.

Learn in depth ↓
Module 07 · Data Cleaning · 13 concepts
In depth  /  15 lines

Outlier Detection with IQR, in more detail

The IQR rule flags outliers using quartiles, so it does not depend on the mean or the standard deviation at all.

That makes it usable on data of any shape, not only on bell-shaped data.

It needs only sorting and counting, so it is easy to compute and easy to explain.

Step 1 is to find the first and third quartiles, the values 25% and 75% of the way through the sorted data.

Step 2 is the IQR itself: the third quartile minus the first.

Step 3 sets the lower fence 1.5 IQRs below the first quartile and the upper fence 1.5 IQRs above the third.

Any value outside the fences is flagged for a closer look.

For a normal distribution the fences sit about 2.7 standard deviations from the mean, so very few values are flagged.

Using 3 IQRs instead of 1.5 gives outer fences, which flag only the most extreme values.

The rule ignores the shape inside the fences, so 2 clusters with a gap between them pass without a flag.

For strongly skewed data, many genuine values in the long tail still fall outside the upper fence.

Taking logarithms before applying the rule often gives a more sensible set of flags for incomes or prices.

A lower fence below what is possible, such as negative minutes, simply means no low values can be flagged.

In practice the rule is a good first screen for every numeric column during profiling.

The usual mistake is treating 1.5 as a law, when it is a convention that can be tuned to the data.

Test yourself

2 questions on this card. Every option explains itself once you pick, right or wrong.

Question 1 of 2
Delivery times have a first quartile of 20 minutes and a third quartile of 40 minutes. Under the IQR rule, where is the upper fence?
Question 2 of 2
Household incomes are strongly skewed, with a few very high earners. Why is the IQR rule a safer choice than z-scores here?
Data Cleaning

13 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close