Start studying
Module 7 · Data Cleaning Module contents
Module 7  ·  Card 8 / 13intermediate

Missing Value Patterns

Missing value patterns describe why values are missing, which decides whether ignoring the gaps biases the result.

Missing completely at random means the gaps have nothing to do with any value, like a form lost in the post.

Missing at random means the gaps depend on other recorded columns, such as younger customers skipping the income question.

Missing not at random means the gaps depend on the missing value itself, such as high earners declining to state income.

Only under the first pattern does dropping incomplete rows leave averages unbiased; under the other 2 it can shift them.

Learn in depth ↓
Module 07 · Data Cleaning · 13 concepts
In depth  /  15 lines

Missing Value Patterns, in more detail

A blank is a fact about how the data was collected, and why it happened matters as much as the values present.

The 3 patterns describe how the chance of a value being missing relates to the data.

They decide whether results from the complete rows can be trusted.

Missing completely at random is checked by comparing rows with and without gaps on the columns that are present.

If customers with a blank income look like everyone else on age, region and spending, the gaps may be random.

If they differ, for example by being mostly under 25, the data is not missing completely at random.

Missing at random can be handled with the related columns, such as estimating income separately within each age group.

Missing not at random cannot be detected from the data alone, because the evidence is the missing value itself.

Spotting it needs knowledge of how the data was collected, such as knowing that high earners tend to decline the question.

Missing values can also hide in plain sight, as zeros, 999, empty strings or the word unknown instead of a true blank.

Converting these placeholders to proper blanks comes before any pattern analysis, or the gaps go uncounted.

A grid showing which cells are present and which are missing, row by row, often reveals the pattern quickly.

A block of blanks across many columns at once often points to one failed source or one broken day of collection.

Recording the share of blanks and the suspected pattern lets readers judge how much the gaps could matter.

The usual mistake is dropping incomplete rows by default, which quietly turns the data into a different population.

Want the full technical article? Wikipedia ↗
Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
In a salary survey, the people with the highest incomes are the most likely to leave the income question blank. Which pattern is this?
Data Cleaning

13 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close