Start studying
Module 1 · Machine Learning Module contents
Module 1  ·  Card 3 / 55beginner

Missing Data

Missing values are normal in real data and must be handled before training.

Dropping the rows is simplest, but it throws away information and can bias what is left.

Filling with the mean or median keeps the row while hiding that anything was absent.

Adding a flag column that marks the value as missing often helps more than the fill itself.

Missing can be informative, because the reason a field is blank may predict the outcome.

Learn in depth ↓
Module 01 · Machine Learning · 55 concepts
In depth  /  15 lines

Missing Data, in more detail

Real datasets have holes, and every model has to be told what to do about them.

Most algorithms simply refuse to run with a blank in the table.

The blunt option is to drop any row with a missing value.

That is fine if a handful of rows are affected and dangerous if many are.

Rows are rarely missing at random, so dropping them quietly changes who is in your data.

If income is blank more often for one group, you have just removed part of that group.

Dropping the whole column is reasonable when almost all of it is empty.

Imputation fills the gap instead, and the mean or the median is the usual filler.

The median is safer, since a few extreme values drag the mean around.

For categories the most common value is the equivalent choice.

Every fill invents data, which shrinks the apparent spread and can mislead the model.

Adding a separate flag column that records the value was missing usually helps more than the fill.

The fact of absence is often predictive in itself, since an unanswered question says something.

Fit the imputer on the training split only, or the test set has leaked into training.

More sophisticated methods exist, and a median plus a flag is a strong default.

Want the full technical article? Wikipedia ↗
Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
Why should the value used to fill gaps be computed on the training split only?
Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close