Start studying
Module 7 · Data Cleaning Module contents
Module 7  ·  Card 1 / 13beginner

Data Profiling

Data profiling is a first pass over a new dataset that summarises what each column actually contains.

For each column it counts rows, blanks and distinct values, and reports the type, the minimum and the maximum.

For example, profiling a customer file might show that 3% of ages are blank and that one age is recorded as 210.

In pandas, the info, describe and value_counts methods give most of a profile in a few lines of code.

Profiling comes before cleaning, because you cannot fix problems you have not yet measured.

Learn in depth ↓
Module 07 · Data Cleaning · 13 concepts
In depth  /  15 lines

Data Profiling, in more detail

Profiling answers one question before any cleaning starts: what is actually in this data?

Column names and documentation describe what a dataset should hold, and a profile shows what it does hold.

That gap is where most cleaning work comes from.

A basic profile has one entry per column, with its type, row count, blank count and number of distinct values.

Numeric columns add the minimum, maximum, mean, median and a few percentiles.

Text columns add the most common values and their counts, which is where spelling variants show up.

Date columns add the earliest and latest dates, which catches placeholder years such as 1900 or 2099.

A distinct count equal to the row count suggests an identifier, while a distinct count of 1 means the column never varies.

Comparing the row count with the number of distinct identifiers is the fastest test for duplicates.

A profile looks at columns one at a time, so it misses problems between columns, such as a delivery before its order.

Summary figures can also hide trouble, because a mean can look normal even when a few values are impossible.

On very large tables a profile is often run on a portion of the rows first, then on the full table for final counts.

In practice, saving each profile alongside the data lets the next load be compared with this one.

A column whose blank rate jumps from 1% to 40% between loads usually means something upstream broke.

The usual mistake is skipping the profile because the data came from a trusted system, which is where silent errors hide best.

Want the full technical article? Wikipedia ↗
Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
You receive a customer file from another team and need to clean it. What should you do before changing anything?
Data Cleaning

13 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close