Start studying
Module 7 · Data Cleaning Module contents
Module 7  ·  Card 7 / 13intermediate

String Normalisation in Data

String normalisation puts text values into one standard form so that equal values compare as equal.

Typical steps are trimming spaces at both ends, making the letter case consistent and collapsing repeated spaces.

For example, a padded ' London', 'london' and 'LONDON' all become London, so a join or a count treats them as one city.

Accented letters can be stored in more than one way, so café may fail to match café until both are normalised.

Normalisation fixes form only; it cannot tell that NYC and New York mean the same place.

Learn in depth ↓
Module 07 · Data Cleaning · 13 concepts
In depth  /  15 lines

String Normalisation in Data, in more detail

Computers compare text character by character, so any difference at all makes 2 strings unequal.

People ignore spaces, capitals and accents when they read, which is why these differences go unnoticed.

Normalisation removes the differences that carry no meaning, before any join, count or deduplication.

Trimming removes spaces at the start and end, which often come from copying out of spreadsheets or forms.

Case folding makes every letter the same case, so London and LONDON compare as equal.

Collapsing runs of spaces inside the text means a name typed with a double space matches the normal one.

Invisible characters, such as non-breaking spaces, look like ordinary spaces but need removing separately.

Unicode, the standard computers use to store text, can write an accented letter as one character or as a letter plus a mark.

Unicode normalisation picks one of those forms, so both versions of café become identical.

Punctuation is a judgement call: removing full stops helps U.K. match UK, but removing every hyphen may merge distinct codes.

Normalisation can also go too far, since some codes are case sensitive and lowercasing them merges different values.

It is usually better to keep the original column and add a normalised one used only for matching.

In pandas, the str methods strip, lower and replace handle most steps across a whole column at once.

After normalising, a fresh value_counts often shows far fewer distinct values than before.

The common misconception is that normalisation fixes meaning, when it only fixes form, so variants like NYC still need a mapping.

Want the full technical article? Wikipedia ↗
Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
Joining a customer table to an orders table on city leaves many orders unmatched, though the cities look identical on screen. What is the likely cause?
Data Cleaning

13 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close