Start studying
Module 7 · Data Cleaning Module contents
Module 7  ·  Card 6 / 13intermediate

Date Parsing and Time Zones

Date parsing turns text such as 03/04/2024 into a real date that can be sorted, compared and subtracted.

The same text means 3 April in the UK and 4 March in the US, so the format must be stated, not guessed.

Timestamps also need a time zone, because 09:00 in London and 09:00 in New York are different moments.

A common approach is to store every timestamp in UTC, the world reference time, and convert to local time only for display.

For example, daily sales totals shift between days if late evening orders are recorded in 2 different time zones.

Learn in depth ↓
Module 07 · Data Cleaning · 13 concepts
In depth  /  15 lines

Date Parsing and Time Zones, in more detail

Dates cause more silent cleaning errors than almost any other type, because wrong dates still look like dates.

A date stored as text cannot be sorted in time order, subtracted or grouped by month.

Parsing converts it into a true date value, which pandas stores as datetime64.

The safest text format is year, month, day, as in 2024-04-03, because it cannot be read 2 ways.

For any other format the parser should be told the order, such as day first, rather than left to guess.

A parser that guesses row by row can read 13/04 as day first and 03/04 as month first in the same column.

A timestamp without a time zone is called naive, and one with a time zone is called aware.

Naive timestamps from 2 sources cannot be compared safely, because nobody knows which clock each came from.

Converting everything to UTC removes that doubt and makes durations correct across sources.

Daylight saving time adds traps: one local hour is skipped each spring and one repeats each autumn.

A naive local timestamp in the repeated hour is ambiguous, because that time happened twice.

Dates without times, such as birthdays, usually should not be converted between time zones at all.

In practice, store UTC, keep the original time zone if local time matters, and convert only for display.

Grouping by day should use the business's local day, or late evening sales fall into the next day.

The usual mistake is checking only dates with a day over 12, which cannot be misread, and missing the ones that can.

Want the full technical article? Wikipedia ↗
Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
A UK shop's order file has dates like 03/04/2024, meaning 3 April. A script parses them month first, the US way. What goes wrong?
Data Cleaning

13 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close