Start studying
Module 7 · Data Cleaning Module contents
Module 7  ·  Card 13 / 13advanced

Schema Enforcement

Schema enforcement rejects data whose structure does not match an agreed schema: the expected columns, types and constraints.

It runs before any row rule, checking that every column exists, has the right type and allows blanks only where permitted.

For example, if a supplier renames price to unit_price, enforcement stops the load before an empty price column arrives.

Databases enforce a schema when data is written, while CSV files carry none, so code must check them when reading.

Strict enforcement can block a whole delivery over one changed column, so teams agree how schema changes are announced.

Learn in depth ↓
Module 07 · Data Cleaning · 13 concepts
In depth  /  15 lines

Schema Enforcement, in more detail

A schema is the agreed shape of a dataset: its column names, their types, and which of them may be blank.

Schema enforcement checks that shape before anything else, because row rules mean little if the columns are wrong.

It protects every later step from structural surprises, such as a missing column or an unexpected new one.

Databases apply schema on write: the table defines the columns, and an insert that does not fit is refused.

CSV and JSON files carry no schema of their own, so the reading code must check their structure itself.

That approach is called schema on read, because the structure is interpreted only when the data is used.

A typical check confirms each expected column exists, has the expected type, and has no blanks where none are allowed.

It also decides what happens to unexpected extra columns, either ignoring them or stopping to ask.

Schema drift is the slow change of a source's structure over time, through renamed, added or retyped columns.

Drift caught early costs a quick fix, while drift caught late means repairing weeks of stored data.

Some changes are safe, such as a new optional column, while others break readers, such as a renamed or removed one.

Teams often write the schema down as a contract with the source team, and update it by agreement when it changes.

Some file formats store a schema inside the file, which makes these checks easier than with plain CSV.

In practice enforcement runs first when data is loaded, followed by validation rules on the rows that pass.

The usual mistake is letting the reading code adapt silently, so a renamed column turns into blanks nobody notices.

Want the full technical article? Wikipedia ↗
Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
A pipeline fills any missing column with blanks. A supplier renames price to unit_price without warning. What does schema enforcement change?
Data Cleaning

13 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close