Preprocessing cleans raw text into a consistent form before modelling.
Typical steps are lowercasing, stripping punctuation and collapsing whitespace.
Stop words such as the and is are sometimes removed for classical models.
Stemming chops word endings while lemmatization maps words to a proper dictionary form.
Modern transformer pipelines do far less of this, since the tokenizer handles most of it.
Preprocessing is everything you do to raw text before a model ever sees it.
Real text arrives with stray whitespace, mixed case, HTML tags and inconsistent punctuation.
A classical model treats every surface difference as a different word.
So Dog, dog and dog. would be three separate entries in the vocabulary.
Lowercasing collapses the first two, at the cost of losing the signal in a proper noun.
Stripping punctuation and collapsing whitespace removes most of the rest.
Stop words are the very common words such as the, is and of.
Removing them shrinks the vocabulary and helps a bag-of-words model considerably.
Stemming chops endings mechanically, turning running into run and studies into studi.
The result need not be a real word, which is fine if you only ever compare it to itself.
Lemmatization uses a dictionary instead, mapping ran and running to run properly.
It is slower but produces something a human can read.
Modern transformer pipelines do almost none of this, because the tokenizer handles it.
Removing stop words before a transformer actively hurts, since it needs the grammar.
The rule is to preprocess hard for classical models and barely at all for modern ones.
One question on this card. Every option explains itself once you pick, right or wrong.
20 concepts in this module, then a graded test with every answer explained.