Running, ran and runs are the same word wearing different endings.
Both techniques collapse those forms so a search for run finds all of them.
Stemming chops endings off by rule, which is fast and sometimes produces non-words like studi.
Lemmatization uses a dictionary and the part of speech to return the real base word, such as run.
Modern models rarely need either, because subword tokenization already shares the common stem.
Language buries the same idea under many surface forms, and the earliest text systems could not see past that.
A keyword index that stores running will not match a query for run unless something normalises them.
Stemming is the crude answer: strip suffixes according to a list of rules.
The Porter stemmer is the classic, a fixed cascade of rules applied to English endings.
It is fast, needs no dictionary, and often returns something that is not a word at all.
Studies becomes studi, and both connect and connection collapse to connect, which is usually what you wanted.
The failures come in pairs: over-stemming merges unrelated words, under-stemming misses related ones.
Universal and university both stem to univers, which is over-stemming doing real damage.
Lemmatization does the job properly by looking the word up and returning its dictionary form.
That requires knowing the part of speech, since meeting as a noun and as a verb lemmatise differently.
It is slower and needs language resources, but the output is always a real word.
Either step shrinks the vocabulary, which used to matter enormously for bag of words and TF-IDF.
A smaller vocabulary meant smaller, denser matrices and better matching on sparse data.
Subword tokenization changed the calculus, since run and running already share their first piece.
So these techniques now live mostly in classical search stacks rather than in anything built on transformers.
One question on this card. Every option explains itself once you pick, right or wrong.
20 concepts in this module, then a graded test with every answer explained.