TF-IDF in NLP, explained in 5 lines

Module 04 · NLP · 20 concepts

In depth / 15 lines

TF-IDF, in more detail

Raw word counts are dominated by words that mean nothing, like the and of.

TF-IDF fixes that by weighting each word by how informative it is here.

It multiplies two things, one about this document and one about the whole collection.

Term frequency is how often the word appears in this document.

Inverse document frequency is a penalty for appearing in many documents.

A word in every document gets a weight of about zero.

A word in a handful of documents keeps most of its weight.

So the and of fall away and a word like mitochondria stands out sharply.

The frequency part is usually softened with a logarithm, since ten mentions is not ten times as relevant.

The output is still one row per document over the vocabulary, only better weighted.

Documents can then be compared with cosine similarity, which ignores their differing lengths.

It powered search engines for decades and still underlies BM25, its refined descendant.

It knows nothing about meaning, so a search for car misses every document saying automobile.

That blindness to synonyms is exactly what embeddings later solved.

It remains a strong baseline and half of every hybrid search system.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath TF-IDF

Two halves, multiplied

\mathrm{tfidf}(t, d) = \mathrm{tf}(t, d) \times \mathrm{idf}(t)

The first half says how much a term is used in this document. The second says how unusual the term is across all documents.

A word only scores highly if it is frequent here and rare elsewhere, which is a reasonable definition of "what this document is about".

Term frequency

\mathrm{tf}(t, d) = \frac{f_{t,d}}{\sum_{t' \in d} f_{t',d}}

$f_{t,d}$ : the raw count of term $t$ in document $d$ .

Dividing by the document's length stops long documents from outscoring short ones on every term.

\mathrm{tf}(t,d) = 1 + \log f_{t,d}

The log variant is often used instead, because a word appearing twenty times is not twenty times more indicative than one appearing once.

Inverse document frequency

\mathrm{idf}(t) = \log \frac{N}{n_t}

$N$ : total number of documents.
$n_t$ : how many of them contain $t$ .

A term in every document has $n_t = N$ , so the log is zero and the term contributes nothing. That is how "the" is removed without a stopword list.

A term in one document out of a million scores $\log(10^6) \approx 13.8$ .

\mathrm{idf}(t) = \log \frac{N}{1 + n_t} + 1

The smoothed form is what libraries actually use.

The inner $1 +$ stops division by zero for a term in the query but not the corpus, and the trailing one keeps a universal term off exactly zero.

Why the logarithm

Without it, rarity would scale linearly and a term appearing in 1 document would be a thousand times more important than one in 1000.

The log compresses that to a factor of about 3, which matches how usefulness actually behaves: rare is good, vanishingly rare is often just a typo.

One document, in order

Tokenise and count terms.
Divide each count by the document length, or take its log.
Look up each term's document frequency across the corpus.
Multiply the two halves.
Normalise the resulting vector to unit length, so cosine similarity behaves.

What it cannot do

Every term is its own dimension, so "car" and "automobile" are as unrelated as "car" and "banana". There is no notion of meaning anywhere in it.

It also cannot match a query whose words do not literally appear, which is the exact gap dense retrieval fills, and the exact reason hybrid search keeps both.

BM25 is the direct descendant and the better default: same two halves, with term frequency saturating and document length handled properly.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What does TF-IDF do that raw word counts do not?

NLP

20 concepts in this module, then a graded test with every answer explained.

Study this module →