TracksStudyAbout
Sign in
Start studying
Module 4 · NLP Module contents
Module 4  ·  Card 4 / 20beginner

Bag of Words

Bag of Words represents a document purely by which words appear and how often.

Word order is thrown away completely, which is where the name comes from.

Each document becomes a long, mostly zero vector over the vocabulary.

It is simple, fast and a decent baseline for topic classification.

It cannot tell dog bites man from man bites dog.

Back Next
Learn in depth ↓
Module 04 · NLP · 20 concepts
In depth  /  15 lines

Bag of Words, in more detail

Bag of words is the simplest way to turn a document into numbers.

You build a vocabulary from your corpus and count how often each word appears.

Each document becomes one long row of counts, one column per vocabulary word.

With a vocabulary of 20,000 words, a short document is 20,000 numbers of which almost all are zero.

The name comes from throwing the words into a bag and shaking it.

Order is destroyed entirely, so the dog bit the man matches the man bit the dog.

For topic classification that hardly matters, since the topic lives in which words appear.

For sentiment or meaning it matters a great deal.

It also has no idea that cat and kitten are related, as they are just two columns.

Binary bag of words records presence rather than count, which sometimes works better.

N-grams recover a little order by counting adjacent pairs alongside single words.

That captures not good as its own feature rather than a good with a not somewhere nearby.

It is fast, needs no training, and is easy to inspect when something goes wrong.

It remains a sensible baseline before you reach for anything heavier.

TF-IDF is the natural next step, weighting the counts by how informative each word is.

Want the full technical article? Wikipedia ↗
Deep dive

What is underneath Bag of Words

A document as a vector of counts

d=[ c1,  c2,  …,  c∣V∣ ]\mathbf{d} = [\, c_1, \; c_2, \; \dots, \; c_{|V|} \,]
  • cic_i: how many times vocabulary term ii appears in the document.
  • ∣V∣|V|: the size of the vocabulary.

Every document becomes a vector of the same length, whatever its actual length, and that fixed shape is what lets ordinary models read text at all.

"the cat sat on the mat"→{the:2,  cat:1,  sat:1,  on:1,  mat:1}\text{"the cat sat on the mat"} \to \{\text{the}: 2, \; \text{cat}: 1, \; \text{sat}: 1, \; \text{on}: 1, \; \text{mat}: 1\}

The order is discarded, which is the assumption in the name. The bag knows what was said and not how it was arranged.

The two things that go wrong immediately

"dog bites man"="man bites dog"\text{"dog bites man"} = \text{"man bites dog"}

Losing order loses the relation, and for some tasks that is fatal while for topic classification it barely matters.

nnz(d)≪∣V∣\text{nnz}(\mathbf{d}) \ll |V|

The vectors are enormous and almost entirely zero. A 50,000-term vocabulary and a 200-word document means 0.4 percent of the entries are non-zero.

That sparsity is the reason these vectors are stored as index and value pairs rather than as arrays, and why linear models on them are fast despite the dimension.

Weighting the counts

tf(t,d)=ct∑t′ct′\mathrm{tf}(t, d) = \frac{c_t}{\sum_{t'} c_{t'}}

Raw counts favour long documents, so the counts are usually normalised by length first.

wt=1+log⁡ctw_t = 1 + \log c_t

Sublinear scaling is the other common correction, since a term appearing 100 times is not 100 times as important as one appearing once.

The frequency problem remains: the most common terms carry the least information, which is exactly what TF-IDF is for and why a bag of raw counts is rarely used on its own.

One document, in order

  1. Lowercase and strip punctuation, unless case or punctuation is a signal.
  2. Split into tokens.
  3. Remove stop words, or keep them and let the weighting handle it.
  4. Map each token to its vocabulary index, dropping anything unseen.
  5. Count, weight, and store the non-zero entries.
h(t) mod mh(t) \bmod m

The hashing trick replaces steps four and five: hash each token into one of mm buckets and skip the vocabulary entirely, which fixes the memory and accepts a small collision rate.

Where it still wins

n-grams:"not good"≠"good"\text{n-grams} : \text{"not good"} \neq \text{"good"}

Adding bigrams recovers a little of the lost order, at the cost of a vocabulary several times larger.

A bag of words with TF-IDF and a linear classifier is a serious baseline on topic classification, spam and sentiment, and it trains in seconds on a laptop.

It has no idea that two words mean the same thing, because every term is its own orthogonal dimension. That single limitation is what embeddings were invented to remove.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
What is the main limitation of Bag of Words?
Concepts these five lines lean on
ClassificationN-Grams
NLP

20 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close