Word Embeddings in NLP, explained in 5 lines

Module 04 · NLP · 20 concepts

In depth / 15 lines

Word Embeddings, in more detail

An embedding represents a word as a list of a few hundred numbers.

Those numbers are coordinates, so every word sits at a point in a space.

Words used in similar contexts end up near each other in that space.

That is the whole trick, and it comes from a very old observation about language.

You know a word by the company it keeps, so words with the same neighbours mean similar things.

Word2Vec learns this by training a small network to predict a word from its neighbours.

The prediction is thrown away and the internal weights are kept as the embeddings.

GloVe reaches a similar place by factorising a table of how often words co-occur.

Directions in the space turn out to carry meaning as well as position.

King minus man plus woman lands near queen, which is the famous demonstration.

Paris minus France plus Italy lands near Rome for the same reason.

Unlike bag of words this captures that cat and kitten are related without being told.

The limitation is that each word gets exactly one vector.

Bank by a river and bank with money share it, so both meanings are blurred together.

Contextual embeddings from transformers fixed that by giving each occurrence its own vector.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Word Embeddings

From an index to a direction

\text{one-hot} : \mathbf{w}_{\text{cat}} \cdot \mathbf{w}_{\text{dog}} = 0

In a bag of words every term is its own axis, so cat and dog are exactly as related as cat and tuesday.

E \in \mathbb{R}^{|V| \times d}, \qquad d \approx 100 \text{ to } 300

An embedding replaces that axis with a dense vector of a few hundred numbers, learned so that words used in similar contexts end up pointing in similar directions.

The idea goes back to the linguist J. R. Firth: "You shall know a word by the company it keeps".

That is the distributional hypothesis, and every embedding method is a way of turning it into an objective function.

Skip-gram, the objective

\max \sum_{t=1}^{T} \sum_{-c \leq j \leq c, \; j \neq 0} \log p(w_{t+j} \mid w_t)

$c$ : the window, how many words on each side count as context.

The model is given a word and asked to predict its neighbours, which is a task nobody wants solved. The vectors learned on the way are the product.

p(w_O \mid w_I) = \frac{\exp(\mathbf{v}'^{\top}_{w_O} \mathbf{v}_{w_I})}{\sum_{w=1}^{|V|} \exp(\mathbf{v}'^{\top}_{w} \mathbf{v}_{w_I})}

Written with a softmax, that denominator sums over the whole vocabulary at every training step, which is hundreds of thousands of dot products per word.

Negative sampling, which made it practical

\log \sigma\big(\mathbf{v}'^{\top}_{w_O} \mathbf{v}_{w_I}\big) + \sum_{k=1}^{K} \mathbb{E}_{w_k \sim P_n} \log \sigma\big(-\mathbf{v}'^{\top}_{w_k} \mathbf{v}_{w_I}\big)

$K$ : the number of negative samples, 5 to 20 for small corpora and 2 to 5 for large ones.

The multiclass problem is replaced by a binary one: separate the real neighbour from $K$ words drawn at random.

P_n(w) \propto f(w)^{3/4}

Negatives are drawn from the unigram distribution raised to the power three quarters, which samples rare words more often than their frequency alone would.

The exponent was found empirically and it matters, which is a fair summary of how much of this method was tuning.

One training run, in order

Build the vocabulary and discard words below a minimum count.
Subsample frequent words, since the thousandth example of the adds nothing.
Slide a window over the corpus, taking each centre word and its neighbours as positive pairs.
Draw $K$ negatives per pair and take a gradient step on the binary objective.
Keep the input matrix as the embeddings and discard the output matrix.

What the geometry ends up containing

\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}} \approx \mathbf{v}_{\text{queen}}

Directions turn out to carry relations, because a consistent difference in context produces a consistent difference in vector.

The result is genuine and oversold.

It holds for frequent words and clean relations, and the nearest neighbour is often one of the three input words, which is why the query usually excludes them.

\mathbf{v}_{\text{doctor}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}} \approx \mathbf{v}_{\text{nurse}}

The same mechanism encodes the corpus's biases as geometry, which is not a flaw in the method but a faithful measurement of the text it was trained on.

The limit that ended the era

\mathbf{v}_{\text{bank}}

One vector per word means one meaning per word, so the river bank and the savings bank are averaged into a single point that fits neither.

Contextual models produce a different vector for each occurrence, which is the whole difference between word2vec and BERT.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What do word embeddings capture that Bag of Words cannot?

NLP

20 concepts in this module, then a graded test with every answer explained.

Study this module →