Static vs Contextual Embeddings

Word embeddings arrived as a lookup table: every word in the vocabulary owned one vector.

Word2Vec and GloVe learned those vectors from co-occurrence statistics over a large corpus.

The famous arithmetic, king minus man plus woman landing near queen, came from exactly that table.

The design has one unavoidable flaw, and it is a flaw of the format rather than the training.

Every sense of a word must share a single vector, because there is only one row to store it in.

So bank ends up somewhere between the financial sense and the river sense, and is a poor match for either.

This is the polysemy problem, and no amount of extra data fixes it while the table shape holds.

Contextual embeddings change the shape: the representation becomes a function of the whole sentence.

The same word in two sentences produces two different vectors, computed on the spot.

ELMo did this with stacked recurrent networks and BERT did it with self-attention.

In a sentence about a riverside branch, the two occurrences of bank land in quite different regions.

That is the direct payoff, and it lifted almost every downstream task when it arrived.

The cost is that you can no longer ship a table: producing a vector now requires running a model.

Static vectors are still worth knowing about, since they are tiny, instant and adequate for rough similarity.

Every embedding model behind modern semantic search is contextual, which is why retrieval improved so sharply.