Each token ID is looked up in an embedding table to get a vector.
That vector might have a few thousand dimensions.
The embedding table is learned during pretraining like any other weights.
These vectors are what actually flow through the transformer layers.
By the final layer each position carries a heavily contextualised meaning.
Token IDs are just numbers naming vocabulary entries, and nothing about ID 4,271 is meaningful.
The first thing the model does is trade each ID for a vector.
That lookup table is the embedding matrix, with one row per vocabulary entry.
A row might be 4,096 numbers wide in a large model.
Those numbers are learned during pretraining like every other weight.
With a 100,000-word vocabulary the table alone holds hundreds of millions of parameters.
After the lookup, position information is added so the model knows the order of the words.
These vectors are what actually flow through the transformer layers.
Each layer rewrites them, mixing in information from the other positions.
So a vector that started meaning bank ends up meaning the riverbank sense specifically.
That is the difference from older word vectors, which had one fixed vector per word.
Here every occurrence gets its own vector, shaped by its context.
The final layer maps back to vocabulary scores to choose the next token.
Many models tie the input and output tables together to save parameters.
The embeddings you buy from an embeddings API are a related but separate product for search.
One question on this card. Every option explains itself once you pick, right or wrong.
22 concepts in this module, then a graded test with every answer explained.