The tokenizer converts text into token IDs and back again.
Byte Pair Encoding repeatedly merges the most frequent character pairs.
The result is a fixed vocabulary, often 50,000 to 200,000 entries.
Every model is locked to the exact tokenizer it was trained with.
Tokenizer quirks explain why models struggle to count letters in a word.
The tokenizer is the piece of software that converts text to token IDs and back.
It is not the model, and it is trained separately, before the model exists.
Byte Pair Encoding is the usual algorithm, and it is simpler than it sounds.
Start with every character as its own token.
Find the most frequent adjacent pair in a large corpus and merge it into one new token.
Repeat that merge tens of thousands of times, recording the order.
Frequent sequences become single tokens and rare ones stay as small pieces.
The finished vocabulary usually holds 50,000 to 200,000 entries.
Modern tokenizers work over bytes, so no input is ever unrepresentable.
That means there is no unknown token, and any string round-trips exactly.
A model is welded to its tokenizer, since token ID 4,271 means whatever it meant in training.
Feeding a model text tokenized by a different tokenizer produces fluent nonsense.
Vocabulary choice has real consequences for languages under-represented in the training corpus.
The same sentence in some languages costs three times the tokens, and three times the money.
Most tokenizer questions in practice are cost questions in disguise.
One question on this card. Every option explains itself once you pick, right or wrong.
22 concepts in this module, then a graded test with every answer explained.