Tokenization splits text into the units a model actually processes.
Word-level tokens are intuitive but produce enormous vocabularies.
Character-level tokens keep the vocabulary tiny but make sequences very long.
Subword methods like BPE and WordPiece sit in between and are what modern models use.
Subwords let a model represent a word it has never seen by combining known pieces.
A model cannot read text, so text has to be cut into units and mapped to numbers.
Those units are tokens, and how you cut decides much of what the model can do.
Splitting on spaces gives whole words, which is the intuitive choice.
It produces a vocabulary of hundreds of thousands of entries and still misses new ones.
Any word the vocabulary lacks becomes an unknown token and its meaning is simply lost.
Splitting into characters fixes that completely, since every word is spellable.
But sequences become five times longer and the model has to learn spelling before meaning.
Subword tokenization sits in the middle and is what modern models use.
Common words stay whole and rare ones break into pieces that already exist.
Tokenization is the word unbreakable might become un, break and able.
Byte Pair Encoding builds this vocabulary by repeatedly merging the most frequent pair.
WordPiece and SentencePiece are variations on the same idea with different merge rules.
A vocabulary of 50,000 to 200,000 subwords covers essentially all text with no unknowns.
Every model is locked to the exact tokenizer it was trained with, and swapping it is fatal.
Counting letters in a word is hard for models because they never saw the letters separately.
A vocabulary of whole words cannot be finished. New words, names, typos and inflections arrive forever, and every one of them becomes an unknown token the model cannot read.
Characters solve that completely and make the sequences several times longer, which costs quadratic attention time for no gain in meaning.
Subword tokenisation is the compromise: common words stay whole, rare words break into pieces, and nothing is ever unknown.
The merge list is the tokeniser. Encoding a new string means applying the same merges in the same order, which is deterministic and fast.
Starting from bytes rather than characters is what guarantees no unknown token: any input, in any language or encoding, is some sequence of 256 possible bytes.
That ratio is the one to remember for cost and context estimates, and it holds only for English.
Hindi, Thai and Tamil take 2 to 4 times more tokens per word.
A tokeniser trained mostly on English spends more tokens per word on everything else, so the same sentence costs several times as much to process in another script.
That is a fairness and a pricing problem at once, and recent tokenisers with larger vocabularies exist largely to reduce it.
The leading space is part of the token, which is why a prompt ending in a space produces noticeably worse continuations: it forces the model onto the rarer no-space variants.
Numbers are split by whatever the merge list happened to learn, so 1234 may be one token or four, and arithmetic errors often trace back to that rather than to reasoning.
Ask a model to count the letters in "strawberry".
A model sees three tokens, not ten letters, so character-level questions are hard for a structural reason and not because the model is careless.
BPE merges the most frequent pair. WordPiece merges the pair that most increases the corpus likelihood, which is a slightly different criterion and a similar result.
Unigram works backwards: start with a large vocabulary and delete the tokens whose removal costs the least likelihood, which allows several valid segmentations of the same string.
SentencePiece treats the raw text as a stream including spaces, so it needs no language-specific pre-tokenisation and works on scripts that do not put spaces between words.
One question on this card. Every option explains itself once you pick, right or wrong.
20 concepts in this module, then a graded test with every answer explained.