A token is the unit of text a language model reads and writes.
A token is usually a word fragment rather than a whole word.
In English roughly four characters make one token on average.
Models are billed and limited by tokens, never by characters or words.
Common words are single tokens while rare ones split into several pieces.
A token is the unit a language model reads and writes, and it is rarely a whole word.
The text is cut into pieces from a fixed vocabulary before the model sees anything.
In English a token averages about four characters, so 100 tokens is roughly 75 words.
Common words are a single token, and rare ones are assembled from several.
A leading space is usually part of the token, so cat and space-cat are different entries.
Numbers often split in unhelpful places, which is one reason arithmetic goes wrong.
Code, punctuation and non-English text usually cost more tokens for the same meaning.
Languages written in other scripts can cost several times more tokens per sentence.
That has a real price, since APIs bill per token in and per token out.
Context limits are counted in tokens too, not in words or characters.
Your prompt, the history, any retrieved documents and the reply all share that budget.
Output tokens usually cost several times more than input tokens.
Trimming a prompt by half genuinely halves that part of the bill.
The model never sees letters, only token identities, which is why counting the r in strawberry is hard.
Every provider offers a tokenizer so you can count before you send.
One question on this card. Every option explains itself once you pick, right or wrong.
22 concepts in this module, then a graded test with every answer explained.