Tokens
A language model does not read whole words. It reads tokens — chunks of text that are often whole words, but sometimes just pieces of them. Everything a model does works in tokens.
Pick a sentence and watch it split into coloured tokens. Common words stay whole; rarer ones break into parts. The count on the right is what the model actually processes.
Each coloured chunk is one token. Common words stay whole, but rarer ones break into pieces. The ␣ marks a leading space — the model treats “cat” and “ cat” as different tokens.
How it works
Tokenization is the first step of reading text. Instead of whole words, models use subword tokens: frequent words become a single token, while rarer words split into smaller familiar pieces. This keeps the vocabulary manageable and lets the model handle words it never saw in training by assembling them from parts. Spaces and capitals matter, so cat, a space then cat, and Cat can each be different tokens, and punctuation is usually its own token. A rough rule for English is that one token is about three quarters of a word, or four characters, so a hundred words is roughly a hundred and thirty tokens. Tokens are the true unit everywhere: the Embeddings lesson gives each token a vector, the Attention lesson connects tokens, and the Language Models lesson predicts the next token. Context windows and API pricing are both measured in tokens for the same reason.
Check yourself
Why might the model break an unusual word into several tokens instead of keeping it whole?
Go deeper (free): Hugging Face LLM Course — Tokenizers ↗