Basic

How models read text: tokens

Models don't read whole words. They break text into chunks called tokens, pieces of words, half words, or sometimes an entire word. Tokens are also how cost is measured, which is why knowing them matters when you're actually using a paid model.

About 2 min of reading

Data Flow

Text splits into tokens, the model processes them, then it reassembles the output into readable text.
1 Split input into tokens
↓
2 Model reads tokens
↓
3 Model writes output tokens
↓
4 Join tokens into text
Why this works
Text becomes tokens

Before a model can read your prompt, it splits the text into tokens. One token is usually part of a word, sometimes a whole word, sometimes just a piece of punctuation. The split is automatic but not always tidy.

Tokens cost money

Most paid models charge per token. Longer prompts and longer answers both use more of them. Being concise saves both time and money, and nobody has to feel bad about the shorter reply.

Examples

Words broken into pieces

"unhappiness" becomes two tokens:
un + happiness

Smaller pieces are faster and cheaper to process. Long or unusual words simply make more tokens, which is why a dense document can quietly cost more than a light one.

Pieces that aren't really words

"running" might split into:
run + ning

Tokens are chunks of characters, not fixed words, and the same word can split differently depending on how the model was trained. You can't predict the exact count, but you can expect rough math to hold.

Counting the cost

A 100-token prompt might cost a few cents.
10,000 tokens costs about 100 times more.

Most paid models charge per token, so being concise literally saves money. A tight, well-worded prompt will cost less than rambling the same idea three times over.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← What a language model is (and isn't)
Previous Lesson
What a language model is (and isn't)
Next Lesson
The context window
The context window →