The Lab · Learn

How AI actually works

Small pages that run on this device. Read the plain version, move the sliders, watch what changes, then say what you saw. Tap any word you do not know.

Lesson 6

Tokens

AI does not read letters or whole words. It reads chunks, and the chunks it knows came from what it read most.

It is like: Like a speed reader who knows “the” and “ing” at a glance, but has to sound out a name they have never seen.
Why it matters at work: Names, codes and other languages cost more chunks, which is part of why they get mangled and why long documents hit limits sooner.

The exact idea

A model reads text as tokens, pieces learned by merging the letter pairs that appear most often.

Words on this page

Try this

Slide merges from 0 up to 10 and watch “lowest” go from six letters to two or three tokens.
Type “zebra” as the word to split. None of its letter pairs were merged, so every letter stays a token.
Add “slowest slowest slowest” to the training text and watch the merge list change.

Change it

In the Workshop, press Pick a part, then tap the token count and ask for a line that also shows how many tokens the whole training text takes.

Why does a rare word usually take more tokens than a common one?

Pick the one that fits what you saw.

Rare words are usually longer, so they need more pieces.Merges are learned from pairs that show up often, and a rare word's pairs were not seen enough to be merged.The model splits unfamiliar words on purpose so it can read them more carefully.Rare words use unusual letters, and each unusual letter is its own token.

Field notes

Reading the notes.