Why Large Models Miscount the R’s in “Strawberry”
Models do not see letters the way people do. They see token IDs — and that design choice explains a classic “counting” failure.
Ask an early large language model how many letter R’s appear in the word strawberry, and it often gets the answer wrong. The easy conclusion is that models cannot count. The more useful conclusion is that they were never looking at letters in the first place.
What the model receives first
When you type a sentence and hit send, the model does not read your characters directly. A small program called a tokenizer turns the string into a sequence of integers. Each integer points to one entry in a fixed vocabulary — often tens of thousands to hundreds of thousands of entries. That conversion step is tokenization.
So when you see seven Chinese characters for “how is the weather today,” the model may see a short list of IDs. From that point on, almost every calculation works on those IDs and the vectors behind them — not on the ink of the original letters.
Why not one token per whole word?
A vocabulary of complete words would grow enormous, and any new word never seen in training would be a hard failure. A vocabulary of single characters stays small, but then the model must relearn spelling patterns from scratch, which is computationally inefficient. Subword tokenization sits in the middle: common fragments become one token; rare or new words are pieced together from smaller chunks.
In that world, a word like tokenization might split into token and ization; running might become run and ning. The model is good at predicting the next chunk. It is not natively inspecting every letter inside a chunk.
The strawberry lesson
That is why the strawberry puzzle is less mysterious than it looks. Humans naturally break the word into letters. The model often sees a handful of subword units whose IDs happen to spell a word people analyze letter by letter. Miscounting R’s is not proof that the model cannot do arithmetic in general; it is a reminder that the interface between text and the model is lossy for letter-level tasks.
- Different model families use different tokenizers — for example BPE-style systems in the GPT line, and SentencePiece-style systems in many LLaMA-family models.
- Fewer tokens for the same sentence usually means less compute; tokenizer choice also affects multilingual coverage.
- The basic shape stays the same: text in, integers out.
Once you see tokenization as an engineering trade-off rather than a magic trick, many odd model behaviors become easier to place. The model is not reading the page the way you do. It is reading a compressed code that was optimized for training efficiency — and that code does not always preserve the letter-level view people take for granted.