Chapter 3Shannon’s bits to Claude’s tokens
The same experiment at a different scale
Put Shannon’s game beside a modern model and the idea is the same, but the scale is not. Shannon’s subject predicted the next letter from up to 100 letters of the text before it.1 But as of 2026, Anthropic’s documentation listed context windows of up to a million tokens for its Claude models.2 A million tokens is at least ten thousand times as much text, and Figure 3.2 draws it to scale.
The context window is the size of the desk in Chapter 2, meaning how much the model can take into account when it guesses the next token. Spot it today: The documentation for any AI model lists its context window, counted in tokens. The number differs from model to model, but it measures the same thing Shannon’s 100 letters did.
Your turn
Try it. Play Shannon’s game against a machine. With any AI chat tool, type the first half of a sentence of your own that has never been published, such as a line from a note to a colleague. Before you send it, write down your five best guesses for the next word. Then ask the tool for the five words it thinks most likely, and compare the lists. On paper, without a tool: Play the guessing game from the start of this chapter twice, once on a routine work email and once on a poem. Score each as Shannon did, letters right on the first guess out of the total. Does the poem score lower, as Shannon found?
Think about it. Think of the writing you do in a week. Which parts could a colleague finish after the first few words, and which carry something only you know? Where does the information in your work actually sit?
The game you played with a book and a card is older than any AI tool, and so are the ideas behind those tools. Those ideas, treating language as guessing what comes next, measuring it in bits, and linking good prediction to good compression, all go back to Shannon’s papers of 1948 and 1951. What took decades was building machines that could learn to make the guesses. How did people get from a person guessing letters to programs that write? The next chapter tells that story, and it starts with a famous program from 1966 that did not predict at all.3
Check in
Try answering in your own words, then open each one to compare.read the answer to compare.
What did Shannon’s letter-guessing game measure, and what did he find?
How much of English is predictable. With up to 100 letters of context, his subject’s guesses put English at roughly one bit per letter, so most of what each letter could carry is already implied by what came before.
Why is a good predictor also a good compressor?
If the receiver has the same predictor, the sender only needs to send what the predictor gets wrong, as in Shannon’s reduced text. The better the guesses, the less there is to send.
Chapter 2 split a sentence into tokens, word pieces such as Gate and way. Where did that way of cutting up text come from?
From compression. Byte pair encoding was a 1994 method for shrinking files by replacing common pairs of bytes, and in 2016 it was adapted to split words into pieces, and GPT-2 used a byte-level version.
A colleague says, “Shannon invented ChatGPT back in 1948.” What would you tell them?
Not quite. Shannon supplied the ideas: language as prediction, the bit as a unit, and the link to compression. Programs that learn their own predictions from text came decades later, such as a neural next-word predictor in 2003 and GPT-3 in 2020.
A sales brochure says a new model “has finally solved language.” Using what this chapter showed, how much weight would you give that claim?
Little, as stated. The 2020 scaling study found that improvement must level off, because language has nonzero entropy, so some of what comes next can’t be predicted. Shannon himself warned that his theory had perhaps been “ballooned” beyond its accomplishments. Ask what was measured, and against what.
Notes (3)
- 1C. E. Shannon, “Prediction and Entropy of Printed English,” Bell System Technical Journal 30, no. 1 (January 1951): 50–64, https://doi.org/10.1002/j.1538-7305.1951.tb01366.x. Cited at page 57.
- 2Anthropic, “Context windows,” Claude Platform Docs, n.d., https://platform.claude.com/docs/en/build-with-claude/context-windows (accessed October 26, 2026).
- 3Joseph Weizenbaum, “ELIZA—A Computer Program for the Study of Natural Language Communication Between Man and Machine,” Communications of the ACM 9, no. 1 (January 1966): 36–45, https://doi.org/10.1145/365153.365168.
Chapter 3 · page 6 of 6
Turn pages with the ← and → keys.