Chapter 3Shannon's bits to Claude's tokens
In Shannon’s own words
Shannon wrote and spoke carefully, and a few of his lines read differently now. The first opens his 1948 paper.
The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point.
That is his definition of the problem his theory solves. A language model is, in a sense, an engine for the “approximately” half.
He was not sure statistics could go all the way. In March 1950 he spoke on the redundancy of English at a conference on cybernetics. In the discussion afterward, the conference’s chairman, Warren McCulloch, asked whether anything could handle units as large as phrases and sentences. Shannon said his 100-letter tests were “beginning to bite into phrases,” and then voiced a doubt.2
I feel there is a point at which this statistical approach is going to break down.
Today’s models test that doubt at a scale he never saw. By 1956 information theory had become, in his words, “something of a scientific bandwagon,” applied to fields from biology to economics.3 He warned against overselling it.
As a consequence, it has perhaps been ballooned to an importance beyond its actual accomplishments.
Readers may find the warning fits much of what is said about AI. In 1989 the journalist John Horgan asked Shannon whether machines could think. “You bet,” he said. “I’m a machine and you’re a machine, and we both think, don’t we?” Computers, Shannon told him, were still “not up to the human level yet.” Then Shannon went further.
It is certainly plausible to me that in a few decades machines will be beyond humans.
That is one pioneer’s view in 1989, not a forecast this book can check. What can be checked is the scale of the change since his experiments.
The same experiment at a different scale
Put Shannon’s game beside a modern model and the idea is the same. The scale is not. His subject predicted the next letter from up to 100 letters of the text before it.5 As of 2026, Anthropic’s documentation listed context windows of up to a million tokens for its Claude models.6 That is at least ten thousand times as much text, and Figure 3.2 draws it to scale.
The context window is the desk of Chapter 2: everything the model can take into account when it guesses the next token. Spot it today: the documentation for any AI model lists its context window, counted in tokens. The number differs from model to model, but it measures the same thing Shannon’s 100 letters did.
Your turn
Try it. Play Shannon’s game against a machine. With any AI chat tool, type the first half of a sentence of your own that has never been published, such as a line from a note to a colleague. Before you send it, write down your five best guesses for the next word. Then ask the tool for the five words it thinks most likely, and compare the lists. On paper, without a tool: play the guessing game from the start of this chapter twice, once on a routine work email and once on a poem. Score each as Shannon did, letters right on the first guess out of the total. Does the poem score lower, as Shannon found?
Think about it. Think of the writing you do in a week. Which parts could a colleague finish after the first few words, and which carry something only you know? Where does the information in your work actually sit?
So the idea behind today’s AI tools is not new. Treating language as guessing what comes next, measuring it in bits, and linking good prediction to good compression all go back to Shannon’s papers of 1948 and 1951. What took decades was building machines that could learn to make the guesses. How did people get from a person guessing letters to programs that write? The next chapter tells that story. It starts with a famous program from 1966 that did not predict at all.7
Check in
Try answering in your own words, then open each one to compare.read the answer to compare.
What did Shannon’s letter-guessing game measure, and what did he find?
How predictable English is. With up to 100 letters of context, his subject’s guesses put English at roughly one bit per letter, so most of each letter is already implied by what came before.
Why is a good predictor also a good compressor?
If the receiver has the same predictor, the sender only needs to send what the predictor gets wrong, as in Shannon’s reduced text. The better the guesses, the less there is to send.
Chapter 2 split a sentence into tokens, word pieces such as Gate and way. Where did that way of cutting up text come from?
From compression. Byte pair encoding was a 1994 method for shrinking files by replacing common pairs of bytes; in 2016 it was adapted to split words into pieces, and GPT-2 used a byte-level version.
A colleague says, “Shannon invented ChatGPT back in 1948.” What would you tell them?
Not quite. Shannon supplied the ideas: language as prediction, the bit as a unit, and the link to compression. Programs that learn their own predictions from text came decades later, such as a neural next-word predictor in 2003 and GPT-3 in 2020.
A sales brochure says a new model “has finally solved language.” Using what this chapter showed, how much weight would you give that claim?
Little, as stated. The 2020 scaling study found that improvement must level off, because language has non-zero entropy: some of what comes next can’t be predicted. Shannon himself warned that his theory had perhaps been “ballooned” beyond its accomplishments. Ask what was measured, and against what.
Notes (7)
- 1C. E. Shannon, “A Mathematical Theory of Communication,” Bell System Technical Journal 27, no. 3 (July 1948): 379–423; 27, no. 4 (October 1948): 623–656, https://doi.org/10.1002/j.1538-7305.1948.tb01338.x. Cited at page 379.
- 2Claude E. Shannon, “The Redundancy of English,” in Cybernetics: Circular Causal and Feedback Mechanisms in Biological and Social Systems. Transactions of the Seventh Conference, March 23–24, 1950, New York, N.Y. (New York: Josiah Macy, Jr. Foundation, 1951), 123–158, https://archive.org/details/cybernetics0000hein. Cited at page 139.
- 3Claude E. Shannon, “The Bandwagon,” IRE Transactions on Information Theory 2, no. 1 (March 1956): 3, https://doi.org/10.1109/TIT.1956.1056774. Cited at page 3.
- 4John Horgan, “Profile of Claude Shannon, Inventor of Information Theory,” Scientific American (Cross-Check blog), July 26, 2017, https://www.scientificamerican.com/blog/cross-check/profile-of-claude-shannon-inventor-of-information-theory/ (accessed October 26, 2026).
- 5C. E. Shannon, “Prediction and Entropy of Printed English,” Bell System Technical Journal 30, no. 1 (January 1951): 50–64, https://doi.org/10.1002/j.1538-7305.1951.tb01366.x. Cited at page 57.
- 6Anthropic, “Context windows,” Claude Platform Docs, n.d., https://platform.claude.com/docs/en/build-with-claude/context-windows (accessed October 26, 2026).
- 7Joseph Weizenbaum, “ELIZA—A Computer Program for the Study of Natural Language Communication Between Man and Machine,” Communications of the ACM 9, no. 1 (January 1966): 36–45, https://doi.org/10.1145/365153.365168.
Chapter 3 · page 4 of 4
Turn pages with the ← and → keys.