Chapter 3Shannon's bits to Claude's tokens
Shannon's bits to Claude's tokens
In 1948, Claude Shannon generated English-like text by hand, picking each letter or word according to how often it follows the ones before.1 Today's large language models make the same kind of prediction, learned from vast amounts of text.2
Specimens excerpted from Shannon's "A Mathematical Theory of Communication" (1948). He made the first two with a book of random numbers and a table of letter frequencies, and the rest by opening printed books at random, one letter or word at a time.1
The core idea is over 75 years old: guess the next symbol.1 What changed is how computers make that guess and how much they've read. Each era below added one new ability. The newest era isn't about writing better; it's about doing things.
Retro LabFrom Shannon's bits to Claude's tokensThe first chapter of this story: Shannon's 1948 specimens, side by side with a modern model. Each thread, step by step, with Shannon's own words.
Read the full story →Go deeper
Shannon did not invent language models. He supplied the ideas they are built and measured on. Three threads run from his papers to today:
- Prediction. Language can be modeled as guessing what comes next.
- Measurement. Information has a unit, and predictability can be counted in it.
- Compression. A better predictor is a better compressor, and the reverse.
Why "Claude"? The New York Times reported in 2023 that, depending on which Anthropic employee you ask, the name is a tribute to Shannon or simply a friendly name;4 The New Yorker in 2026 called the Shannon link "company lore."5 The science connecting the two, shown here, does not depend on the name.
Notes (5)
- 1C. E. Shannon, “A Mathematical Theory of Communication,” Bell System Technical Journal 27, no. 3 (July 1948): 379–423; 27, no. 4 (October 1948): 623–656, https://doi.org/10.1002/j.1538-7305.1948.tb01338.x. Cited at pages 388–389.
- 2Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al., “Language Models are Few-Shot Learners” (preprint, arXiv, May 28, 2020), https://doi.org/10.48550/arXiv.2005.14165.
- 3John Horgan, “Profile of Claude Shannon, Inventor of Information Theory,” Scientific American (Cross-Check blog), July 26, 2017, https://www.scientificamerican.com/blog/cross-check/profile-of-claude-shannon-inventor-of-information-theory/ (accessed October 26, 2026).
- 4Kevin Roose, “Inside the White-Hot Center of A.I. Doomerism,” The New York Times, July 11, 2023, https://www.nytimes.com/2023/07/11/technology/anthropic-ai-claude-chatbot.html (accessed October 26, 2026).
- 5Gideon Lewis-Kraus, “What Is Claude? Anthropic Doesn't Know, Either,” The New Yorker, February 9, 2026, https://www.newyorker.com/magazine/2026/02/16/what-is-claude-anthropic-doesnt-know-either (accessed October 26, 2026).
Chapter 3 · page 1 of 1
Turn pages with the ← and → keys.