Chapter 3Shannon's bits to Claude's tokens
His groundwork, and his own words
Shannon's experiment and today's models make the same kind of guess at a vastly different scale, and his other work laid the ground the whole field stands on.
The same experiment, at a different scale
The groundwork beneath it all
- 1937Logic in circuits. His MIT master's thesis showed Boolean algebra can describe switching circuits.34 Every chip running a model today is, at bottom, a vast switching circuit.
- 1950Search and evaluation. "Programming a Computer for Playing Chess" laid out look-ahead search scored by an evaluation function.5
- 1950A machine that learns. Theseus, his relay-controlled mechanical mouse, found its way through a maze by trial and error and remembered the route.674
- 1955Proposing the field. Shannon co-authored the 1955 Dartmouth proposal, whose title used the term "artificial intelligence" and which set the goal of finding "how to make machines use language."8
In Shannon's words
"reproducing at one point either exactly or approximately a message selected at another point"
His definition of the problem. A language model is, in a sense, an engine for the "approximately" half.
"of the order of one bit per letter"
His estimate of how much new information each letter of English carries once context is known.
"a point at which this statistical approach is going to break down"
Shannon doubted statistics could capture the long-range structure of language. Modern LLMs test that doubt at a scale he never saw.
"it has perhaps been ballooned to an importance beyond its actual accomplishments"
Shannon cautioned against overselling his own theory. It remains good advice for how we talk about AI.
A Mind at Play. Jimmy Soni and Rob Goodman's biography, How Claude Shannon Invented the Information Age (New York: Simon & Schuster, 2017).4 The story behind everything on this page, from the thesis to the unicycles.
Check in
Try answering in your own words, then open each one to compare.
What do Shannon’s 1948 experiment and today’s models have in common?
Both guess the next symbol from the ones before it. What changed is how the guess is made and how much text is behind it.
Shannon’s idea was prediction. Not every famous chatbot predicted, though. Next, a short interlude with one that didn’t. On to Chapter 4: Interlude: meet ELIZA (1966) →
Notes (11)
- 1C. E. Shannon, “Prediction and Entropy of Printed English,” Bell System Technical Journal 30, no. 1 (January 1951): 50–64, https://doi.org/10.1002/j.1538-7305.1951.tb01366.x. Cited at pages 50, 57.
- 2Anthropic, “Context windows,” Claude Platform Docs, n.d., https://platform.claude.com/docs/en/build-with-claude/context-windows (accessed October 26, 2026).
- 3Claude E. Shannon, “A Symbolic Analysis of Relay and Switching Circuits,” Transactions of the American Institute of Electrical Engineers 57, no. 12 (December 1938): 713–723, https://doi.org/10.1109/T-AIEE.1938.5057767. Cited at page 713.
- 4Jimmy Soni and Rob Goodman, A Mind at Play: How Claude Shannon Invented the Information Age (New York: Simon & Schuster, 2017). Cited at pages 39, 203–204.
- 5Claude E. Shannon, “Programming a Computer for Playing Chess,” The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 41, no. 314 (March 1950): 256–275, https://doi.org/10.1080/14786445008521796. Cited at §§3–4.
- 6Claude E. Shannon, “Presentation of a Maze-Solving Machine,” in Cybernetics: Circular Causal and Feedback Mechanisms in Biological and Social Systems. Transactions of the Eighth Conference, March 15–16, 1951, New York, N.Y. (New York: Josiah Macy, Jr. Foundation, 1952), 173–180, https://archive.org/details/cybernetics0000hein_i4t4. Cited at page 173.
- 7Deborah Halber, “Shannon collection shows wit and whimsy,” MIT News, May 30, 2007, https://news.mit.edu/2007/shannon-0530 (accessed October 26, 2026).
- 8J. McCarthy, M. L. Minsky, N. Rochester, and C. E. Shannon, A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence (John McCarthy's archive, Stanford University, August 31, 1955), https://www-formal.stanford.edu/jmc/history/dartmouth/dartmouth.html (accessed October 26, 2026).
- 9C. E. Shannon, “A Mathematical Theory of Communication,” Bell System Technical Journal 27, no. 3 (July 1948): 379–423; 27, no. 4 (October 1948): 623–656, https://doi.org/10.1002/j.1538-7305.1948.tb01338.x. Cited at page 379.
- 10Claude E. Shannon, “The Redundancy of English,” in Cybernetics: Circular Causal and Feedback Mechanisms in Biological and Social Systems. Transactions of the Seventh Conference, March 23–24, 1950, New York, N.Y. (New York: Josiah Macy, Jr. Foundation, 1951), 123–158, https://archive.org/details/cybernetics0000hein. Cited at page 139.
- 11Claude E. Shannon, “The Bandwagon,” IRE Transactions on Information Theory 2, no. 1 (March 1956): 3, https://doi.org/10.1109/TIT.1956.1056774. Cited at page 3.
Chapter 3 · page 3 of 3
Turn pages with the ← and → keys.