steve sparks
9 of 58
ContentsPart II · Chapter 3

Chapter 3Shannon's bits to Claude's tokens

Three threads from 1948 to today

Shannon did not invent language models. He supplied the ideas they are built and measured on. Three threads run from his papers, through later work, into how today's models are trained.

The transformer, the design today's models are built on, gets its place in the story in Chapter 5. Here it is one step in the prediction thread.

Prediction

Language can be modeled as guessing what comes next.

  1. 1948Shannon generates English-like text from letter and word statistics, and concludes that a complex enough random process can represent a language source.1
  2. 1951In "Prediction and Entropy of Printed English," a test subject guesses a passage letter by letter, and the guesses measure how predictable English is.2
  3. 2003Bengio and colleagues train a neural network to predict the next word, learning word embeddings (numbers that stand for words) along the way.3
  4. 2017The Transformer's attention mechanism trains far more in parallel than earlier networks,4 and transformers improved predictably as they grew.5
  5. TodayLanguage models write each response one token at a time, sampling from a learned probability distribution over their vocabulary.67

Measurement

Information has a unit, and predictability can be counted in it.

  1. 1948Shannon defines entropy and names its unit the bit, a word he credits to John Tukey.18
  2. 1951With up to 100 letters of context, English carries roughly one bit per letter, within Shannon's estimated bounds of about 0.6 to 1.3.2
  3. 2003Researchers judge language models by perplexity, built from how much probability a model gives the real text: in plain words, how surprised it is, so lower is better.3
  4. 2020Scaling-laws research, four of whose ten authors later co-founded Anthropic, shows cross-entropy loss falling predictably with model size, data and compute, and notes the trend must level off because language itself has non-zero entropy.59
  5. TodayPretraining minimizes cross-entropy loss: how surprised the model is by the actual next token. Kaplan and colleagues measured it in nats, the natural units of information that Shannon's 1948 paper also describes.51

Compression

A better predictor is a better compressor, and the reverse.

  1. 1948The source coding theorem: no code can average fewer bits per symbol than the source's entropy, and good codes can come arbitrarily close.1
  2. 1994Philip Gage's Byte Pair Encoding compresses files by repeatedly replacing the most common pair of bytes with a single unused byte.10
  3. 2006The Hutter Prize rewards compressing Wikipedia, treating compression as a measurable stand-in for intelligence.11
  4. 2016Sennrich and colleagues adapt BPE to split words into subword tokens.12 GPT-2 used a byte-level version of BPE.7
  5. 2023Google DeepMind's "Language Modeling Is Compression" shows a large language model, DeepMind's 70-billion-parameter Chinchilla, trained mostly on text, compressing image patches to 43% and audio to 16% of their raw size, better than PNG (58%) and FLAC (30%), not counting the size of the model itself.13

Shannon's own work Later research Today's models

Notes (13)
  1. 1C. E. Shannon, “A Mathematical Theory of Communication,” Bell System Technical Journal 27, no. 3 (July 1948): 379–423; 27, no. 4 (October 1948): 623–656, https://doi.org/10.1002/j.1538-7305.1948.tb01338.x. Cited at pages 380, 389, 392–393, 401–403.
  2. 2C. E. Shannon, “Prediction and Entropy of Printed English,” Bell System Technical Journal 30, no. 1 (January 1951): 50–64, https://doi.org/10.1002/j.1538-7305.1951.tb01366.x. Cited at pages 50, 54–57, 64.
  3. 3Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin, “A Neural Probabilistic Language Model,” Journal of Machine Learning Research 3 (February 2003): 1137–1155, https://www.jmlr.org/papers/v3/bengio03a.html. Cited at pages 1137, 1141.
  4. 4Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention Is All You Need” (preprint, arXiv, June 12, 2017), https://doi.org/10.48550/arXiv.1706.03762.
  5. 5Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei, “Scaling Laws for Neural Language Models” (preprint, arXiv, January 23, 2020), https://doi.org/10.48550/arXiv.2001.08361. Cited at pages 1, 6, 9, 17.
  6. 6Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi, “The Curious Case of Neural Text Degeneration” (preprint, arXiv, April 22, 2019), https://doi.org/10.48550/arXiv.1904.09751.
  7. 7Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever, Language Models are Unsupervised Multitask Learners (San Francisco: OpenAI, February 2019), https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf (accessed October 26, 2026). Cited at page 4.
  8. 8Jimmy Soni and Rob Goodman, A Mind at Play: How Claude Shannon Invented the Information Age (New York: Simon & Schuster, 2017). Cited at page 141.
  9. 9Anthropic, “Leadership at Anthropic,” n.d., https://www.anthropic.com/company/leadership (accessed October 26, 2026).
  10. 10Philip Gage, “A New Algorithm for Data Compression,” The C Users Journal 12, no. 2 (February 1994): 23–38, https://jacobfilipp.com/DrDobbs/articles/CUJ/1994/9402/gage/gage.htm.
  11. 11Marcus Hutter, “Hutter Prize for Lossless Compression of Human Knowledge,” prize.hutter1.net, 2006 (updated 2020), http://prize.hutter1.net/ (accessed October 26, 2026).
  12. 12Rico Sennrich, Barry Haddow, and Alexandra Birch, “Neural Machine Translation of Rare Words with Subword Units,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2016), 1715–1725, https://doi.org/10.18653/v1/P16-1162. Cited at pages 1715–1717.
  13. 13Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness, “Language Modeling Is Compression,” in International Conference on Learning Representations (ICLR 2024) (2024), https://doi.org/10.48550/arXiv.2309.10668.

Chapter 3 · page 2 of 3

Turn pages with the ← and → keys.

Cite this page

https://aifromthepartsup.com/shannons-bits-to-claudes-tokens/2/

Using this book

© 2026 Steve Sparks. All rights reserved. You're welcome to quote short passages and cite any page, with a link back. To reuse diagrams or larger parts, ask through GitHub.

Quoted material and figures from other sources remain their owners'.