Chapter 2How it writes
Your turn
Try it. With any AI chat tool you already use, open a new chat and ask for a one-sentence thank-you note to a coworker. Open a second new chat and ask exactly the same thing. Compare the two notes: any difference in wording is the weighted draw at work. Then open a third new chat and ask, “What did I ask you to write a few minutes ago?” If it knows, look in the app’s settings for a memory or chat-search feature; that is software putting notes on the desk. If it doesn’t, you have met the empty desk. On paper instead: take a sentence you often write at work, leave off its last word, and list five endings with a rough chance for each. Do your chances add up to 100?
Think about it. Think of one job you might hand to an AI tool every week. What would it need on its desk each time to do that job well, and who would put it there: you, or the software?
So when an AI tool writes you a reply, it splits the text into tokens, scores every possible next token, draws one and adds it, then does it all again. Its chances come from training that ended before you arrived, and the only things it can see are what is on its desk right now. The idea of writing by chance, one piece at a time, is older than any of these models. In his 1948 paper, Claude Shannon built English-looking text by choosing letters at random: first with equal chances, then weighted by how often each occurs in English, and, in later steps, by how often it follows the ones before.1 How did that idea grow into a way of measuring information itself, and what does it have to do with tokens? That is the next chapter.
Check in
Try answering in your own words, then open each one to compare.read the answer to compare.
What is a token, and why does this book keep counting in them?
A word-piece. Short, common words are often one token; longer ones are split into several. The model reads and writes tokens, its desk is measured in them, and use of a model is counted in them.
Why can the same question, asked twice, come back worded differently?
At each step the model scores every possible next token and then makes a weighted draw, so it doesn’t always take the top one. One different pick early on changes everything after it. Temperature sets how even the chances are.
A coworker says, “I told the AI tool about my client’s preferences last month, so it knows them now.” What would you tell them?
The model sees only what is on its desk for this reply, and its parameters didn’t change when they chatted. Unless the app saved notes and puts them back (a memory feature they can check in its settings), last month’s chat isn’t there. The safe move is to paste the key preferences in again.
Chapter 1 told of a lawyer whose AI tool assured him that court cases it had invented were real. In this chapter’s terms, what was on the desk, and what did each pick depend on?
His questions and the conversation so far were on the desk; no real court records were. Each token was a likely next piece given that text, so a reply saying the cases were real was a likely one. Nothing in predict, pick, repeat checks a reply against the world.
You have one question about a single section of a 300-page report. Would you paste in the whole report, or just that section? Why?
Usually just the section, with a line saying where it comes from. The whole report could crowd the desk, push other material off it, and leave the part you need buried in the middle, where models often use information less reliably. If your tool can fetch the relevant part of a document on its own, that does the same job.
End of Part I, The basics.Next: Chapter 3: Shannon's bits to Claude's tokens →
Notes (1)
- 1C. E. Shannon, “A Mathematical Theory of Communication,” Bell System Technical Journal 27, no. 3 (July 1948): 379–423; 27, no. 4 (October 1948): 623–656, https://doi.org/10.1002/j.1538-7305.1948.tb01338.x.
Chapter 2 · page 6 of 6
Turn pages with the ← and → keys.