Where good chances come from

How does the model come up with chances good enough to write a clear paragraph? Part of the answer is the design today’s models are built on, the transformer, introduced in a June 2017 paper called “Attention Is All You Need.”¹ Its key idea, attention, relates each token to the other tokens in the text, all at once, and weighs which ones matter for what it means.¹ When the model is writing, each new token can look back at every token before it.¹

Take the word bank. In “we fished from the river bank,” the words fished and river come before it; in “I deposited the check at the bank,” deposited and check do. (The example is this book’s, not the paper’s.) Attention lets the model weigh those earlier words, so the same token can be read one way in one sentence and another way in the next.

The rest of the answer is training, which happens in stages. In the first and biggest, called pretraining, the model plays the very game you played at the start of this chapter. It works through a huge amount of text and, at every point, predicts the next token.² Its internal settings, called parameters, are adjusted over and over so that its predictions improve.² GPT-3, a model from 2020, had 175 billion of them.³

A model fresh from pretraining is good at continuing text, which is not the same as being a good assistant. As the researchers behind one well-known coaching method put it, making language models bigger “does not inherently make them better at following a user’s intent.”⁴ So the model is coached in a second stage. In that method, people first wrote examples of good answers to prompts, and the model was trained on those examples. Then people ranked the model’s own answers from best to worst, and the model was trained further toward the kinds of answers people ranked higher.⁴

The coaching made a large difference. In the researchers’ tests, people preferred answers from a 1.3-billion-parameter model coached this way over answers from the 175-billion-parameter GPT-3, a model more than 100 times its size.⁴ Feedback can come from AI as well as from people. In a method Anthropic calls Constitutional AI, a model critiques and revises answers against a written list of principles, and a model, rather than a person, judges which of two answers is better.⁵

All of this happens before you use the model. Training adjusts the parameters and takes enormous computing power.⁶³ (What that power costs in energy is one of the questions Chapter 15 takes up.) Use is different, and it happens every time you send a message. The parameters don’t change:³ the model reads what is in front of it and predicts a reply.⁷ Whether your conversations are later used to train a future model is a separate question. It depends on your account and settings,⁸⁹ and Chapter 10 takes it up.

One way to hold all of this in mind is to picture an apprentice. The apprentice learned language by playing fill-in-the-blank on an enormous scale, then went to finishing school to learn how to help. Everything the apprentice knows was settled before you arrived. And the apprentice works at a desk that holds only so many pages at once.