The groundwork beneath it all

Prediction, measurement, and compression run straight to today’s models. Shannon’s other work lies underneath the whole field, and four pieces of it matter here.

The first is logic in circuits, from his master’s thesis at MIT, published as a paper in 1938. In it, Shannon showed that Boolean algebra, the algebra of true and false, can describe circuits of switches.¹² Every chip that runs a model today is, at bottom, a vast switching circuit.

The second is search, meaning trying out possible moves before choosing one, and in 1950 he laid out a plan for a computer to play chess that way. The plan looked ahead at possible moves and scored each resulting position with an evaluation function, a rule for judging how strong a position is.³ Chapter 1 used that plan as an example of AI built from rules people wrote by hand.

The third is a machine that learns from trying. That machine was Theseus, a mechanical mouse controlled by relays, which are electrical switches. The mouse found its way through a maze by trial and error and remembered the route.⁴² Shannon showed the mouse at a conference on cybernetics, the study of feedback and control in machines and living things, in New York in March 1951. According to MIT, he had built it in 1950.⁴⁵

The fourth is the field’s founding plan. That plan dates from 1955, when Shannon and three other scientists proposed a summer research project at Dartmouth College, and the proposal’s title used the term “artificial intelligence.” One of its goals was to find “how to make machines use language,”⁶ and that goal is the subject of this book.

In Shannon’s own words

Shannon wrote and spoke carefully, and a few of his lines read differently now, starting with the one that opens his 1948 paper.

The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point.

That is his definition of the problem his theory solves. A language model is, in a sense, an engine for the “approximately” half of that problem, with a twist. No one sends the model a message to reproduce, so instead it writes a likely one of its own.

Shannon himself was not sure that statistics like his could reach all the way up to phrases and sentences. In March 1950, a year before he showed his mouse, he spoke on the redundancy of English at a conference on cybernetics. In the discussion afterward, the conference’s chairman, Warren McCulloch, asked whether anything could handle units as large as phrases and sentences. Shannon said his 100-letter tests were “beginning to bite into phrases,” and then voiced a doubt.⁸

I feel there is a point at which this statistical approach is going to break down.

Today’s models put that doubt to the test. By 1956 he had a second doubt, because information theory, as his field had come to be called, had become “something of a scientific bandwagon,” in his words. That bandwagon reached fields from biology to economics,⁹ and he warned against overselling it.

As a consequence, it has perhaps been ballooned to an importance beyond its actual accomplishments.

You may find that the warning fits much of what is said about AI. Yet in 1989 the journalist John Horgan asked Shannon whether machines could think. “You bet,” he said. “I’m a machine and you’re a machine, and we both think, don’t we?” Computers, Shannon told him, were still “not up to the human level yet,” and then he went further.

It is certainly plausible to me that in a few decades machines will be beyond humans.

That is one pioneer’s view in 1989, not a forecast this book can check. What this book can check is the scale of the change since his experiments.