Longer is not always better

It is easy to take the wrong lesson from the six parts: that the longer the request, the better the answer. Researchers have tested one kind of length, the amount of material placed in front of the model, and the idea doesn’t hold. In one study, models answered questions with the help of documents a search system had found. Handing a model fifty documents instead of twenty improved its answers only a little, while making the input much longer. More material may help, the authors wrote, but it also gives the model more to work through, “potentially decreasing accuracy.”¹

Where the material sits matters too. In the same study, models were often best at using information at the very beginning or the very end of a long input, and much worse when it sat in the middle, even models built to handle long inputs.¹ Putting the question both before and after the documents gave near-perfect results on one of the study’s tests, a simple lookup task, but hardly changed results on the other.¹ The study tested the models of 2023, and newer ones may do better; the advice that follows costs nothing either way.

So the fix isn’t “keep it short.” It is to include what the model would otherwise guess, leave out what it doesn’t need, and put the key instruction at the start or the end, not in the middle of pasted material. If you paste a long report, say what you want before it, and say it again after. Chapter 2 pictured the model’s working space, its context window, as a desk measured in tokens. A bigger pile on the desk is no help if the one page that matters is buried in the middle of it.

Why examples work

Of the six parts, the example can do the most work in the fewest words. A model writes by continuing the text in front of it, predicting each next piece from what came before. Put a sample of what you want in front of it, and the likely continuation follows the sample’s pattern: its tone, its length, its layout. Researchers call this few-shot learning, a “shot” being one example. In 2020, the paper that introduced GPT-3 showed that the model could do many tasks it hadn’t been specially trained for from just a few examples written into the prompt, with no retraining, though it still struggled on some.²

An example shows what is hard to describe. “Friendly but professional” means different things in a law firm and a youth soccer league; last year’s email shows which one you mean. Pick an example you would be glad to see imitated, since imitation is the point, and say what to take from it: “match the tone, not the length.”

A related method helps with problems that take several steps. In a 2022 study, researchers showed models a few worked examples that wrote out their reasoning step by step, so the model wrote out its own steps before it answered. That improved results on some math, commonsense and symbolic problems, with the gains largest in the biggest models; for math, the researchers found it helped only in models of about 100 billion parameters.³ One way to see why, in Chapter 2’s terms: each step the model writes goes back onto its desk and becomes part of what it predicts the next words from. When you want to follow its reasoning, ask it to show the steps.