Everything the model reads

This chapter uses the word prompt in its full sense: everything the model reads for one reply. Your message is part of it, usually the last part. People who build these systems also call the whole of it the context. Anthropic defines context as “the set of tokens included when sampling from” a language model, which means every piece of text the model is given when it writes its reply.¹ Tokens, as Chapter 2 showed, are the word-pieces a model reads and writes.

Two parts from Chapter 1’s picture show up here before their own chapters. The harness is the app or software around the model; it puts the prompt together (Chapter 7). Tools are the actions the harness can take when the model asks, such as searching the web or reading a file (Chapter 8). For now, that is all you need to know about them.

A prompt is built in layers, and different people write different layers. Figure 6.1 shows a typical stack, top to bottom.

At the top sits the system prompt. In a company’s own chat app, the company that made the model writes it. In Anthropic’s case it supplies the date and nudges behavior, such as always putting computer code in a tidy format called Markdown.² Anthropic publishes the system prompts for its Claude apps, and notes that those prompts don’t apply when developers use its models directly, through what the industry calls an API.² Next come the app instructions, written by whoever built the app: a school, a bank, a software company. PS2 Pal’s persona was this layer. OpenAI’s rules name the same split: “system” messages added by OpenAI, “developer” messages from the app’s builder, and “user” messages from you.³

Below those sit the tool descriptions, a short note for each tool the app offers, saying what it does and how to ask for it. Anthropic says that when an app gives Claude tools through its API, the API “constructs a special system prompt from the tool definitions” and the app’s own instructions.⁴ The model never sees a tool itself. It sees only the description and, later, whatever the tool sends back.⁵

Then come memory and documents. A document might be a file you uploaded or pages a search found. In a 2020 study, a model that looked up relevant documents and read them along with the question gave more specific and factual answers than the same kind of model without that step.⁶ Memory is notes the harness saved earlier and feeds back in. As of 2026, for example, Anthropic says Claude can “remember context from your chats and carry it into new conversations”; a summary, updated every 24 hours, “provides context for every new standalone conversation.”⁷ Spot it today: look for a memory setting in your app, and see whether it lets you read what was saved.

Try it: read each layer of the stack, and who wrote it.

The conversation so far comes next. The model keeps nothing between replies. Each time, the harness sends the stack again, with the conversation included; Anthropic describes each new request as “containing the original messages” along with the model’s last reply and any new results.⁵ In a long chat, the harness may send a summary instead of every word.¹ Then come earlier tool results: what came back the last time a tool ran, as plain text. Last of all comes your message, the only layer you wrote.

Even the smallest piece of tool text counts. In a 2025 test of AI agents, a tool that ran code replied “Code executed successfully with no output” whenever a program printed nothing. Even strong models read “no output” as a sign that something was wrong. They tried again, and some decided the tool was broken and fell back on counting words by hand, which failed.⁸ The study’s lesson was that everything about a tool, down to “the messages it sends back,” is part of the prompt.⁸