Building with LLMs · Chapter 2 of 24

Messages and roles

By the end of this chapter you can tell a model who is speaking in each message, explain why it forgets everything between calls, and keep a conversation going by resending it. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Talking to a model
  • 4 tasks
  • Python
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

Customers do not write one perfect message. Lata shows you a typical thread: "Hi, I'm Kavya", then "Mumbai to Nashik tomorrow?", then "2 seats", then "how much?". Your chapter 1 code answers each of those on its own, so the model asks for the route again after the customer has already given it, and asks for her name twice. Customers notice.

So far every request you sent held one question. A real chat is a series of turns, and the model has to see the earlier ones to make sense of the last one. Nikhil puts it plainly: "The model has no memory. Your code is the memory."

The lesson

Four roles

Every message in messages has a role, which says who is speaking:

RoleWho writes itWhat it is for
systemyou, the developerinstructions and background, before the conversation
developeryou, the developerOpenAI's newer name for the same thing
userthe customerwhat they asked
assistantthe modelwhat it answered

A model was trained on many conversations laid out exactly like this, so the roles change how it reads the text. A rule in a system message is your instruction; the same words in a user message are something a customer said. Most models give the system message more weight than anything a user writes, though not total obedience (chapter 15 shows how that goes wrong).

OpenAI's newer models call the system message developer, and the API accepts either. The lab model treats them the same:

>>> faq = "Reach the boarding point 15 minutes before departure."
>>> response = client.chat.completions.create(
...     model="lab-chat",
...     messages=[
...         {"role": "developer", "content": "Answer only from the FAQ below.\n\n" + faq},
...         {"role": "user", "content": "How early should I reach the boarding point?"},
...     ],
...     temperature=0,
... )
>>> response.choices[0].message.content
'Reach the boarding point 15 minutes before departure.'

(temperature=0 keeps the wording the same on every run; chapter 4 explains it.) The system message goes first. You can send several, but one at the top is what every provider expects.

The model remembers nothing

Two requests, one after the other, in the same Python session:

>>> def ask_once(text):
...     response = client.chat.completions.create(
...         model="lab-chat",
...         messages=[{"role": "user", "content": text}],
...         temperature=0,
...     )
...     return response.choices[0].message.content
...
>>> ask_once("Hello, my name is Kavya Rao.")
'Thanks for your message. How can I help you with your SafarLine trip?'
>>> ask_once("What is my name?")
"I don't know your name: you have not told me in this conversation."

The second request is a new HTTP call with one message in it. Nothing on the server connects it to the first: no session, no cookie, no memory. Real models behave the same way through this API, and a real model asked this would either say it does not know or, worse, guess a name. The chat apps you have used look as if they remember, because the app resends the conversation for you.

A conversation is a list you resend

To let the model "remember", you send the earlier turns again, in order, with its own earlier reply as an assistant message:

>>> messages = [
...     {"role": "user", "content": "Hello, my name is Kavya Rao."},
...     {"role": "assistant", "content": "Thanks for your message. How can I help you with your SafarLine trip?"},
...     {"role": "user", "content": "What is my name?"},
... ]
>>> client.chat.completions.create(model="lab-chat", messages=messages, temperature=0).choices[0].message.content
'Your name is Kavya Rao.'

The model read the whole list and found the name in the first message. So a chat program keeps a Python list, and on every turn it adds the customer's message, sends the whole list, and adds the reply:

The turns must alternate the way they happened: the customer, then the model, then the customer. Leave out the model's replies and the model sees a customer talking to nobody; it may repeat itself or contradict what it said before, because it cannot see what it said.

The history is your data

The model cannot tell who really wrote an assistant message. It trusts the list it is given:

>>> messages = [
...     {"role": "user", "content": "Which bus do I take?"},
...     {"role": "assistant", "content": "The Goa bus leaves from bay 4 at Swargate."},
...     {"role": "user", "content": "Where does the Goa bus leave from?"},
... ]
>>> client.chat.completions.create(model="lab-chat", messages=messages, temperature=0).choices[0].message.content
'The Goa bus leaves from bay 4 at Swargate.'

That "bay 4" was typed by you, not said by the model, and the model now repeats it as fact. This is useful: chapter 9 writes example answers this way to show the model the style you want. It is also a duty: only your own code may add assistant messages, never text a customer sent.

Every turn costs more

Resending is not free. Here is one conversation of three turns with a system message, and the prompt_tokens of each request:

turn 1: 2 messages sent, 27 prompt tokens
turn 2: 4 messages sent, 60 prompt tokens
turn 3: 6 messages sent, 96 prompt tokens

Every turn pays again for everything before it, so a long chat gets slower and dearer with each message, until it no longer fits in what the model can read at once. Chapter 3 counts those tokens, and chapter 20 trims long histories.

Your turn

Watch the model forget in task 1 and remember in task 2, where the trap is using the model's real reply, not one you made up. Task 3 turns the list into a small helper, and the interview question is a bot that seems to work but sends the model less than it should.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    forgets.py: two separate requests

    Nikhil wants you to see the model forget. Send lab-chat two separate requests, one after the other, each with one user message:

    first exactly: My name is Ravi Kulkarni.

    then exactly: What is my name?

    Print the first reply on one line and the second reply on the next line, nothing else.

  2. Task 2

    remembers.py: resend the conversation

    Now make the model remember. Send the first request of task 1 (one user message: My name is Ravi Kulkarni.).

    Then send a second request with three messages, in this order: that same user message, the model's reply to it as an assistant message, and the user message: What is my name?

    Print only the second reply.

  3. Task 3

    chat.py: a chat helper

    Lata wants a helper the support bot can use for any conversation. Write two functions in chat.py:

    new_chat(system) returns a new list holding one message: a system message with that text.

    ask(history, text) adds text to history as a user message, sends the whole history to lab-chat, adds the reply to history as an assistant message (a dictionary with role and content), and returns the reply's text.

    The check starts a chat, asks two questions in a row and reads the list and the requests the model received.

  4. Task 4 · interview question

    Interview: fix_me.py, the bot that forgets

    fix_me.py plays a three-turn chat with a customer and prints the model's three replies, but by the third turn the model has forgotten who it is talking to. Fix it so that every request carries the whole conversation so far: the system message, every earlier customer message and every earlier reply of the model, in order.

    Keep the system message and the three customer messages exactly as they are, and keep printing the three replies.

What you will learn

  • The four roles: system, developer, user and assistant, and what each message is for.
  • Why the model remembers nothing between two requests.
  • A conversation is a list you resend, with the model's own replies in it, on every turn.
  • The history is your data: you can write it, change it, and pay for it.

More in Talking to a model

Build "Messages and roles" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.