Building with LLMs · Chapter 20 of 24

Conversation memory

By the end of this chapter you can keep a long conversation going without it overflowing the model's window: trim the history to a token budget without breaking it, summarise the part you drop, and know what each choice makes the assistant forget. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Building with LLMs
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.

Why you need this

Chapter 2 showed that a model remembers nothing: a chatbot "remembers" because your code resends the whole conversation every turn. That works for five messages. SafarLine's customers write fifty. Yash opened his chat with his name and his booking, then asked question after question in long messages about a family wedding trip, and on turn 39 the assistant stopped answering:

openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's
maximum context length is 8192 tokens. However, your messages resulted in
8250 tokens. Please reduce the length of the messages.", 'type':
'invalid_request_error', 'param': 'messages', 'code':
'context_length_exceeded'}}

Harpreet has a second complaint. Every turn resends every earlier turn, so the requests grow: Yash's first was 44 prompt tokens, his tenth 1,960, his thirtieth 6,301. His 38 answered turns sent 152,833 tokens in all, about ₹30.57 at lab-chat's ₹200 per million, for one customer's chat. You need to decide what a conversation keeps, and that is a decision about what the assistant forgets.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    trim.py: trim to a budget

    Write trim(messages, max_tokens) in trim.py. It returns a new list that fits in max_tokens: keep the system messages at the start; drop the oldest other messages, one at a time, until it fits, but never the last message; then, while the first message after the system messages is not a user message, drop it too. If the system messages and the last message alone do not fit, raise ValueError. Do not change the list you are given.

    Count tokens the way the API does: 3 for the request, plus, for each message, 3 and its content's tokens with tiktoken's o200k_base encoding.

  2. Task 2

    chat.py: a conversation that never overflows

    Write the Chat class in chat.py. Chat(system, max_tokens) starts self.history as a list holding the system message. ask(text) adds the user's message to self.history, sends lab-chat the history trimmed to max_tokens (task 1's trim), adds the model's reply to self.history as an assistant message, and returns the reply's text.

    self.history keeps the whole conversation; only what is sent is trimmed. The check has a long conversation with it, with a small budget and a large one.

  3. Task 3

    compact.py: summarise the old part

    Write compact(messages, keep=4) in compact.py. messages starts with one system message; everything after it except the last keep messages is old. Write the old messages as a transcript, one line per message: its role, a colon, a space and its content. Ask lab-chat in one request to summarise it, with the system message: Summarise this conversation in 50 words. and the transcript as the user message.

    Return a new list: the system message, then a system message holding Earlier in this conversation: followed by a space and the summary, then the last keep messages. With nothing old to summarise, return a copy of messages and send no request.

  4. Task 4 · interview question

    Interview: fix_trim.py, a trim that breaks the conversation

    recent(messages, n) in fix_trim.py should keep a conversation short, but the API refuses what it returns. Run the file and read the error.

    Fix recent so that it returns the system message (the first message), always, then the longest run of the latest messages, at most n of them, that starts with a user message. Do not change CONVERSATION; the check also tries other conversations and other values of n.

What you will learn

  • Why history grows, and what it costs.
  • Trimming to a token budget: what to keep, what to drop, in what order.
  • Not breaking the conversation: the system message, user turns, tool calls.
  • Summarising the old part (compaction), and what a summary loses.

More in Building with LLMs

Build "Conversation memory" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.