Building with LLMs (TypeScript) · Chapter 20 of 24
Conversation memory
By the end of this chapter you can keep a long conversation going without it overflowing the model's window: trim the history to a token budget without breaking it, summarise the part you drop, and know what each choice makes the assistant forget. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Building with LLMs
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Chapter 2 showed that a model remembers nothing: a chatbot "remembers" because your code resends the whole conversation every turn. That works for five messages. SafarLine's customers write fifty. Yash opened his chat with his name and his booking, then asked question after question in long messages about a family wedding trip, and on turn 38 the assistant stopped answering:
BadRequestError: 400 This model's maximum context length is 8192 tokens. However, your messages resulted in 8429 tokens. Please reduce the length of the messages.
Harpreet has a second complaint. Every turn resends every earlier turn, so the requests grow: Yash's first was 41 prompt tokens, his tenth 1,667, his thirtieth 6,360. His 37 answered turns sent 139,538 tokens in all, about ₹27.91 at lab-chat's ₹200 per million, for one customer's chat. You need to decide what a conversation keeps, and that is a decision about what the assistant forgets.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
trim.ts: trim to a budget
Write and export the ChatMessage type and trim(messages: readonly ChatMessage[], maxTokens: number): ChatMessage[] in trim.ts. trim returns a new array that fits in maxTokens: keep the system messages at the start; drop the oldest other messages, one at a time, until it fits, but never the last message; then, while the first message after the system messages is not a user message, drop it too. If the system messages and the last message alone do not fit, throw an Error. Do not change the array you are given.
Count tokens the way the API does: 3 for the request, plus, for each message, 3 and its content's tokens with js-tiktoken's o200k_base encoding.
- Task 2
chat.ts: a conversation that never overflows
Write and export the Chat class in chat.ts. new Chat(system, maxTokens) starts this.history as an array holding the system message. ask(text) adds the user's message to this.history, sends lab-chat the history trimmed to maxTokens (task 1's trim), adds the model's reply to this.history as an assistant message, and resolves to the reply's text.
this.history keeps the whole conversation; only what is sent is trimmed. The check has a long conversation with it, with a small budget and a large one.
- Task 3
compact.ts: summarise the old part
Write and export compact(messages: readonly ChatMessage[], keep = 4): Promise<ChatMessage[]> in compact.ts. messages starts with one system message; everything after it except the last keep messages is old. Write the old messages as a transcript, one line per message: its role, a colon, a space and its content. Ask lab-chat in one request to summarise it, with the system message: Summarise this conversation in 50 words. and the transcript as the user message.
Return a new array: the system message, then a system message holding Earlier in this conversation: followed by a space and the summary, then the last keep messages. With nothing old to summarise, return a copy of messages and send no request.
- Task 4 · interview question
Interview: fix-trim.ts, a trim that breaks the conversation
recent(messages, n) in fix-trim.ts should keep a conversation short, but the API refuses what it returns. Run the file and read the error.
Fix recent so that it returns the system message (the first message), always, then the longest run of the latest messages, at most n of them, that starts with a user message. Do not change CONVERSATION; the check also tries other conversations and other values of n.
What you will learn
- Why history grows, and what it costs.
- Trimming to a token budget: what to keep, what to drop, in what order.
- Not breaking the conversation: the system message, user turns, tool calls.
- Summarising the old part (compaction), and what a summary loses.
More in Building with LLMs
Build "Conversation memory" yourself
SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant, in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.