Memory and multi-agent systems · Chapter 2 of 16

Short-term memory

By the end of this chapter you can replay a long thread as a conversation, measure how a resend-everything assistant grows, keep a window of the most recent turns that fits a token budget without breaking the conversation, and pin the messages that must not be trimmed away. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Memory
  • 5 tasks
  • Python
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

Chapter 1 ended with a list: memory is what your code resends. Karan tried it on WD1042, Ananya and Rohan's wedding: every message of the thread, then the family's question. It worked for months. Then, one session in the summer, the API started refusing every request with context_length_exceeded, and the assistant went silent.

Sameer had a second complaint before the first was fixed. Had the window been big enough, answering after every one of WD1042's 305 messages with the whole thread would have sent 1,544,438 prompt tokens, about ₹309 at lab-chat's ₹200 per million, for one wedding. Keeping each request under 3,000 tokens halves it (767,170 tokens, about ₹153). A conversation needs a short-term memory: the part that goes out with every request, chosen on purpose.

The lesson

A thread as a conversation

/workspace/data is read for you by baraat, a small module on your PYTHONPATH (/opt/track/lib/baraat.py; it only reads, it never writes). Chapter 1 had you read the files by hand; from now on use it:

from baraat import as_message, sessions, thread, tokens

wd1042 = thread("WD1042")          # 305 messages, oldest first
print(as_message(wd1042[16]))
{'role': 'user', 'content': 'Ananya Sharma: One important thing: my mother Sunita Sharma is strictly vegetarian.'}

as_message turns a thread message into a user message, Sender: text, for every sender, Rekha and Sameer too: the assistant reads the whole WhatsApp group as input. sessions("WD1042") groups the messages by session (one day's conversation, numbered from 1), and tokens(messages) counts a request the API's way, as chapter 1 did by hand: 3, plus 3 and the content's o200k tokens per message (and a tool call's JSON).

Replay a short wedding, WD1057, session by session, and count each session alone and the resend after it:

session 1: 14 messages, 457 tokens, resend 457 tokens
session 2: 6 messages, 177 tokens, resend 631 tokens
session 3: 8 messages, 275 tokens, resend 903 tokens
session 4: 14 messages, 582 tokens, resend 1482 tokens
session 5: 13 messages, 419 tokens, resend 1898 tokens

The sessions stay about the same size; the resend only grows. WD1057 is done in five sessions and fits. A ten-month wedding does not.

Turns, the unit of a conversation

A conversation is not a list of independent messages. A turn starts at a user message and runs up to the next one: the question, the assistant's reply, and, when the assistant used a tool, the tool call and its result in between. Cut inside a turn and the request breaks. Keep a tool result without the assistant message that asked for it, and the API refuses the request:

openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid
parameter: messages with role 'tool' must be a response to a preceeding
message with 'tool_calls'.", 'type': 'invalid_request_error', 'param':
'messages.[0].role', 'code': None}}

An assistant reply whose question was cut off is accepted, but it reads as an answer to nothing, and real models are confused by it. So a short memory is made of whole turns, and the system messages at the start (the instructions) always stay.

A window, and a budget

The simplest short-term memory is a sliding window: the system messages, then the last n turns. It is cheap, needs no model call, and the newest messages, which are most often what the family is talking about, are always there.

Counting turns does not bound the size, though: one family writes "ok", another pastes a whole menu. What the API limits is tokens, so the window you want is the most recent turns that fit a token budget. The budget is not the window: leave room for the reply (and, from chapter 4, the notes). At 3,000 tokens, WD1042's conversation keeps about 80 of its messages, and the cost per request stops growing.

What a trim forgets

A window forgets from the start, and in a wedding thread the start is where the decisions are. Here is WD1042 replayed with a system message ("You are Baraat Bliss's planning assistant. Answer from the conversation."), the question "Who is the caterer?", and the conversation trimmed to 3,000 tokens, at temperature=0:

A classic Rajasthani menu works for every guest: laal maas, dal baati churma, paneer tikka and ghewar.

The caterer was chosen in April, months before the window starts, so the lab model has nothing to go on and invents, laal maas and all, for a family with a Jain mother.

Pinning what matters

Not every message is equal. Most of WD1042 is the real noise of planning: dances, buses, dress codes, small talk in Hinglish. A few messages state facts: 28 of 305 carry a facts label in the data. Pinning keeps those messages, in their places, and trims only the rest, oldest first. Pinned, the same question at the same budget gets:

Ananya Sharma: We are thinking of Shree Annapurna Caterers, they did my cousin's wedding. Rohan Mehta: Final caterer: Royal Rasoi.

Better: the decision survived. But look at what came with it. The request now holds the option the bride mentioned in February and the final choice, and the lab model repeats both. Pinning keeps every old value next to the new one. Here the data's labels chose what to pin; a real assistant has to decide itself which messages state facts, and which value is the current one. That is chapters 4 to 6.

Counting tokens honestly

A common shortcut is "about four characters per token". For WD1042 it says 9,448 tokens; the API counts 10,640. Two things make the estimate low:

  • Every message costs 3 tokens on top of its text, and a thread has hundreds of short messages.
  • Hinglish splits into more tokens than English. "Mehndi Nahargarh Courtyard mein kar lete hain, it is smaller and cosy." is 70 characters: the estimate says 17 tokens, o200k counts 20.

An estimate that is 12% low fills an 8,000-token budget with a request the API counts as nearly 9,000, and it is refused. Count with the tokenizer the API uses, the way the API counts.

Your turn

Measure WD1042's resend session by session, then build the window, the trim and the pinned trim, and ask about the mehndi both ways. The interview question is a trim that trusts a token estimate.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    growth.py: a thread that outgrows the window

    An assistant that resends everything sends, after each session, every message so far. Replay WD1042 session by session with baraat's sessions and as_message, and count with baraat's tokens. For each session print one line: session, its number, a colon, its number of messages, the tokens of its messages alone as one request, and the tokens of the resend (every message from session 1 to this one, as one request), for example:

    session 1: 14 messages, 457 tokens, resend 457 tokens

    Then print a last line: first session that does not fit: and the number of the first session whose resend is over lab-chat's window, baraat's WINDOW (8192).

  2. Task 2

    window.py: a window of the last turns

    A turn starts at a user message and runs up to the next user message: the user's message and every assistant reply, tool call and tool result that answers it. Write last_turns(messages, n) in window.py: it returns a new list with the system messages at the start of messages, then the last n turns, whole. With fewer than n turns it keeps them all (but never an assistant or tool message before the first user message); with n of 0, only the system messages.

    Do not change the list you are given. The check tries conversations with two system messages, a greeting before the first user message, and a tool call with its result.

  3. Task 3

    trim.py: trim to a token budget

    Write trim(messages, budget) in trim.py: it returns the system messages and the most recent whole turns whose request fits in budget tokens, counted with baraat's tokens: as many turns as fit, dropping the oldest first. The last turn is always kept; if the system messages and the last turn alone do not fit, raise ValueError. Do not change the list you are given.

    The check trims replayed threads and a conversation with a tool call to several budgets, and compares what you keep with the most that fits.

  4. Task 4

    pinned.py: pin what matters

    Write trim_pinned(messages, budget, pinned) in pinned.py, for a replayed thread (system messages, then user messages); pinned is a set of indexes into messages. Keep the system messages at the start, the last message and every pinned message; drop the other messages oldest first, one at a time, until the request fits in budget tokens (baraat's tokens). Kept messages stay in their order. If what must be kept does not fit, raise ValueError.

    Then, under if __name__ == "__main__":, build WD1042 as one conversation: a system message telling the assistant to answer from the conversation, every message of the thread as as_message makes it, and the question When is the mehndi?. Pin the messages whose facts label is not empty. Ask lab-chat twice at temperature 0, with the conversation trimmed to 3000 tokens: first with task 3's trim, then with trim_pinned. Print plain: and the first reply, then pinned: and the second, each on its own line.

  5. Task 5 · interview question

    Interview: fix_me.py, a trim that overflows

    trim(messages, budget) in fix_me.py should keep a replayed thread under budget tokens, but the API refuses what it returns for WD1042. Run it and read the error.

    Fix trim so that what it returns always fits the budget counted the API's way, and drops no more than it must: the first message, then the newest messages that fit. Keep the function's name and parameters. The check tries other threads (Hinglish too) and other budgets.

What you will learn

  • Replaying a thread as a chat, and how fast the resend grows.
  • Turns: the unit a conversation can be cut at without breaking it.
  • A window of the last turns, and trimming to a token budget.
  • Pinning: keeping the messages that hold decisions.
  • Counting tokens honestly, not by the length of the text.

More in Memory

Build "Short-term memory" yourself

Baraat Bliss, a Jaipur wedding planner, must remember months of family decisions across many agents. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.