Memory and multi-agent systems · Chapter 5 of 16
Semantic memory
By the end of this chapter you can turn memories into vectors with an embedding model, store them in SQLite, find the ones closest in meaning to a question, rank them by similarity, recency and importance, and put only the best few into the request. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Memory
- 5 tasks
- Python
- In the all labs plan
The first 2 chapters of Memory and multi-agent systems are free. This one comes with the all labs plan.
Why you need this
Chapter 4's sheet answers "When is the walima?" because the question shares a word with walima_date. Families do not ask in keys. Rekha read out three questions from last week's night chats: "Who will take the photos?", "What colours is the decor?", "Kitne log aa rahe hain?". Not one of them contains photographer, theme or guests.
And the sheets grow. A family with a guest list, every allergy, the rooms, the transport and the menu notes has hundreds of facts, and Sameer does not want all of them in every request. The assistant needs to find the few memories that matter for this question, by meaning, not by spelling.
What you build: 5 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
sentences.py: a fact as a sentence
Write sentence(key, value) in sentences.py: one fact as a plain sentence. budget gives The budget is ₹38,00,000. (the rupees with Indian digit grouping: the last three digits, then groups of two); guests gives The guest count is 320.; an event's date, The mehndi is on 2026-12-09.; an event's venue, The mehndi is at Nahargarh Courtyard.; a diet, Sunita Sharma's diet is jain.; an allergy, Dev Mehta is allergic to peanuts.; theme, The decor theme is marigold and ivory.; and caterer, decorator or photographer, The caterer is Royal Rasoi.
The check tries every fact of every wedding, and budgets of other sizes, such as 95,000 and 1,20,00,000.
- Task 2
embed.py: a memory index
Write a class MemoryIndex in embed.py. MemoryIndex(path) opens or creates a SQLite file with a table memories with the columns id, wedding_id, text, at, importance and vector. add(wedding_id, memories) takes a list of dicts with text, at and importance, embeds every text with lab-embed in requests of at most 100 texts, and stores one row per memory with its vector as float32 bytes. search(wedding_id, query, k=3) embeds the query and returns that wedding's k most similar memories, best first, each a dict with text, at, importance and similarity (the cosine, a float).
lab-embed's vectors have length 1, so the cosine is the dot product. The check adds a whole thread of 305 messages, reads the table with sqlite3, and searches two weddings with a new MemoryIndex on the same file.
- Task 3
score.py: similarity, recency and importance
Write score(similarity, hours_ago, importance, weights=(1.0, 1.0, 1.0), decay=0.99) in score.py: the first weight times the similarity, plus the second weight times decay to the power hours_ago, plus the third weight times importance divided by 5.
Then write recall(index, wedding_id, query, now, k=3, weights=(1.0, 1.0, 1.0)): give every memory of the wedding from index.search (task 2) a score key, with hours_ago the hours from its at to now (both ISO times, possibly in different time zones), and return the k best by score, best first. The check tries memories from an hour to months old, and other weights.
- Task 4
answer.py: answer from what you recall
In answer.py, put every current fact of WD1042 (baraat's facts) into a MemoryIndex on a fresh answer.db (delete the file first): the text is task 1's sentence, at is the fact's at, and importance is 5 for the budget, a date, a diet or an allergy, 4 for a venue or the guest count, and 3 for the caterer, decorator, photographer or theme.
Recall (task 3) the top 3 for the question Is anyone allergic to nuts?, with now 2026-10-22T10:00:00+05:30 and the default weights. Ask lab-chat the question at temperature 0 with a system message holding an instruction to answer from the notes, a blank line, Notes: on its own line, and the three memories, each a dash, a space and its text. Print the three texts, best first, one per line, then the reply.
- Task 5 · interview question
Interview: fix_me.py, vectors that come back wrong
TinyIndex in fix_me.py keeps memories and their vectors in SQLite, but its search ranks by nonsense: run it and look at the similarities it prints.
Fix how vectors are written to and read from bytes, so that every vector comes back exactly as it went in, as 384 float32 values (1536 bytes in the table), and search ranks by the true cosine. Keep the class's name and its methods.
What you will learn
- Embeddings: text as a vector, and similarity as a dot product.
- What to embed: facts as plain sentences beat raw messages.
- A vector index in SQLite, vectors as float32 bytes.
- Scoring by similarity, recency and importance, and why weights matter.
- Recall into the request: the top few memories as notes.
More in Memory
Build "Semantic memory" yourself
Baraat Bliss, a Jaipur wedding planner, must remember months of family decisions across many agents. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Memory and multi-agent systems are free. This one comes with the all labs plan.