Memory and multi-agent systems in TypeScript · Chapter 5 of 16

Semantic memory

By the end of this chapter you can turn memories into vectors with an embedding model, store them in SQLite as Float32Array bytes, find the ones closest in meaning to a question, rank them by similarity, recency and importance, and put only the best few into the request. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Memory
  • 5 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Memory and multi-agent systems in TypeScript are free. This one comes with the all labs plan.

Why you need this

Chapter 4's sheet answers "When is the walima?" because the question shares a word with walima_date. Families do not ask in keys. Rekha read out three questions from last week's night chats: "Who will take the photos?", "What colours is the decor?", "Kitne log aa rahe hain?". Not one of them contains photographer, theme or guests, so a lookup by key, or a search for the words, finds nothing.

And the sheets grow. A family with a guest list, every allergy, the rooms, the transport and the menu notes has hundreds of facts, and Sameer does not want all of them in every request. The assistant needs to find the few memories that matter for this question, by meaning, not by spelling.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    sentences.ts: a fact as a sentence

    Write and export sentence(key, value) in sentences.ts: one fact as a plain sentence. budget gives The budget is ₹38,00,000. (the rupees with Indian digit grouping: the last three digits, then groups of two); guests gives The guest count is 320.; an event's date, The mehndi is on 2026-12-09.; an event's venue, The mehndi is at Nahargarh Courtyard.; a diet, Sunita Sharma's diet is jain.; an allergy, Dev Mehta is allergic to peanuts.; theme, The decor theme is marigold and ivory.; and caterer, decorator or photographer, The caterer is Royal Rasoi.

    The check tries every fact of every wedding, and budgets of other sizes, such as 95,000 and 1,20,00,000.

  2. Task 2

    embed.ts: a memory index

    Write and export a class MemoryIndex in embed.ts. new MemoryIndex(path) opens or creates a SQLite file with a table memories with the columns id, wedding_id, text, at, importance and vector. The async method add(weddingId, memories) takes an array of objects with text, at and importance, embeds every text with lab-embed in requests of at most 100 texts, and stores one row per memory with its vector as float32 bytes. The async method search(weddingId, query, k = 3) embeds the query and returns that wedding's k most similar memories, best first, each an object with text, at, importance and similarity (the cosine, a number), or an empty array when the wedding has none.

    lab-embed's vectors have length 1, so the cosine is the dot product. The check adds a whole thread of 305 messages, reads the table itself, and searches two weddings with a new MemoryIndex on the same file.

  3. Task 3

    score.ts: similarity, recency and importance

    Write and export score(similarity, hoursAgo, importance, weights = [1, 1, 1], decay = 0.99) in score.ts: the first weight times the similarity, plus the second weight times decay to the power hoursAgo, plus the third weight times importance divided by 5.

    Then write and export the async function recall(index, weddingId, query, now, { k = 3, weights = [1, 1, 1] } = {}): give every memory of the wedding from index.search (task 2) a score field, with hoursAgo the hours from its at to now (both ISO times, possibly in different time zones), and return the k best by score, best first. The check tries memories from an hour to months old, and other weights.

  4. Task 4

    answer.ts: answer from what you recall

    In answer.ts, put every current fact of WD1042 (facts from shared/baraat.ts) into a MemoryIndex on a fresh answer.db (delete the file first): the text is task 1's sentence, at is the fact's at, and importance is 5 for the budget, a date, a diet or an allergy, 4 for a venue or the guest count, and 3 for the caterer, decorator, photographer or theme.

    Recall (task 3) the top 3 for the question Is anyone allergic to nuts?, with now 2026-10-22T10:00:00+05:30 and the default weights. Ask lab-chat the question at temperature 0 with a system message holding an instruction to answer from the notes, a blank line, Notes: on its own line, and the three memories, each a dash, a space and its text. Print the three texts, best first, one per line, then the reply.

  5. Task 5 · interview question

    Interview: fix-me.ts, vectors that come back wrong

    TinyIndex in fix-me.ts keeps memories and their vectors in SQLite, but its search ranks by nonsense: run it and look at the similarities it prints.

    Fix how vectors are written to and read from bytes, so that every vector comes back exactly as it went in, as 384 float32 values (1536 bytes in the table), and search ranks by the true cosine. Keep the class's name, its methods and the export.

What you will learn

  • Embeddings: text as a vector, and similarity as a dot product.
  • What to embed: facts as plain sentences beat raw messages.
  • A vector index in SQLite, vectors as float32 bytes.
  • Scoring by similarity, recency and importance, and why weights matter.
  • Recall into the request: the top few memories as notes.

More in Memory

Build "Semantic memory" yourself

Baraat Bliss, a Jaipur wedding planner, must remember months of family decisions across many agents. You build its assistant in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Memory and multi-agent systems in TypeScript are free. This one comes with the all labs plan.