RAG: search that answers · Chapter 22 of 37

Find small, give big

By the end of this chapter you can search with small pieces and hand the model bigger ones (a chunk with its neighbours, the sentences around the best sentence, or the whole document) inside a token budget, and say what each choice costs and gains. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Better retrieval
  • 4 tasks
  • In the all labs plan

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.

Why you need this

Kabir has found an answer that is right but useless. A merchant asks when a refund's clock starts, and the assistant quotes one chunk: "The clock starts when the refund status becomes processing." Starts counting what? The chunk was found because it is small and focused, which is exactly why it makes poor context on its own: the table that says cards take 5 to 7 working days is in the chunk before it.

Small pieces search well: their embedding is about one thing. Big pieces answer well: they carry the surrounding facts. You do not have to pick one size for both jobs.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    expandWindow: a chunk and its neighbours

    Return the chunk with this id and the chunks up to window places before and after it in the same document, as Chunk objects in the document's order. A chunk's place is the number after # in its id; chunksOf(docId) gives a document's chunks in order.

    expandWindow('help-refund-timelines#2') is #1, #2 and #3 of help-refund-timelines. expandWindow('help-refund-timelines#0') is #0 and #1, and the last chunk, #6, gives #5 and #6. With window 0 it is the chunk alone, and expandWindow('dev-changelog#10', 2) is #8 to #12.

  2. Task 2

    sentenceWindow: the best sentences, with their neighbours

    Cut every chunk into sentences with sentences(), chunk by chunk in order, and rerank all the sentences for the query in one rerank request. Walk the results best first. For each, make a Block: id is the chunk id, a colon and the sentence's place in its chunk (from 0), like 'help-upi-error-u30-z9#1:2'; text is the sentence with up to window sentences before and after it from the same chunk, joined by one space. Skip a block whose text equals one you already kept, and return the first keep blocks.

    sentenceWindow(query, chunks, 1, 3) is 3 blocks, the first built around the reranker's best sentence.

  3. Task 3

    pack: fit the blocks in a token budget

    Go through the blocks in order and keep each one whose tokens, count(text), still fit in what is left of budget; skip a block that does not fit and try the next. Return { blocks: the kept blocks in order, tokens: their total }.

    With blocks of 300, 900 and 200 tokens and a budget of 1000, pack keeps the first and the third: tokens 500. A block that fits exactly is kept, and pack(blocks, 0) is { blocks: [], tokens: 0 }.

  4. Task 4 · interview question

    Interview: smallToBig, chunks, windows or documents

    Search the query's 10 closest chunks with vectorSearch, turn them into blocks by mode, best first, and pack them into budget. 'chunks': each chunk, { id: the chunk id, text: its text }. 'window': each chunk with expandWindow (window 1), minus the chunks an earlier block already has (skip the hit when none are left), { id: the chunk id, text: those chunks' texts joined by a blank line }. 'documents': each chunk's document once, { id: the document id, text: documentText(document id) }.

    Return what pack returns. tokens is never over budget.

    Arjun will ask: why not always send the whole document?

What you will learn

  • Small-to-big: search one unit, return a bigger one that contains it.
  • Three parents: a chunk's neighbours, a sentence window, the whole document.
  • The token budget: packing blocks best first, skipping what does not fit.
  • The trade: measured tokens, right documents, and how much of the answer the context holds.

More in Better retrieval

Build "Find small, give big" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.