RAG in Python · Chapter 22 of 37

Find small, give big

By the end of this chapter you can search with small pieces and hand the model bigger ones (a chunk with its neighbours, the sentences around the best sentence, or the whole document) inside a token budget, and say with numbers what each choice costs and gains. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Better retrieval
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.

Why you need this

Kabir has found an answer that is right and useless. A merchant asks when a refund's clock starts, and the assistant quotes one chunk: "The clock starts when the refund status becomes processing." Starts counting what? The table that says cards take 5 to 7 working days is in the chunk just before it:

$ python3 demo.py
help-refund-timelines#1  'Refund times by payment method'  92 tokens
| How the customer paid | Time to reach the customer |
|---|---|
| UPI | 1 to 3 working days |
| Credit or debit card | 5 to 7 working days |
| Net banking | 5 to 7 working days |
| Wallet | Instant to 1 day |
| Instant refund (UPI and cards, Growth and Scale plans) | Within minutes |

Working days do not include Sundays and bank holidays.

help-refund-timelines#2  'When does the clock start?'  46 tokens
The clock starts when the refund status becomes **processing**. If a refund is waiting with the status **pending_balance** (because your balance was too low to cover it), the customer's wait only starts once it moves to processing.

Search found #2 because it is small and about one thing, which is exactly why it makes poor context on its own. Everything you have built so far hands the model the same pieces it searched with. You do not have to pick one size for both jobs.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    small_to_big.py: expand_window, a chunk and its neighbours

    Kabir's assistant found the right chunk and still gave a useless answer: the facts were in the chunk before it. Write expand_window(id, window=1) in small_to_big.py: the chunk with this id and the chunks up to window places before and after it in the same document, as Chunk objects in the document's order. A chunk's place is the number after # in its id (place(id) gives it), and chunks_of(doc_id) gives a document's chunks in order.

    expand_window("help-refund-timelines#2") is #1, #2 and #3 of help-refund-timelines. expand_window("help-refund-timelines#0") is #0 and #1, and the last chunk, #6, gives #5 and #6. With window 0 it is the chunk alone, and expand_window("dev-changelog#10", 2) is #8 to #12.

  2. Task 2

    small_to_big.py: sentence_window, the best sentences in context

    Write sentence_window(query, chunks, window=1, keep=3): cut every chunk into sentences with sentences(), chunk by chunk in order, and rerank all of them for the query in one request with rerank. Walk the results best first. For each, make a Block whose id is the chunk id, a colon and the sentence's place in its chunk (from 0), like "help-upi-error-u30-z9#1:2", and whose text is the sentence with up to window sentences before and after it from the same chunk, joined by one space. Skip a block whose text equals one you already kept, and return the first keep blocks.

    sentence_window(query, three chunks, 1, 3) is 3 blocks, the first built around the reranker's best sentence. When the best sentence is the first of its chunk, its window is that sentence and the next one only.

  3. Task 3

    small_to_big.py: pack, fit the blocks in a token budget

    Write pack(blocks, budget): go through the blocks in order and keep each one whose tokens, count(block.text), still fit in what is left of budget; skip a block that does not fit and try the next. Return Packed(blocks=the kept blocks in order, tokens=their total).

    With blocks of 300, 900 and 200 tokens and a budget of 1000, pack keeps the first and the third: tokens 500. A block that fits exactly is kept, and pack(blocks, 0) keeps nothing, with tokens 0.

  4. Task 4 · interview question

    Interview: small_to_big.py, small_to_big, chunks, windows or documents

    Write small_to_big(conn, query, mode, budget, table="chunks"): search the query's 10 closest chunks with vector_search (in table), turn them into blocks by mode, best first, and return pack(blocks, budget). "chunks": each chunk as Block(id=the chunk id, text=its text). "documents": each chunk's document once, as Block(id=the document id, text=document_text(document id)). "window": each chunk with expand_window (window 1), minus the chunks an earlier block already has (skip the hit when none are left), as Block(id=the chunk id, text=those chunks' texts joined by a blank line).

    tokens is never over budget, and no chunk or document is sent twice.

    Arjun will ask: why not always send the whole document?

What you will learn

  • Small-to-big: search one unit, give the model a bigger one that contains it.
  • Three parents: a chunk's neighbours, a sentence window, the whole document.
  • A token budget: packing blocks best first, skipping what does not fit.
  • The trade, measured: tokens, right documents, and how much of the answer the context holds.

More in Better retrieval

Build "Find small, give big" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.