RAG in Python · Chapter 23 of 37

Chunks that make sense alone

By the end of this chapter you can give every chunk a line of context before it is embedded, measure what that gains on the labelled questions, and work out what a model-written context sentence would cost for the whole index. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Better retrieval
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.

Why you need this

Kabir found the chunk that should answer "what should my code do when it gets a refund.failed webhook". It ranks badly, and reading it shows why: it is a JSON payload and two sentences. Nothing in it says "webhook". The words that would, the page title "Webhook events and payload examples" and the heading "refund.failed", were left behind when chapter 11 cut the page into chunks.

Chapter 20 found the same problem from the other side: the reranker read each chunk with its title in front. Search has it earlier, at the moment each chunk is embedded. A better search query or a cleverer index cannot help: the meaning is simply not in the text you embedded.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    contextual.py: with_context, the context line

    Kabir's refund.failed chunk is a JSON payload that never says "webhook": the words that would were left in the title and the heading. Write with_context(c) in contextual.py: the chunk's text with a line of context in front, the title, then " > " and the heading when the chunk has one, then a blank line, then the text. A chunk whose heading is "" gets the title alone, with no " > ".

    For help-refund-timelines#2 the first line is "How long does a refund take to reach the customer? > When does the clock start?", then a blank line and the chunk's text. A heading path like "A > B" stays as it is. This is exactly the text the lab model has cached embeddings for.

  2. Task 2

    contextual.py: build_context_table, the context embeddings

    Write build_context_table(conn, chunks, table=TABLE): embed with_context(c) for every chunk given with embed_all, drop table if it exists and create it with the columns id text PRIMARY KEY, doc_id text NOT NULL and embedding vector(384) NOT NULL, insert one row per chunk (its id, its doc_id and its context embedding), and return the number of rows.

    build_context_table(conn, read_chunks()) is 866, and building it again with 100 chunks leaves exactly 100 rows. The embedding stored for a chunk is the embedding of its context text, not of its text alone. Insert in bulk (COPY or executemany), not one statement per chunk.

  3. Task 3

    contextual.py: context_gain, what the line is worth

    Write context_gain(conn, questions, plain="chunks", context=TABLE): for every question whose relevant list is not empty, search its 20 closest chunks with vector_search in the plain table and in the context table, and score each list with the given reciprocal_rank (1 over the rank of the first relevant document). Return Gain(questions=how many were scored, plain_mrr=the average in plain, context_mrr=the average in context).

    On all 80 labelled questions, questions is 70 and context_mrr is higher than plain_mrr. With no answerable questions, return Gain(questions=0, plain_mrr=0.0, context_mrr=0.0).

  4. Task 4 · interview question

    Interview: contextual.py, situate, a sentence from the model

    Write situate(c): ask MODEL to situate the chunk, with SITUATE as the system message and one user message holding the whole document and the chunk in tags, each tag on its own line: "<document>", document_text(c.doc_id), "</document>", "<chunk>", c.text, "</chunk>", joined by newlines. Return Situated(sentence=the reply's text, stripped, tokens_in=the response's usage.prompt_tokens).

    For help-refund-timelines#2 the sentence names its section, "When does the clock start?", and tokens_in is far more than the chunk's 46 tokens: the whole document was sent.

    Arjun will ask: one model call per chunk, with the whole document each time. What does that cost for 866 chunks, and is it worth it here?

What you will learn

  • The context line: the title and heading path in front of the chunk's text, for embedding only.
  • A second table of context embeddings, and the gain, by question kind.
  • Situating with a model: a sentence of context written per chunk, as in Anthropic's "contextual retrieval".
  • The cost of doing that for 866 chunks, and when it pays.

More in Better retrieval

Build "Chunks that make sense alone" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.