RAG: search that answers · Chapter 24 of 37

Building the prompt

By the end of this chapter you can turn search results into the request a model answers from: rules it follows, passages it can cite, a size you control, and an order it reads well. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Answers
  • 5 tasks
  • In the all labs plan

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.

Why you need this

Retrieval works now: for 58 of the 62 answerable merchant questions, a right document is in the top five. But the model still has to answer from it, and Kabir has two replies from the first trial that worry him. One merchant asked about card fees, the passages said nothing about fees, and the model answered "around 2%" anyway. Another answer was right, but nobody could say which passage it came from. Arjun adds a third problem: some questions pull in eight long chunks, and he wants every question capped at 800 prompt tokens.

Everything you built so far ends at a list of chunks. Pasting that list under the question is not a prompt: nothing tells the model to stay inside it, nothing names the passages, and nothing stops the list from growing past the budget.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    SYSTEM: the rules

    SYSTEM is the system message of every request. Right now it only says who the assistant is, so when the passages do not answer a question the model answers anyway, from what payment companies usually do.

    Add the rules: answer only from the context, say you do not know when the context does not say, and cite the id of every passage used, in square brackets. With a context about the soundbox and a question about card fees, the reply must start with "I don't know"; with a question the context does answer, the reply must still answer it and cite the passage.

  2. Task 2

    buildMessages: the context, then the question

    buildMessages(question, chunks) returns two messages: the system message SYSTEM, then a user message with the line Context:, a blank line, the passages, a blank line, then Question: and the question.

    Each passage is its id in square brackets, a space, and its text, for example [help-refund-timelines#2] Refunds to cards take ... Passages are separated by a blank line, in the order given. A chunk's text can hold blank lines of its own: keep it as it is.

    buildMessages('How long is a link valid?', [{ id: 'a#1', text: 'Valid for 30 days.' }]) has the user message 'Context:\n\n[a#1] Valid for 30 days.\n\nQuestion: How long is a link valid?'.

  3. Task 3

    fitToBudget: drop the lowest-ranked chunks first

    chunks come best first. Return the longest run from the start of the list whose request, buildMessages(question, kept), costs at most budget prompt tokens, counted the way the provider counts: each message costs the tokens of its content plus 3, and the reply is primed with 3 more. tokens(text) counts a text's tokens.

    Keep the order, and do not change the list you were given. With the 8 best chunks for a question, a budget of 100000 keeps all eight, and a budget of 10 keeps none ([]).

    The trap: adding up only the chunks' sizes. The rules, the ids, the blank lines and the question are in the request too, and the Check measures the request you would send.

  4. Task 4 · interview question

    Interview: orderForModel, the best at the start and end

    Models read the start and the end of a long prompt best, and miss things in the middle. chunks come best first: return a new list with the best first, the second best last, the third second, the fourth second to last, and so on, so the weakest land in the middle.

    Six chunks ranked 1 to 6 become 1 3 5 6 4 2; five become 1 3 5 4 2; two stay 1 2; one stays 1; an empty list stays empty. Do not change the list you were given.

    Arjun will ask why the order matters when the model reads the whole prompt.

  5. Task 5

    ask: retrieve, fit, order, send

    ask(question) retrieves the 8 best chunks, keeps what fits in BUDGET (800) with fitToBudget, arranges the kept ones with orderForModel, builds the messages and sends them to MODEL.

    Return answer (the reply's text, '' when it is null), sent (the ids of the chunks sent, in the order they appear in the prompt) and promptTokens (usage.prompt_tokens of the reply, 0 when usage is missing).

    Fit first, then order: the budget must drop the lowest-ranked chunks, and only the kept ones are arranged.

What you will learn

  • Rules: a system message that keeps the model inside the context.
  • Passages with ids: the form the model can cite, [id] text.
  • A token budget: counting the whole request, and dropping the lowest-ranked chunks first.
  • Order: why the best chunks go at the start and the end.

More in Answers

Build "Building the prompt" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.