RAG: search that answers · Chapter 37 of 37

The interview

By the end of this chapter you can build a small RAG service end to end, in one sitting, for documents you have never seen: chunks, keyword search from scratch, a reranker, permissions, citations, a refusal, and a report that proves it works. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The interview
  • 5 tasks
  • In the all labs plan

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.

Why you need this

Arjun ends the lab with the interview he gives every senior candidate. "Forget Paisa Setu. Here are ten documents from Thali Box, a meal tiffin company in Bengaluru. Customers and staff will both ask questions, and three of the documents are for staff only. Build me the assistant, and show me it works."

Everything so far stood on Paisa Setu's ready index: chunks made for you, embeddings cached, labelled questions written. None of that exists for Thali Box. The interview tests whether you understand the pieces well enough to build them again, quickly, and whether you prove the result instead of saying "it looks fine".

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1 · interview question

    Interview: chunkDocs, chunks

    chunkDocs(docs) splits every document by its '## ' headings. The text before the first heading is chunk 0 with heading '', but only if it is not blank; then one chunk per heading, numbered in order. A chunk is { id, doc_id, title, audience, heading, text }: id is the document's id, '#' and the number; heading is the heading's text; text is the section without its heading line, trimmed; doc_id, title and audience are the document's.

    chunkDocs(DOCS) has 20 chunks. The first is tb-plans#0 with heading '' and text 'Thali Box delivers home-style meals in steel tiffins, every day except Sunday.', then tb-plans#1 'Monthly plans'. tb-delivery starts with a heading, so tb-delivery#0 is 'Where we deliver' and tb-delivery#1 'Delivery times'. The 5 staff chunks keep audience 'staff'.

  2. Task 2 · interview question

    Interview: bm25, keyword search from scratch

    bm25(chunks, k1 = 1.2, b = 0.75) indexes words(searchText(chunk)) for every chunk and returns a function that scores a query. For each distinct query word in a chunk where it appears f times: idf = ln(1 + (N - df + 0.5) / (df + 0.5)), and the score grows by idf x f x (k1 + 1) / (f + k1 x (1 - b + b x length / avgLength)). N is the number of chunks, df how many contain the word, length the chunk's word count and avgLength the average. Return { id, score } for chunks scoring above 0, best first.

    On three small chunks, 'how long do UPI refunds take?' gives a#0 2.211471 then c#0 1.487218; a word repeated in the query counts once; a query no chunk shares gives [].

  3. Task 3 · interview question

    Interview: search, with permissions

    search(question, user, k = 3) chunks DOCS, keeps only what the user may see (staff see every chunk, customers only audience 'customer') and only then builds bm25 over those chunks. Take BM25's 10 best, score them with one rerankScores call, each as its heading, a newline and its text, and return the k best as Hits (the chunk with score, the rerank score), best first. When BM25 finds nothing, return [] without calling the reranker.

    Every answerable question in QUESTIONS finds its document in the top 3. For a customer, 'how much is the rider paid per delivery?' returns no staff chunk, and the staff page on rider pay never reaches the reranker; for staff, ops-rider-pay#0 comes first.

  4. Task 4 · interview question

    Interview: answer, cite, or refuse

    answer(question, user) searches (k 3). With no hits, or a best score below MIN_SCORE, return { text: REFUSAL, citations: [], refused: true } and do not call the model. Otherwise send lab-chat two messages: SYSTEM, then a user message 'Documents:', each hit as [id] text separated by blank lines, and 'Question: ' with the question last. Return { text, citations, refused }: the reply, the ids citedIds finds that were among the hits, and whether the reply starts with "I don't know".

    'how much does the non-veg plan cost?' is answered citing tb-plans#1. 'do you have a gluten free plan?' scores 0.000 at best and is refused with no chat request (the model would quote "Jain meals ... are free on any plan"). 'do you cater weddings?' makes no request at all.

  5. Task 5 · interview question

    Interview: report, prove it works

    report(questions) searches and answers every question as its role and returns { recall, answered, cited, refusals, leaks }, each share rounded to 3 decimals (0 when there is nothing to count). recall: for answerable questions, the share of their relevant docs among the hits' doc_ids, averaged. answered: answerable questions not refused. cited: of those answered, the share citing a chunk of a relevant doc. refusals: unanswerable questions (relevant empty) that were refused. leaks: the number of staff chunks in customers' hits.

    report(QUESTIONS) is { recall: 1, answered: 1, cited: 1, refusals: 1, leaks: 0 }. For the pause question plus 'do you cater weddings?' labelled as answerable from tb-plans, it is { recall: 0.5, answered: 0.5, cited: 1, refusals: 0, leaks: 0 }. Arjun will ask: how do you know it works, and that it never leaks?

What you will learn

  • A plan for the sitting: the five steps, in the order that lets each one be tested.
  • Search without embeddings: BM25 from the formula, and a reranker to fix its order.
  • Permissions and refusals as code, not as instructions in a prompt.
  • A report as the proof: recall, citations, refusals and leaks on labelled questions.

More in The interview

Build The interview yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.