RAG in Python · Chapter 37 of 37
The interview
By the end of this chapter you can build a small RAG service end to end, in one sitting, for documents you have never seen: chunks, keyword search from scratch, a reranker, permissions, citations, a refusal, and a report that proves it works. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The interview
- 5 tasks
- Python
- In the all labs plan
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.
Why you need this
Arjun ends the lab with the interview he gives every senior candidate. "Forget Paisa Setu. Here are ten documents from Thali Box, a meal tiffin company in Bengaluru. Customers and staff will both ask questions, and three of the documents are for staff only. Build me the assistant, and show me it works."
Everything so far stood on Paisa Setu's ready index: chunks made for you, embeddings cached, labelled questions written. None of that exists for Thali Box beyond the documents and fourteen labelled questions in thali_box.py. The interview tests whether you understand the pieces well enough to build them again, quickly, and whether you prove the result instead of saying "it looks fine".
What you build: 5 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1 · interview question
Interview: interview.py, chunk_docs, chunks
Arjun's interview: build Thali Box's assistant from its ten documents (DOCS in thali_box.py, given). Write chunk_docs(docs) in interview.py: split every document by its "## " headings. The text before the first heading is chunk 0 with heading "", but only if it is not blank; then one chunk per heading, numbered in order. A Chunk has id (the document's id, "#" and the number), doc_id, title and audience (the document's), heading (the heading's text) and text (the section without its heading line, trimmed).
chunk_docs(DOCS) has 20 chunks. The first is tb-plans#0 with heading "" and text "Thali Box delivers home-style meals in steel tiffins, every day except Sunday.", then tb-plans#1 "Monthly plans". tb-delivery starts with a heading, so tb-delivery#0 is "Where we deliver" and tb-delivery#1 "Delivery times". The 5 staff chunks keep audience "staff". A part whose text is blank is skipped, and the numbers have no gaps.
- Task 2 · interview question
Interview: interview.py, bm25, keyword search from scratch
Write bm25(chunks, k1=1.2, b=0.75): index words(search_text(chunk)) for every chunk and return a function that scores a query. For each distinct query word found f times in a chunk, idf = ln(1 + (N - df + 0.5) / (df + 0.5)), and the chunk's score grows by idf * f * (k1 + 1) / (f + k1 * (1 - b + b * length / avg_length)). N is the number of chunks, df how many contain the word, length the chunk's word count and avg_length the average. The function returns Scored(id, score) for the chunks scoring above 0, best first.
bm25(chunk_docs(DOCS))("how do I pause my tiffin?") puts tb-pause#1 first. A word repeated in the query counts once; a query no chunk shares gives []. The check compares your scores with the formula on three small chunks, and with other k1 and b.
- Task 3 · interview question
Interview: interview.py, search, with permissions
Write search(question, user, k=3): chunk DOCS, keep only what the user may see (staff see every chunk, customers only audience "customer"), and only then build bm25 over those chunks. Take BM25's 10 best, score them with one rerank_scores call, each as its heading, a newline and its text, and return the k best as Hits (the chunk's fields with score, the rerank score), best first. When BM25 finds nothing, return [] without calling the reranker.
Every answerable question in QUESTIONS finds its document in the top 3. For a customer, "how much is the rider paid per delivery?" returns no staff chunk, and the staff page on rider pay never reaches the reranker; for staff, ops-rider-pay#0 comes first. "do you cater weddings?" shares no word with any chunk: [] and no rerank request.
- Task 4 · interview question
Interview: interview.py, answer, cite, or refuse
Write answer(question, user): search (k 3). With no hits, or a best score below MIN_SCORE, return Reply(text=REFUSAL, citations=[], refused=True) and do not call the model. Otherwise send lab-chat two messages: SYSTEM, then a user message "Documents:", a blank line, each hit as [id] text separated by blank lines, a blank line and "Question: " with the question last. Return Reply(text=the reply, citations=the ids cited_ids finds that were among the hits, refused=whether the reply starts with "I don't know").
"how much does the non-veg plan cost?" is answered citing tb-plans#1. "do you have a gluten free plan?" scores 0.000 at best and is refused with no chat request. "do you cater weddings?" makes no request at all.
- Task 5 · interview question
Interview: interview.py, report, prove it works
Write report(questions): search and answer every question as its role (User(id="eval", role=q.role)) and return Report(recall, answered, cited, refusals, leaks), each share rounded to 3 decimals (0 when there is nothing to count). recall: for answerable questions, the share of their relevant docs among the hits' doc_ids, averaged. answered: answerable questions not refused. cited: of those answered, the share citing a chunk of a relevant doc (the id before the "#"). refusals: unanswerable questions (relevant empty) that were refused. leaks: the number of staff chunks in customers' hits.
report(QUESTIONS) is Report(recall=1, answered=1, cited=1, refusals=1, leaks=0). For the pause question plus "do you cater weddings?" labelled as answerable from tb-plans, it is recall 0.5, answered 0.5, cited 1, refusals 0, leaks 0. Arjun will ask: how do you know it works, and that it never leaks?
What you will learn
- A plan for the sitting: the five steps, in the order that lets each one be tested.
- Search without embeddings: BM25 from the formula, and a reranker to fix its order.
- Permissions and refusals as code, not as instructions in a prompt.
- A report as the proof: recall, citations, refusals and leaks on labelled questions.
More in The interview
Build The interview yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.