RAG: search that answers · Chapter 33 of 37
What happened on that question
By the end of this chapter you can follow one question through every stage of the pipeline, say how long each stage took and what the question cost, and write a log line that Neha can audit, with no personal data in it. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Production
- 5 tasks
- In the all labs plan
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.
Why you need this
Kabir forwards a complaint: merchant M-2290 asked "When will I receive my settlement?" and the assistant said it did not know. The answer is in a help article, so what went wrong? Retrieval, the reranker, the prompt, the model? The pipeline prints nothing, so nobody can tell. Neha has a harder question: a month from now, a merchant says the assistant gave them a wrong date. She needs the exact answer, the chunks it came from, and who asked, without a phone number or a UPI id sitting in a log file.
You can run the question again, but that tells you what happens today, not what happened then: the index, the documents and the prompt have changed since. What you need is a record written at the time, for every question.
What you build: 5 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
span: time one stage
span(trace, name, run) runs one stage, run (async or a plain function), and adds a Span to trace.spans: { name, ms, ok }, where ms is the time from performance.now() before the stage to after it, and ok says whether it worked. It returns what run returned: a stage that waits 40 ms and returns 42 gives 42 and a span of about 40 ms with ok true.
When run throws, span must still add the span, with ok false, and then throw the same error again. A trace that loses the failed stage hides exactly the stage you need to see.
- Task 2
answerWithTrace: a trace for every question
Do what answer does, but run each stage inside span, named 'embed', 'retrieve', 'rerank', 'prompt' and 'model', in that order. Each question starts a new Trace: id from randomUUID(), spans empty, and totalMs measured with performance.now() around the whole pipeline.
Fill trace.tokens from what the stages report: embed is embed's tokens, input and output are callModel's tokensIn and tokensOut. Return { answer, hits, trace }: the reply's text, the 5 chunks the reranker kept (the ones sent to the model), and the trace. One question makes one embeddings, one rerank and one chat request.
- Task 3
costOf: what a question costs
costOf(tokens) is the cost in rupees, with PRICES in rupees per million tokens: embed tokens at PRICES.embed, input at PRICES.input, output at PRICES.output. costOf({ embed: 10, input: 640, output: 70 }) is (10 x 2 + 640 x 25 + 70 x 100) / 1,000,000, which is 0.02302.
costPerDay(traces) returns { perQuestion, perDay }: perQuestion is the average costOf of the traces' tokens, and perDay is perQuestion times QUESTIONS_PER_DAY (600). With no traces both are 0, not NaN.
- Task 4
redact: no personal data in the logs
redact(text) replaces phone numbers with [phone], emails with [email] and UPI ids with [upi]. Phone numbers: 98220 12345, 9822012345, +91 98220 12345 and the landline +91 20 4000 9999. An email: priya.sweets@gmail.com. A UPI id is a name, an @ and a bank handle with no dot after it: ramesh.k@okaxis, ravi@ybl.
Replace emails before UPI ids, or neha@paisasetu.in becomes [upi].in. Leave everything else alone: redact must return 'UTR 612345678901 for ₹1,00,000 on 2026-03-01, merchant M-1042, chunk help-refund-timelines#2, 11 am' unchanged, because a 12-digit UTR, amounts, dates and ids are what Neha audits.
- Task 5 · interview question
Interview: auditLine, a log line Neha can audit
auditLine(question, user, result) returns one line of JSON (JSON.stringify with no indentation) with: at (new Date().toISOString()), trace (the trace id), user (user.id), audience (user.audience), question (redacted), chunks (the ids of result.hits, in order), answer (redacted), citations (the ids citedIds finds in the answer that are among the chunks sent; drop any the model made up), ms (trace.totalMs rounded) and rupees (costOf of the trace's tokens).
With hits help-refund-timelines#1 and help-refund-how-to#0, and an answer citing help-refund-timelines#1 and help-made-up#9, citations is only help-refund-timelines#1. Arjun will ask: a merchant says the assistant told them T+2 a month ago, so how do you find out why?
What you will learn
- Spans and traces: the time of each stage of one question, kept together, including the stage that failed.
- Tokens and cost: what one question costs, and 600 a day.
- Structured logs: one JSON line per question, which a program can search.
- Redaction: taking phone numbers, emails and UPI ids out before anything is written.
More in Production
Build "What happened on that question" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.