RAG: search that answers · Chapter 33 of 37

What happened on that question

By the end of this chapter you can follow one question through every stage of the pipeline, say how long each stage took and what the question cost, and write a log line that Neha can audit, with no personal data in it. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Production
  • 5 tasks
  • In the all labs plan

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.

Why you need this

Kabir forwards a complaint: merchant M-2290 asked "When will I receive my settlement?" and the assistant said it did not know. The answer is in a help article, so what went wrong? Retrieval, the reranker, the prompt, the model? The pipeline prints nothing, so nobody can tell. Neha has a harder question: a month from now, a merchant says the assistant gave them a wrong date. She needs the exact answer, the chunks it came from, and who asked, without a phone number or a UPI id sitting in a log file.

You can run the question again, but that tells you what happens today, not what happened then: the index, the documents and the prompt have changed since. What you need is a record written at the time, for every question.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    span: time one stage

    span(trace, name, run) runs one stage, run (async or a plain function), and adds a Span to trace.spans: { name, ms, ok }, where ms is the time from performance.now() before the stage to after it, and ok says whether it worked. It returns what run returned: a stage that waits 40 ms and returns 42 gives 42 and a span of about 40 ms with ok true.

    When run throws, span must still add the span, with ok false, and then throw the same error again. A trace that loses the failed stage hides exactly the stage you need to see.

  2. Task 2

    answerWithTrace: a trace for every question

    Do what answer does, but run each stage inside span, named 'embed', 'retrieve', 'rerank', 'prompt' and 'model', in that order. Each question starts a new Trace: id from randomUUID(), spans empty, and totalMs measured with performance.now() around the whole pipeline.

    Fill trace.tokens from what the stages report: embed is embed's tokens, input and output are callModel's tokensIn and tokensOut. Return { answer, hits, trace }: the reply's text, the 5 chunks the reranker kept (the ones sent to the model), and the trace. One question makes one embeddings, one rerank and one chat request.

  3. Task 3

    costOf: what a question costs

    costOf(tokens) is the cost in rupees, with PRICES in rupees per million tokens: embed tokens at PRICES.embed, input at PRICES.input, output at PRICES.output. costOf({ embed: 10, input: 640, output: 70 }) is (10 x 2 + 640 x 25 + 70 x 100) / 1,000,000, which is 0.02302.

    costPerDay(traces) returns { perQuestion, perDay }: perQuestion is the average costOf of the traces' tokens, and perDay is perQuestion times QUESTIONS_PER_DAY (600). With no traces both are 0, not NaN.

  4. Task 4

    redact: no personal data in the logs

    redact(text) replaces phone numbers with [phone], emails with [email] and UPI ids with [upi]. Phone numbers: 98220 12345, 9822012345, +91 98220 12345 and the landline +91 20 4000 9999. An email: priya.sweets@gmail.com. A UPI id is a name, an @ and a bank handle with no dot after it: ramesh.k@okaxis, ravi@ybl.

    Replace emails before UPI ids, or neha@paisasetu.in becomes [upi].in. Leave everything else alone: redact must return 'UTR 612345678901 for ₹1,00,000 on 2026-03-01, merchant M-1042, chunk help-refund-timelines#2, 11 am' unchanged, because a 12-digit UTR, amounts, dates and ids are what Neha audits.

  5. Task 5 · interview question

    Interview: auditLine, a log line Neha can audit

    auditLine(question, user, result) returns one line of JSON (JSON.stringify with no indentation) with: at (new Date().toISOString()), trace (the trace id), user (user.id), audience (user.audience), question (redacted), chunks (the ids of result.hits, in order), answer (redacted), citations (the ids citedIds finds in the answer that are among the chunks sent; drop any the model made up), ms (trace.totalMs rounded) and rupees (costOf of the trace's tokens).

    With hits help-refund-timelines#1 and help-refund-how-to#0, and an answer citing help-refund-timelines#1 and help-made-up#9, citations is only help-refund-timelines#1. Arjun will ask: a merchant says the assistant told them T+2 a month ago, so how do you find out why?

What you will learn

  • Spans and traces: the time of each stage of one question, kept together, including the stage that failed.
  • Tokens and cost: what one question costs, and 600 a day.
  • Structured logs: one JSON line per question, which a program can search.
  • Redaction: taking phone numbers, emails and UPI ids out before anything is written.

More in Production

Build "What happened on that question" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.