RAG in Python · Chapter 33 of 37

What happened on that question

By the end of this chapter you can follow one question through every stage of the pipeline, say how long each stage took and what the question cost, and write a log line that Neha can audit, with no personal data in it. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Production
  • 5 tasks
  • Python
  • In the all labs plan

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.

Why you need this

Kabir forwards a complaint: merchant M-2290 asked "When will I receive my settlement?" and the assistant said it did not know. The answer is in a help article, so what went wrong? Retrieval, the reranker, the prompt, the model? The pipeline prints nothing, so nobody can tell. Neha has a harder question: a month from now, a merchant will say the assistant gave them a wrong date. She needs the exact answer, the chunks it came from and who asked, without a phone number or a UPI id sitting in a log file.

You can run the question again, but that tells you what happens today, not what happened then: the index, the documents and the prompt have changed since. What you need is a record written at the time, for every question.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    observe.py: span, time one stage

    Kabir cannot tell which stage failed merchant M-2290's question. Write span(trace, name) in observe.py, a context manager (the @contextmanager decorator is given): the stage runs in its with block. Read time.perf_counter() before the stage and after it, and add Span(name=name, ms=the time in milliseconds, ok=whether the stage worked) to trace.spans. A stage that sleeps 40 ms gives one span of about 40 ms with ok True.

    When the stage raises, span must still add the span, with ok False, and the same error must go on to the caller. A trace that loses the failed stage hides exactly the stage you need to see, and a context manager that catches the error without raising it again swallows it.

  2. Task 2

    observe.py: answer_with_trace, a trace for every question

    Write answer_with_trace(conn, question, user, table="chunks"): do what answer does, with each stage inside span, named "embed", "retrieve", "rerank", "prompt" and "model", in that order. Each question starts a new Trace(id=str(uuid.uuid4())), and total_ms is the whole pipeline in milliseconds, measured with time.perf_counter().

    Fill trace.tokens from what the stages report: embed is embed_query's tokens, input and output are call_model's tokens_in and tokens_out. Return Traced(answer=the reply's text, hits=the chunks the reranker kept, trace=the trace). One question makes one embeddings, one rerank and one chat request; two questions get two traces with different ids.

  3. Task 3

    observe.py: cost_of and cost_per_day, what a question costs

    Write cost_of(tokens): the cost in rupees, with PRICES in rupees per million tokens: embed tokens at PRICES["embed"], input at PRICES["input"], output at PRICES["output"]. cost_of(Tokens(embed=10, input=640, output=70)) is (10 x 2 + 640 x 25 + 70 x 100) / 1,000,000, which is 0.02302.

    Write cost_per_day(traces): Cost(per_question=the average cost_of of the traces' tokens, per_day=per_question times QUESTIONS_PER_DAY, 600). With no traces both are 0, not a ZeroDivisionError.

  4. Task 4

    observe.py: redact, no personal data in the logs

    Neha wants no personal data in a log file. Write redact(text): phone numbers become [phone], emails [email], PANs [pan] and UPI ids [upi]. Phone numbers: 98220 12345, 9822012345, +91 98220 12345 and the landline +91 20 4000 9999. Emails: priya.sweets@gmail.com, neha@paisasetu.in. UPI ids: a name, an @ and a bank handle with no dot after it, such as ramesh.k@okaxis and ravi@ybl. A PAN is five capitals, four digits and a capital: ABCPS1234K.

    Replace emails before UPI ids, or neha@paisasetu.in becomes [upi].in. Leave everything else alone: "UTR 612345678901 for ₹1,00,000 on 2026-03-01, merchant M-1042, GSTIN 27ABCPS1234K1Z5, chunk help-refund-timelines#2, 11 am" must come back unchanged, because a 12-digit UTR, amounts, dates, a GSTIN and ids are what Neha audits.

  5. Task 5 · interview question

    Interview: observe.py, audit_line, a log line Neha can audit

    Write audit_line(question, user, result): one line of JSON (json.dumps with no indent) with at (datetime.now(timezone.utc).isoformat()), trace (the trace's id), user (user.id), audience (user.audience), question (redacted), chunks (the ids of result.hits, in order), answer (redacted), citations (the ids cited_ids finds in the answer that are among the chunks sent; drop any the model made up), ms (the trace's total_ms, rounded) and rupees (cost_of the trace's tokens).

    With hits help-refund-timelines#1 and help-refund-how-to#0, and an answer citing help-refund-timelines#1 and help-made-up#9, citations is only help-refund-timelines#1. Arjun will ask: a merchant says the assistant told them T+2 a month ago, so how do you find out why?

What you will learn

  • Spans and traces: the time of each stage of one question, kept together, including the stage that failed.
  • Tokens and cost: what one question costs, and 600 a day.
  • Structured logs: one JSON line per question, which a program can search.
  • Redaction: taking phone numbers, emails, PANs and UPI ids out before anything is written.

More in Production

Build "What happened on that question" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.