RAG in Python · Chapter 25 of 37
Show your sources
By the end of this chapter you can read the chunk ids out of an answer, reject the ones the model could not have read, and show the merchant the documents behind every answer, with links, and never an internal one. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Answers
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.
Why you need this
Neha, head of compliance, has one question for every answer the assistant gives: where does this come from? If a merchant disputes a refund because "your assistant told me 5 days", she must be able to open the exact policy the answer came from. An answer without a source is one she cannot defend.
The prompt from chapter 24 already asks the model to cite ids, and it does. But a citation is only text the model wrote. Nothing so far reads those ids back, checks them, or turns [help-refund-timelines#1] into something a merchant can click. And as you will see, the model can cite a passage it was never given.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
citations.py: parse_citations, read the ids back
Neha wants to know where every answer comes from, so first read the ids back. Write parse_citations(answer) in citations.py: the chunk ids the answer cites, in the order they first appear, each only once. A citation is a chunk id in square brackets: a document id made of lower-case letters, digits and dashes, then #, then a number. Markers can touch, as in [a#1][b#2].
parse_citations("Cards take 5 to 7 days. [help-a#1]") is ["help-a#1"]; parse_citations("A [help-a#1]. B [help-b#2][help-a#1]") is ["help-a#1", "help-b#2"]; an answer with no markers gives []. Markdown links like Webhook events and checkboxes like [ ] are not citations.
- Task 2 · interview question
Interview: citations.py, verify_citations, only what was sent
Write verify_citations(answer, sent): sent is the ids of the chunks in the prompt. Return Checked(text, cited, invented): cited is the cited ids that were sent, in order; invented is the cited ids that were not sent, in order; text is the answer with every invented marker removed together with the spaces just before it, everything else unchanged.
For "Fees are 2% [policy-fee-schedule#2]. Refunds take 5 days. [help-a#1]" with sent ["help-a#1"], text is "Fees are 2%. Refunds take 5 days. [help-a#1]", cited is ["help-a#1"] and invented is ["policy-fee-schedule#2"]. An answer with no markers comes back unchanged, with both lists empty.
policy-fee-schedule#2 is a real chunk in the index, but it was not sent, so it is still invented. Arjun will ask why you reject an id that exists in your index.
- Task 3
citations.py: render_sources, sources a merchant can open
Write render_sources(answer, chunks): chunks are the Hits sent (id, doc_id, title, audience). Replace each chunk-id marker with the number of its document: [1] for the first document cited, [2] for the next, and so on. Two chunks of one document share a number, and the same number twice in a row (only spaces between) is shown once. Then add a blank line, the line Sources: and one line per document in number order, [n] title (link), where the link is doc_url(doc_id): for example [1] Refund Policy (https://paisasetu.in/docs/policy-refund-policy).
A merchant never sees an internal document: remove an internal chunk's marker with the spaces before it, and give it no number and no Sources line. An answer with no citations comes back unchanged, with no Sources list.
- Task 4
citations.py: answer_with_sources, the whole answer
Write answer_with_sources(conn, question, table="chunks"): retrieve the 5 best chunks, send build_messages(question, hits) to MODEL, check the reply with verify_citations against the ids you sent, and return that Checked with its text rendered by render_sources from the hits.
For "how long is a payment link valid?" the text ends with a Sources list linking the cited document, and no raw chunk ids are left in it. For a question the context does not answer, nothing is cited and text is the reply as it came.
What you will learn
- Parsing citations: finding chunk ids in an answer, and only those.
- Checking them: every cited id must be one you sent.
- Rendering sources: numbered markers and a list of documents with links, internal ones left out.
More in Answers
Build "Show your sources" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.