RAG in Python · Chapter 28 of 37

Answers the app can use

By the end of this chapter you can get answers as JSON your code can trust: shaped by a schema, validated strictly with Pydantic, repaired when a model fences or garbles it, and retried with the error when it cannot be repaired. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Answers
  • 5 tasks
  • Python
  • In the all labs plan

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.

Why you need this

The assistant is moving into Paisa Setu's app. The app does not show a paragraph: it shows the answer, a "Sources" button with the documents, and a "Talk to an agent" button when nothing was found. Kabir wants the same for his agents' dashboard. So the code needs three separate things from every reply: the text, the cited ids and whether an answer was found.

Parsing those out of free text with regular expressions breaks the first time the wording changes. Asking the model for JSON is better, but models return JSON the way people write it: in a code fence, with a stray comma, sometimes with a field missing. One json.loads that raises in production is one merchant with a blank screen.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    structured.py: parse_answer, JSON you can trust

    The app needs three things from every reply: Answer has answer (a str), citations (a list of str) and found (a bool). Write parse_answer(text): parse text as JSON and validate it with Answer, strictly, so it either returns an Answer or raises.

    parse_answer('{"answer":"30 days.","citations":["dev-payment-links-api#4"],"found":true}') is that Answer. These raise: found as the string "yes" (Pydantic's lax mode would turn it into True), citations missing, citations as a single string, and text that is not JSON, such as JSON in a code fence. Make Answer strict, or validate with strict=True.

  2. Task 2

    structured.py: answer_format and ask_json, ask for the schema

    Write answer_format(): the response_format {"type": "json_schema", "json_schema": {"name": "support_answer", "strict": True, "schema": Answer's JSON schema with "additionalProperties" set to False}}. The schema's properties are exactly answer, citations and found, all required.

    Write ask_json(question): retrieve the 5 best chunks, send build_messages(question, hits) to MODEL with response_format=answer_format(), and return the reply's content (or "" when it is None) through parse_answer. ask_json("how long is a payment link valid?") has found True and cites only chunks that were sent.

  3. Task 3

    structured.py: repair_json, repair the usual damage

    Write repair_json(text): return the JSON alone, from the first { or [ to the last } or ], with every comma that comes right before a } or ] (spaces between are fine) removed, but only outside strings. Valid JSON comes back unchanged.

    repair_json('``json\n{"a":[1,2,],}\n`') is '{"a":[1,2]}', and a sentence before or after the fence is dropped too. In '{"answer":"Send {\"amount\": 500, } as the body",}' only the last comma goes: the one inside the string is text, and the escaped quotes do not end the string. lab-chat-sloppy's real reply to "how do I verify the X-Setu-Signature header on webhooks?" has a ` inside its answer, and must still parse after the repair, with that `` kept.

  4. Task 4

    structured.py: ask_with_retry, retry with the error

    Write ask_with_retry(question, model): ask like ask_json but with the model given (lab-chat-sloppy by default, which fences its JSON), and parse with parse_answer, without repairing. When that raises, send the same request again (same system message, same response format, same model) with the user message changed to: Your last reply was not valid JSON: and the error, a blank line, then the whole original user message.

    Return Retried(answer, attempts): the parsed answer, and 1 or 2. If the second reply fails too, let the error go. With lab-chat-sloppy attempts is 2; with lab-chat the first reply is clean and attempts is 1.

  5. Task 5 · interview question

    Interview: structured.py, ask_structured, repair first, retry last

    Write ask_structured(question, model): send the request once and parse the reply; if that raises, parse repair_json(reply); only if that raises too, retry with the error as in task 4. Return Structured(answer, attempts, repaired), with repaired True when the repair is what made it parse.

    With lab-chat-sloppy and "how do I verify the X-Setu-Signature header on webhooks?" that is one request, attempts 1, repaired True. With lab-chat, one request, repaired False. When the repair does not help, two requests, attempts 2, repaired False.

    Arjun will ask: repair or retry first, and what does each one cost?

What you will learn

  • A schema: describing the reply with a Pydantic model, and asking the model for exactly that shape with response_format.
  • Validation: why json.loads checks nothing, and why Pydantic's default mode is not strict enough.
  • Repair: fixing the usual damage (fences, trailing commas) without breaking the text inside.
  • Retry: sending the error back when repair is not enough, and what it costs.

More in Answers

Build "Answers the app can use" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.