RAG: search that answers · Chapter 28 of 37

Answers the app can use

By the end of this chapter you can get answers as JSON your code can trust: shaped by a schema, validated with zod, repaired when a model fences or garbles it, and retried with the error when it cannot be repaired. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Answers
  • 5 tasks
  • In the all labs plan

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.

Why you need this

The assistant is moving into Paisa Setu's app. The app does not show a paragraph: it shows the answer, a "Sources" button with the documents, and a "Talk to an agent" button when nothing was found. Kabir wants the same for his agents' dashboard. So the code needs three separate things from every reply: the text, the cited ids and whether an answer was found.

Parsing those out of free text with regular expressions breaks the first time the wording changes. Asking the model for JSON is better, but models return JSON the way people write it: in a code fence, with a stray comma, sometimes with a field missing. One JSON.parse that throws in production is one merchant with a blank screen.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    parseAnswer: a schema, and JSON you can trust

    AnswerSchema is a zod object with three fields: answer (a string), citations (an array of strings) and found (a boolean). You can derive the Answer type from it with z.infer, or keep the one in the starter.

    parseAnswer(text) parses text as JSON and validates it with AnswerSchema, so it either returns an Answer or throws. parseAnswer('{"answer":"30 days.","citations":["dev-payment-links-api#4"],"found":true}') is that object; '{"answer":"30 days.","found":"yes"}' throws (citations is missing and found is not a boolean), and so does citations as a single string, and text that is not JSON.

    JSON.parse returns any, and a cast to Answer checks nothing at runtime.

  2. Task 2

    askJson: ask for the schema

    askJson(question) retrieves the 5 best chunks and sends buildMessages(question, hits) to MODEL with response_format set to zodResponseFormat(AnswerSchema, 'support_answer'). Return the reply's content, through parseAnswer.

    The request's format is then of type json_schema, named support_answer, strict, with exactly the properties answer, citations and found. await askJson('how long is a payment link valid?') has found true and cites dev-payment-links-api#4.

  3. Task 3

    repairJson: repair the usual damage

    repairJson(text) returns the JSON alone: from the first { or [ to the last } or ], with every comma that comes right before a } or ] (spaces between are fine) removed, but only outside strings. Valid JSON comes back unchanged.

    repairJson('``json\n{"a":[1,2,],}\n``') is '{"a":[1,2]}', and a sentence before or after the fence is dropped too. In '{"answer":"Send {\"amount\": 500, } as the body",}' only the last comma goes: the one inside the string is text.

    Answers quote code with fences of their own: lab-chat-sloppy's real reply about webhook signatures has a ``` inside its answer, and must still parse after the repair.

  4. Task 4

    askWithRetry: retry with the error

    askWithRetry(question, model) asks like askJson but with the model given (lab-chat-sloppy by default, which fences its JSON), and parses with parseAnswer, without repairing. When that throws, send the same request again (same system message, same response format) with the user message changed to: Your last reply was not valid JSON: and the error's message, a blank line, then the whole original user message.

    Return { answer, attempts }: the parsed answer, and 1 or 2. If the second reply fails too, let the error go. With lab-chat-sloppy, attempts is 2; with lab-chat, the first reply is clean and attempts is 1.

  5. Task 5 · interview question

    Interview: askStructured, repair first, retry last

    askStructured(question, model) parses the reply; if that throws, it repairs it with repairJson and parses again; only if that throws too, it retries with the error as in task 4. Return { answer, attempts, repaired }, with repaired true when the repair is what made it parse.

    With lab-chat-sloppy and the webhook signature question, that is one request, attempts 1, repaired true. With lab-chat, one request, repaired false.

    Arjun will ask: repair or retry first, and what does each one cost?

What you will learn

  • A schema: describing the reply with zod, and asking the model for exactly that shape with response_format.
  • Validation: why JSON.parse plus a cast checks nothing.
  • Repair: fixing the usual damage (fences, trailing commas) without breaking the text inside.
  • Retry: sending the error back when repair is not enough, and what it costs.

More in Answers

Build "Answers the app can use" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.