AI support copilot project · Chapter 7 of 12
The answer
By the end of this chapter you can turn a question and its five chunks into the messages a chat model reads, ask the model for an answer in a fixed JSON shape with structured outputs, read back its answer and the tokens it used, and keep only the citations that point at a source the member can really open. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Answers
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of AI support copilot project are free. This one comes with the all labs plan.
Why you need this
Chapter 6's search hands back the five chunks that answer a question best. Advait still has no answer: he asked "My dabba came 40 minutes late. Do I get my money back?", and the app can only show him five chunks of help pages. Farah's three rules for the copilot are about the answer itself: from the help pages only, with the page it used, and "I don't know" when the pages do not say.
Nothing so far talks to a chat model. And the WhatsApp tool that will call the copilot needs more than a paragraph of text: it needs the answer, the ids it can link to, and a yes or no for "did the pages say", every time, in the same shape. A model that writes free text sometimes puts the ids at the end, sometimes in the middle, and sometimes cites a page that does not exist.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
prompt.py: the instructions, the sources and the question
In app/answer/prompt.py write format_source(hit): two lines, "[<chunk_id>] <title> > <heading>" then the hit's text. Then build_messages(question, hits): a list of two messages, a system message whose content is INSTRUCTIONS, a blank line, "Sources:", a blank line, then every hit's format_source in order with a blank line between them; then a user message whose content is the question.
The Check formats the first source for "How much is the delivery charge?" and compares both messages with its own, character for character.
- Task 2
service.py: a structured answer, with its tokens
Write ask_model(ai, messages) in app/answer/service.py. Call ai.chat.completions.parse(model=settings.chat_model, messages=messages, response_format=AnswerOut, temperature=0). Return a tuple: the parsed AnswerOut (res.choices[0].message.parsed) and {"input": res.usage.prompt_tokens, "output": res.usage.completion_tokens}.
The Check reads the lab model's request log: one chat request with a json_schema response format, the model and temperature 0; your AnswerOut must equal the model's reply and your usage its usage.
- Task 3 · interview question
Interview: service.py, only real citations
Write answer_question(db, ai, question) in app/answer/service.py. hits = retrieve(db, ai, question); ask_model with build_messages(question, hits). Keep only the citations that are chunk ids of those hits, each once, in the model's order. Return {"question", "answer" (the model's), "found" (the model's found, and only when a citation was kept), "citations" (for each kept id {"chunk_id", "page_id", "title", "heading"} from its hit), "tokens" (ask_model's usage)}.
The Check (its own model and build_messages, so this task stands alone) has the model cite a made-up id, two real ones and a repeat, then only a made-up id; then it calls POST /ask.
Kunal will ask: the answer schema already says citations is a list of strings, and the model is told to cite. Why check the ids again in Python?
What you will learn
- Messages: a system message and a user message, and what goes in each.
- Sources with ids: why each chunk is sent with its id in square brackets, best first.
- Instructions that matter: what "answer only from the sources" changes, measured.
- Structured outputs: a Pydantic class turned into a strict JSON schema, chat.completions.parse, message.parsed, temperature=0, and the tokens in usage.
- What a schema cannot promise: that the ids are real.
More in Answers
Build The answer yourself
DabbaDrop answers the same customer questions all day. You build the copilot that answers them from its own help pages, with sources, and says so when it does not know. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI support copilot project are free. This one comes with the all labs plan.