AI support copilot project · Chapter 8 of 12
Streaming
By the end of this chapter you can write Server-Sent Events by hand, stream a model's answer piece by piece with a Python generator, send it from FastAPI with a StreamingResponse, and say why the search runs before the stream starts. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Answers
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of AI support copilot project are free. This one comes with the all labs plan.
Why you need this
POST /ask answers Advait in one piece: the WhatsApp tool waits, then gets the whole JSON at the end. A real model writes its answer a few words at a time, and a long answer takes seconds. For all that time the member's chat shows nothing at all.
Farah wants the WhatsApp tool to show the sources at once and then the answer as it is written, the way chat apps do. The code so far cannot: ask_model asks for the whole answer and returns once, and a FastAPI route returns one body. There is no way yet to send part of a response, keep the connection open, and send more.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
stream.py: one Server-Sent Event
Write sse_event(event, data) in app/answer/stream.py. Return "event: <event>\n", then "data: " and the data as JSON (json.dumps(data, ensure_ascii=False), so ₹ stays ₹) and "\n", then the blank line that ends the event, "\n".
sse_event("done", {"found": True}) is 'event: done\ndata: {"found": true}\n\n'. The Check runs that, a token with ₹ in it, and a list.
- Task 2
stream.py: stream the answer as the model writes it
Write stream_answer(ai, question, hits) in app/answer/stream.py, a generator. Yield, in order: "sources", a list of {"chunk_id", "title", "heading"} for every hit (before you ask the model); a "token" event {"text": piece} for every piece the model streams (ai.chat.completions.create with model=settings.chat_model, messages=build_messages(question, hits), temperature=0, stream=True; a part's text is part.choices[0].delta.content, skip parts with no choices or no content); "citations", the ids parse_citations(text, the hits' ids) finds in the whole text; "done", {"answer": the text without its markers, "found": true when it cited}.
The Check (its own sse_event, so this task stands alone) reads your events for "Can I pay cash on delivery?": sources first and before any model request, one streamed request, tokens that add up to the model's text, then citations and done.
- Task 3 · interview question
Interview: router.py, POST /ask/stream
Write the route ask_stream in app/modules/ask/router.py. Search first, in the route: hits = retrieve(db, ai, body.question). Then return StreamingResponse(stream_answer(ai, body.question, hits), media_type="text/event-stream", headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}).
The Check (its own stream, so this task stands alone) streams POST /ask/stream for "How do I pause my subscription?", reads the content type and the two headers, wants the stream given the retrieved hits and a body that ends with done, and wants 422 for an empty question.
Kunal will ask: why does the route search before it returns the StreamingResponse, instead of searching inside the generator with the request's db session?
What you will learn
- Server-Sent Events: what an event looks like on the wire, and how model providers use the same format.
- One event in Python: JSON on one line, and ensure_ascii=False.
- Generators: yield, and when the body of a generator runs.
- Streaming from the model: stream=True, chunks, delta.content, and the pieces with nothing in them.
- StreamingResponse: what FastAPI sends when, and what an error after the 200 looks like.
- Search first: why the route searches before it starts the stream.
More in Answers
Build "Streaming" yourself
DabbaDrop answers the same customer questions all day. You build the copilot that answers them from its own help pages, with sources, and says so when it does not know. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI support copilot project are free. This one comes with the all labs plan.