RAG in Python · Chapter 27 of 37
Answers as they are written
By the end of this chapter you can stream an answer to the merchant token by token, send its sources with the stream, stop it when the merchant leaves, and measure what the merchant actually waits for. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Answers
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.
Why you need this
Kabir watched merchants use the assistant. A long answer, like the fees for net banking, takes about a second and a half, and for that time the chat window shows nothing. Merchants clicked again, which sent the question twice. Some closed the chat, and the model kept writing an answer nobody would read, and Paisa Setu paid for every token.
Everything so far waits for the whole reply: create returns once the model has finished. A model writes one token at a time, so it can send you each one as it is written. Then the first words appear at once, and you can stop the model the moment nobody is reading.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
streaming.py: stream_answer, stream the tokens
Kabir wants the first words on screen at once. Write stream_answer(question, on_token): retrieve the 5 best chunks and send build_messages(question, hits) to MODEL with stream=True and stream_options={"include_usage": True}. Call on_token with each piece of text as it arrives, never with "" or None, and return Streamed(text=the whole reply, usage=the usage from the stream's last chunk, as a Usage, or None if none came).
Not every chunk has text: the first has an empty content, the finish chunk has None, and the usage chunk has no choices at all. The pieces joined, and text, must be exactly the reply, with no None in it.
- Task 2
streaming.py: answer_events, the context first, the sources last
Write the async generator answer_events(question), which the web page reads. It yields, in order: ContextEvent(ids=the ids of the 5 retrieved chunks), before the model is called; one TokenEvent(text=...) per piece of text, streamed with new_client(), an AsyncOpenAI; SourcesEvent(ids=cited_ids(the whole reply, the ids sent)); and DoneEvent(usage=the usage from the last chunk).
A citation arrives split over several tokens, so the sources can only be worked out once the stream is over.
- Task 3
streaming.py: stream_until, stop when the merchant leaves
Write stream_until(question, left, on_token): stream like task 1 with client, stop as soon as left (a threading.Event) is set, and return Partial(text=the pieces received so far, joined, cancelled=whether left is set). No piece reaches on_token after left is set, and the function returns at once, not when the model would have finished.
If left is already set, never send the request: return Partial(text="", cancelled=True). When nobody leaves, the result is the whole reply with cancelled=False. Leaving is not an error, so never raise for it. Leaving the loop is not enough: the response must be closed when stream_until returns, or the model keeps writing into an open connection.
- Task 4 · interview question
Interview: streaming.py, time_answer, the first token and the last
Write time_answer(question): retrieve, then stream an answer (with the usage) and measure with time.perf_counter, in milliseconds, from just before the request is sent: first_token_ms until the first chunk with text arrives, total_ms until the stream ends. Return Timing(first_token_ms, total_ms, tokens=the usage's completion_tokens).
For a 23-token answer the first token comes within a few milliseconds and the last after a few hundred: first_token_ms is far below total_ms.
Arjun will ask: streaming does not make the model any faster, so what does it buy you, and what does it cost?
What you will learn
- Streaming: stream=True, chunks with a delta, and the usage at the end.
- Sources with a stream: what can be sent before the answer, and what only after it, as an async generator of events.
- Stopping: leaving the loop and closing the response, so the model stops too.
- Time to first token: the wait a merchant feels, measured.
More in Answers
Build "Answers as they are written" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.