RAG: search that answers · Chapter 27 of 37
Answers as they are written
By the end of this chapter you can stream an answer to the merchant token by token, send its sources with the stream, stop it when the merchant leaves, and measure what the merchant actually waits for. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Answers
- 4 tasks
- In the all labs plan
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.
Why you need this
Kabir watched merchants use the assistant. A long answer, like the fee table for net banking, takes about two and a half seconds, and for that time the chat window shows nothing. Merchants clicked again, which sent the question twice; some closed the chat and the model kept writing an answer nobody would read, and Paisa Setu paid for every token.
Everything so far waits for the whole reply: create returns once the model has finished. A model writes one token at a time, so it can send you each one as it is written. Then the first words appear at once, and you can stop the model the moment nobody is reading.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
streamAnswer: stream the tokens
streamAnswer(question, onToken) retrieves the 5 best chunks and sends buildMessages(question, hits) to MODEL with stream: true and stream_options: { include_usage: true }. Call onToken with each token's text as it arrives (never with an empty one), and return { text, usage }: text is the whole reply, usage the usage from the stream's last chunk (null if none came).
Not every chunk has text: the first has an empty content, the finish chunk has none, and the usage chunk has no choices at all. The tokens joined, and text, must be exactly the reply, with no undefined in it.
- Task 2
answerEvents: the context first, the sources last
answerEvents(question) is an async generator the web page reads. It yields, in order: { type: 'context', ids } with the ids of the 5 chunks, before the model is called; one { type: 'token', text } per streamed token; { type: 'sources', ids } with citedIds of the whole reply and the ids sent; and { type: 'done', usage } with the usage from the last chunk.
A citation arrives split over several tokens, so the sources can only be worked out once the stream is over.
- Task 3
streamUntil: stop when the merchant leaves
streamUntil(question, signal, onToken) streams like task 1, stops as soon as signal is aborted, and returns { text, cancelled }: the tokens received so far, joined, and whether signal was aborted. No token reaches onToken after the abort, and the function returns at once, not when the model would have finished.
An abort is not an error, so never throw for it. With a signal that is already aborted, the request must never reach the model, and the result is { text: '', cancelled: true }. When nobody leaves, the result is the whole reply with cancelled: false. Any other error is still thrown.
Stopping your loop is not enough: the request itself must stop, or the model keeps writing, and you keep paying, for a merchant who is gone.
- Task 4 · interview question
Interview: timeAnswer, the first token and the last
timeAnswer(question) retrieves, then streams an answer and measures from the moment it sends the request: firstTokenMs until the first chunk with text arrives, totalMs until the stream ends, and tokens, the usage's completion_tokens from the stream. Use performance.now().
For a 23-token answer the first token comes within a few milliseconds and the last after a few hundred: firstTokenMs is far below totalMs.
Arjun will ask: streaming does not make the model any faster, so what does it buy you, and what does it cost?
What you will learn
- Streaming: stream: true, chunks of tokens, and the usage at the end.
- Sources with a stream: what can be sent before the answer, and what only after it.
- Cancelling: an AbortController that stops the request, not just your loop.
- Time to first token: the wait a merchant feels, measured.
More in Answers
Build "Answers as they are written" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.