AI support copilot project (TypeScript) · Chapter 8 of 12
Streaming
By the end of this chapter you can write Server-Sent Events by hand, stream a model's answer piece by piece with an async generator, send it from Express with res.writeHead, res.write and a for await loop, stop when the member leaves, and say why the search runs before the stream starts. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Answers
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of AI support copilot project (TypeScript) are free. This one comes with the all labs plan.
Why you need this
POST /ask answers Pranav in one piece: the caller waits, then gets the whole JSON at the end. A real model writes its answer a few words at a time, and a long answer takes seconds. For all that time the member sees nothing at all.
Rukhsana, who builds Parda's React site, wants to put the copilot in the site's help box. She wants the sources at once and then the answer as it is written, the way chat apps do, and she needs the event names and their shapes exactly, because her code reads them. The code so far cannot do it: askModel asks for the whole answer and returns once, and res.json sends one body. There is no way yet to send part of a response, keep the connection open, and send more.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
stream.ts: one Server-Sent Event
Write sseEvent(event, data) in src/answer/stream.ts. Return "event: <event>", a new line, "data: " and the data as JSON on one line (JSON.stringify(data)), a new line, then the blank line that ends the event: one more new line.
sseEvent('done', { found: true }) is 'event: done\ndata: {"found":true}\n\n'. The Check runs that, a token with ₹ in it, a token with a new line inside its text, and an array.
- Task 2
stream.ts: stream the answer as the model writes it
Write streamAnswer(ai, question, hits) in src/answer/stream.ts, an async generator (async function*). Yield, each made with sseEvent and in this order: "sources", an array of { chunkId, title, heading } for every hit (before you ask the model); a "token" event { text: piece } for every piece the model streams (ai.chat.completions.create with model config.chatModel, messages buildMessages(question, hits), temperature 0, stream: true; a part's text is part.choices[0]?.delta?.content, and parts with no choices or no content are skipped); "citations", the ids parseCitations(text, the hits' ids) finds in the whole text; "done", { answer: the text without its markers, found: true when at least one id was cited }.
The Check reads your events for "Can I bring my own food to Sahyadri Lawns?": sources first and before any model request, one streamed request, tokens that add up to the model's text, then citations and done; then it wants found false for "Do you sell popcorn and cold drinks?".
- Task 3 · interview question
Interview: ask.routes.ts, POST /ask/stream
Write the handler of POST /ask/stream in src/modules/ask/ask.routes.ts. Search first: hits = await retrieve(ai, question). Then res.writeHead(200) with the headers content-type "text/event-stream; charset=utf-8", cache-control "no-cache" and x-accel-buffering "no". Then for await every event of streamAnswer(ai, question, hits) and res.write it; break out of the loop when res.destroyed is true (the member closed the page). Then res.end().
The Check streams "How do I change my seats?" through the app, reads the content type, the two headers and the events (the sources of the retrieved hits, done at the end), wants 422 for an empty question, closes the connection after the first token of a long answer and wants the route to stop reading the model's stream, and then makes the search fail and wants a 500.
Nikhil will ask: why does the route search before it writes the headers, instead of searching inside the stream?
What you will learn
- Server-Sent Events: what an event looks like on the wire, and how model providers use the same format.
- One event: JSON on one line, and the blank line that ends it.
- Async generators: async function*, yield, for await, and what break does to the generator.
- Streaming from the model: stream: true, chunks, delta.content, and the pieces with nothing in them.
- Streaming from Express: res.writeHead, res.write, res.end, and res.destroyed when the member leaves.
- Search first: why the route searches before it writes the headers.
More in Answers
Build "Streaming" yourself
Parda answers the same audience questions all day. You build the copilot that answers them from its own help pages, with sources, and says so when it does not know. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI support copilot project (TypeScript) are free. This one comes with the all labs plan.