Build an AI gateway (TypeScript) · Chapter 16 of 21

Streaming through the gateway

By the end of this chapter you can read server-sent events from raw bytes, turn three providers' stream formats into one stream of pieces, pass those pieces to the customer as they arrive, tell an answer that ended from a connection that dropped, decide when a stream may still fall back and when it must not, and make sure a customer who leaves stops the provider too. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Run it in production
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Jawab's live chat widget shows replies as they are written, word by word, the way people expect AI chat to look. 46 of Tuesday's replies came from it, with stream: true. A reply takes about half a second to write in full; the first words can be on the screen in about 300 ms. For a customer watching a chat bubble, that difference is the product.

Darwaza has no stream. Every request goes through handle, which waits for the whole answer and returns it at once. And a stream brings problems a whole answer never had: three providers stream in three formats; a stream can drop in the middle, leaving half an answer that looks like a whole one; once a customer is reading one model's words, Darwaza cannot quietly switch to another; and a customer who closes the widget should not leave a provider writing, and billing, to nobody.

Joseph wants every stream to end in one of two clear ways, never silence. Lakshmi wants streams billed like everything else, from their usage.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    sse.ts: read server-sent events

    Export async function* sseEvents(body: AsyncIterable<Uint8Array>) yielding { event?, data } per server-sent event: decode with one TextDecoder and { stream: true }, treat \r\n as \n, keep a buffer, and each time it holds a blank line yield the block before it (event: gives the name, data: lines the data without "data: ", several joined with \n). A block with no data is not an event, nor is a block the body ends in the middle of. Yield each event as soon as its blank line has arrived.

  2. Task 2

    stream-from.ts: three stream formats, one kind of piece

    Export async function* streamFrom(target, req, signal?) yielding pieces: { type: 'delta', text } for every piece of text as it comes, then exactly one { type: 'end', usage, finish } or { type: 'fail', error }. Fetch build(target, req, true) with the signal (fetch throws: a fail of kind 'timeout' if the signal aborted, else 'network'; an error status: a fail from normalizeError). Read the body with sseEvents and each provider's format (Nimbus: choices[0].delta.content, usage in a last chunk, [DONE]; Kestrel: message_start, content_block_delta, message_delta, message_stop; Tara: candidates per piece, finishReason and usageMetadata on the last). usage is Darwaza's Usage (input at full price, cached, cacheWrite, output); finish is 'length' for length, max_tokens or MAX_TOKENS. Only the provider's own end means the answer is whole: a body that ends or breaks before it gives a fail of kind 'dropped' (status 502).

  3. Task 3

    gateway.ts: streaming through Darwaza

    Export async function* stream(req) yielding GatewayEvents, and gateway = { handle: createDarwaza().handle, stream }. For each target of route(req): abort it if no piece has come within TIMEOUT_MS[target] (1500 if missing); yield each delta right away; on end yield { type: 'done', model: target, usage, costInr, finish } and stop; on a fail before any delta go to the next target; on a fail after a delta yield { type: 'error', status: 502, error } and stop; however the target ends, clear the timer and abort its controller. No target left: one error with the last failure's status and message (503 if none).

    Under if (import.meta.main): replay 12:00 to 21:00 through gateway, print the report, then from the results of streamed requests (ttftMs or complete set):

    streams: N, ended with done or error: C

    time to first piece: p50 X ms, p95 Y ms

  4. Task 4 · interview question

    Interview: cancel.ts, the customer closed the chat

    Export firstWords(req, n): read stream(req) from ./gateway.ts only until the text has n words (split on whitespace), stop reading, and return the first n words joined with single spaces.

    Under if (import.meta.main), with Thenga's first reply ({ ...day({ features: ['reply'], teams: ['thenga'] })[0].request, stream: true }) and kestrel-pro's output from ledger() before and after each run (after await settle()), print:

    whole answer: N output tokens billed (reading the stream to the end)

    first 8 words: (the words)

    stopped after 8 words: N output tokens billed (with firstWords(req, 8))

What you will learn

  • Server-sent events: the text format every provider streams in, read from raw bytes.
  • Three formats: where Nimbus, Kestrel and Tara put the text, the usage and the end.
  • Pieces as they come: time to first piece, and why buffering throws it away.
  • Drops: why only the provider's end marker means an answer is whole.
  • Fallback before the first piece, error after it: and a first-piece timeout.
  • Cancelling: stopping the provider when the reader goes away, and what it is billed.

More in Run it in production

Build "Streaming through the gateway" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.