Build an AI gateway · Chapter 16 of 21

Streaming through the gateway

By the end of this chapter you can read server-sent events from a response's lines, turn three providers' stream formats into one stream of pieces, pass those pieces to the customer as they arrive, tell an answer that ended from a connection that dropped, decide when a stream may still fall back and when it must not, and make sure a customer who leaves stops the provider too. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Run it in production
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.

Why you need this

Jawab's live chat widget shows replies as they are written, word by word, the way people expect AI chat to look. 46 of Tuesday's replies came from it, with "stream": True. A reply takes about half a second to write in full; the first words can be on the screen in about 300 ms. For a customer watching a chat bubble, that difference is the product.

Darwaza has no stream. Every request goes through handle, which waits for the whole answer and returns it at once. And a stream brings problems a whole answer never had: three providers stream in three formats; a stream can drop in the middle, leaving half an answer that looks like a whole one; once a customer is reading one model's words, Darwaza cannot quietly switch to another; and a customer who closes the widget should not leave a provider writing, and billing, to nobody.

Joseph wants every stream to end in one of two clear ways, never silence. Lakshmi wants streams billed like everything else, from their usage.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    sse.py: read server-sent events

    Write async def sse_events(lines), an async generator over a response's lines (as httpx's aiter_lines() gives them, without line endings), yielding {"event": name or None, "data": text} per server-sent event: each blank line ends a block; event: gives the name (stripped), data: lines the data (without "data:" and one space after it, if there is one; several joined with "\n"). A block with no data is not an event, nor is a block the lines end in the middle of. Yield each event as soon as its blank line has arrived.

  2. Task 2

    stream_from.py: three stream formats, one kind of piece

    Write async def stream_from(target, req), an async generator yielding pieces: {"type": "delta", "text"} for every piece of text as it comes, then exactly one {"type": "end", "usage", "finish"} or {"type": "fail", "error": ProviderError}. Send build(target, req, True) with async with client.stream("POST", ...) (client from darwaza); an error status: one fail from normalize_error; httpx.HTTPError before the stream starts: a fail of kind "network". Read the lines with sse_events and each provider's format (Nimbus: choices[0].delta.content, usage in a last chunk, [DONE]; Kestrel: message_start, content_block_delta, message_delta, message_stop; Tara: candidates per piece, finishReason and usageMetadata on the last). usage is Darwaza's Usage (input at full price, cached, cache_write, output); finish is "length" for length, max_tokens or MAX_TOKENS. Only the provider's own end means the answer is whole: lines that end or break (httpx.HTTPError) before it give a fail of kind "dropped" (status 502).

  3. Task 3

    gateway.py: streaming through Darwaza

    Write async def stream(req), an async generator of GatewayEvents, and gateway = SimpleNamespace(handle=darwaza.handle, stream=stream) with darwaza = create_darwaza(). For each target of route(req): until the first delta, wait for each piece at most TIMEOUT_MS[target] (1500 if missing) with asyncio.timeout around await anext(pieces, None), and give the target up when it passes; yield each delta right away; on end yield {"type": "done", "model": target, "usage", "cost_inr", "finish"} and stop; on a fail before any delta go to the next target; on a fail after a delta yield {"type": "error", "status": 502, "error"} and stop; however the target ends (finally), await pieces.aclose(). No target left: one error with the last failure's status and message (503 if none).

    Under if __name__ == "__main__":, replay "12:00" to "21:00" through gateway, print the report, then from the results of streamed requests (ttft_ms or complete not None):

    streams: N, ended with done or error: C

    time to first piece: p50 X ms, p95 Y ms

  4. Task 4 · interview question

    Interview: cancel.py, the customer closed the chat

    Write async def first_words(req, n): read stream(req) from gateway only until the text has n words (split on whitespace), stop reading, close the stream (aclose, or async with contextlib.aclosing(...)), and return the first n words joined with single spaces.

    Under if __name__ == "__main__":, with Thenga's first reply ({**day(features=["reply"], teams=["thenga"])[0].request, "stream": True}) and kestrel-pro's "output" from ledger() before and after each run (after settle()), print:

    whole answer: N output tokens billed (reading the stream to the end)

    first 8 words: (the words)

    stopped after 8 words: N output tokens billed (with first_words(req, 8))

What you will learn

  • Server-sent events: the text format every provider streams in.
  • Async generators: Python's way to hand on pieces as they come.
  • Three formats: where Nimbus, Kestrel and Tara put the text, the usage and the end.
  • Pieces as they come: time to first piece, and why buffering throws it away.
  • Drops: why only the provider's end marker means an answer is whole.
  • Fallback before the first piece, error after it: and a first-piece timeout.
  • Closing: stopping the provider when the reader goes away, and what it is billed.

More in Run it in production

Build "Streaming through the gateway" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.