Building with LLMs · Chapter 5 of 24

Streaming

By the end of this chapter you can show a model's reply as it is being written, put the pieces back together, read how and why it ended, and get the token counts of a streamed reply. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Talking to a model
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.

Why you need this

Zoya tried the support bot on her phone and asked for a summary of the help centre. The screen showed nothing for five seconds, then a whole paragraph at once. "I thought it had crashed," she says. Every chat app you have used starts printing the answer almost at once, word by word.

A model writes one token at a time (chapter 1), so the beginning of the reply exists long before the end. Everything you have sent so far waited for the end. Streaming asks the API to send each piece the moment it is written.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    live.py: print as it arrives

    Zoya wants to see the answer appear word by word. Ask lab-chat, as one user message:

    Is food served during the trip?

    as a stream, and print each piece of the reply the moment it arrives, all on one line, then end the line.

    The output is the whole reply on one line, nothing else.

  2. Task 2

    collect.py: put a stream back together

    Lata's code needs the whole text of a streamed reply, and to know whether it was cut off. Write a function stream_reply(messages, **options) in collect.py: it sends messages to lab-chat as a stream, passing any options on to the request (for example max_tokens=5), and returns a tuple of two strings: the whole reply's text and its finish_reason.

    The check calls it with and without max_tokens.

  3. Task 3

    timing.py: the first token and the whole reply

    Zoya asks why streaming feels faster. Send lab-chat, as a stream, a system message exactly: Summarise this policy in 8 sentences.

    and a user message holding the whole text of /workspace/data/policy.md.

    Measure, from just before the request, the seconds until the first piece of text arrives and until the stream ends, and print them with two decimals:

    first token: 0.05 s

    whole reply: 3.21 s

    (your numbers will differ). Print only these two lines.

  4. Task 4 · interview question

    Interview: fix_me.py, the broken stream

    fix_me.py should stream the answer to a question, print the reply on one line as it arrives, then print tokens: N on the next line, where N is the reply's completion tokens from the stream's usage. It has three mistakes. Run it, read the output and the error, and fix it.

    Keep the question and the request's settings as they are.

What you will learn

  • stream=True: the response becomes a stream of chunks.
  • Deltas: what each chunk adds, and putting them back together.
  • The last chunks: finish_reason, and usage with stream_options={"include_usage": True}.
  • What goes over the wire: server-sent events.
  • Time to first token: why a stream feels faster, though it is not.

More in Talking to a model

Build "Streaming" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.