Building with LLMs · Chapter 7 of 24

Async and concurrency

By the end of this chapter you can call a model with AsyncOpenAI, send many requests at once with asyncio.gather, cap how many are in flight with a semaphore, and keep a batch going when one request fails. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Talking to a model
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.

Why you need this

Every morning Lata opens WhatsApp to the messages that came in overnight: two hundred and forty of them on a weekend. Zoya wants a draft answer ready for each one before the support desk opens at six.

The drafting script you could write today asks the model about one message, waits for the reply, then asks about the next. With a real provider a reply takes one to five seconds, so 240 messages at 3 seconds each is twelve minutes, and almost all of it is your program doing nothing: the request is on the network or the provider's machines are writing the answer. Your computer could be waiting on many replies at once. You already know the tool for that from Python, async and await: this chapter points it at a model, and deals with what goes wrong when you send too much at once.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    first_async.py: one call, awaited

    Ask lab-chat one user message, exactly:

    What is UPI?

    using AsyncOpenAI, and print only the text of its reply. The program runs its async code with asyncio.run.

  2. Task 2

    gather.py: five questions at once

    Lata wants five answers fast. Ask lab-chat-slow, which takes 4 seconds per reply, these five questions, each as its own one-message request:

    Can I cancel my ticket?

    How much luggage can I take?

    Can I bring my cat?

    Is my bus late?

    How early should I reach the boarding point?

    Send them at the same time, so the whole program takes about 4 seconds, not 20. Print the five replies, one per line, in the order of the questions above.

  3. Task 3

    limited.py: at most limit at a time

    Write async def answer_all(client, questions, limit) that asks lab-chat each question (each as its own one-message request, using the AsyncOpenAI client it is given) and returns the list of reply texts, in the order of the questions.

    Never have more than limit requests in flight at once, and do use all limit: with 6 questions and a limit of 2, two requests are always in flight until the last ones.

    The check counts the requests in flight, with limits of 1, 2 and 4.

  4. Task 4 · interview question

    Interview: batch.py, one bad request must not cost the rest

    Write async def answer_batch(client, conversations, limit=3). Each conversation is a list of messages; send each one to lab-chat as its own request, with the AsyncOpenAI client it is given, with at most limit requests in flight.

    Return a list with one entry per conversation, in order: the reply's text, or None when that request failed.

    Some conversations the check sends are broken (the API answers them with a 400). answer_batch must not raise: every good conversation still gets its reply.

What you will learn

  • AsyncOpenAI: the same API, awaited.
  • asyncio.gather: many requests in flight, the replies in order.
  • A semaphore: at most N requests at once, to stay under a rate limit.
  • Failures in a batch: one bad request must not cost you the rest.

More in Talking to a model

Build "Async and concurrency" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.