Building with LLMs (TypeScript) · Chapter 7 of 24

Async and concurrency

By the end of this chapter you can send many model requests at once with Promise.all, write a limiter that keeps at most N of them in flight, and keep a batch going when one request fails with Promise.allSettled. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Talking to a model
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Every morning Lata opens WhatsApp to the messages that came in overnight: two hundred and forty of them on a weekend. Zoya wants a draft answer ready for each one before the support desk opens at six.

The drafting script you could write today loops over the messages: ask the model about one, await the reply, then ask about the next. With a real provider a reply takes one to five seconds, so 240 messages at 3 seconds each is twelve minutes, and almost all of it is your program doing nothing: the request is on the network, or the provider's machines are writing the answer. Your program could be waiting on many replies at once. You have used await on every call since chapter 1; this chapter is about where you put it, and about what goes wrong when you send too much at once.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    all-at-once.ts: five questions at once

    Lata wants five answers fast. Ask lab-chat-slow, which takes 4 seconds per reply, these five questions, each as its own one-message request:

    Can I cancel my ticket?

    How much luggage can I take?

    Can I bring my cat?

    Is my bus late?

    How early should I reach the boarding point?

    Send them at the same time, so the whole program takes about 4 seconds, not 20. Print the five replies, one per line, in the order of the questions above.

  2. Task 2

    limiter.ts: a limiter of your own

    Write and export makeLimiter(limit: number): <T>(task: () => Promise<T>) => Promise<T>.

    It returns a function that takes a task (a function that starts some work and returns a promise) and returns a promise of the task's result. However many tasks it is given, at most limit of them run at once: the others wait, and each starts as soon as a running one finishes.

    A task that fails must reject its promise with its own error and still free its slot. The check runs many short timer tasks through limits of 1, 2 and 3, with no model.

  3. Task 3

    limited.ts: stay under the rate limit

    Write and export answerAll(client: OpenAI, questions: string[], limit: number): Promise<(string | null)[]> that asks lab-chat-limited each question (each as its own one-message request, using the client it is given) and returns the reply texts, in the order of the questions.

    lab-chat-limited answers at most 3 requests at a time: a 4th in flight gets a 429. Never have more than limit requests in flight at once, and do use all of limit: with 6 questions and a limit of 2, two requests are always in flight until the last ones.

    Use your makeLimiter from limiter.ts. The check passes a client with retries off, which counts the requests in flight, with limits of 3, 1 and 2.

  4. Task 4 · interview question

    Interview: batch.ts, one bad request must not cost the rest

    Write and export answerBatch(client: OpenAI, conversations: OpenAI.ChatCompletionMessageParam[][], limit = 3): Promise<(string | null)[]>. Each conversation is an array of messages; send each one to lab-chat as its own request, with the client it is given, with at most limit requests in flight.

    Return an array with one entry per conversation, in order: the reply's text, or null when that request failed.

    Some conversations the check sends are broken (the API answers them with a 400). answerBatch must not throw: every good conversation still gets its reply.

What you will learn

  • Why await in a loop is slow, and how Promise.all waits on many requests at once, with the replies in order.
  • A limiter written by hand: at most N requests in flight, to stay under a rate limit.
  • Failures in a batch: Promise.allSettled, so one bad request does not cost you the rest.

More in Talking to a model

Build "Async and concurrency" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant, in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.