Building with LLMs (TypeScript) · Chapter 6 of 24

Errors and retries

By the end of this chapter you can tell an error worth retrying from one that never will be, retry with backoff and jitter, put a timeout on every call, and fall back to another model instead of leaving a customer waiting. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Talking to a model
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Last Saturday evening, the busiest hour of SafarLine's week, the model provider was overloaded. For twenty minutes, one call in three failed. The drafting tool crashed on the first failure and showed Lata a stack trace, and the queue grew by two hundred messages while she answered them by hand. Another call did not fail at all: it simply never came back, and the tool waited with it.

On your own laptop a function either returns or throws, and running it again gives the same result. A model is a service on someone else's servers, over a network, shared with thousands of other customers. Some of its failures pass in a second, some never pass, and some look like nothing at all. Zoya wants the tool to survive a bad Saturday: retry what is worth retrying, give up quickly on what is not, and never wait forever.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    no-retry.ts: see a raw failure

    Nikhil wants to see what the provider really sends when it fails. Ask lab-chat-down, as one user message:

    Is my Pune to Nashik bus running today?

    with the openai package's own retries turned off, so exactly one request goes out. It will fail with a 503.

    Catch the error and print one line: its HTTP status code, a colon, a space, and the API's own message (the message inside the error's body, not the error's whole message). The program must not crash.

  2. Task 2

    backoff.ts: how long to wait

    Write and export a function waitTime(attempt: number): number that returns how many milliseconds to wait before the next try, with exponential backoff and full jitter.

    attempt counts the failures so far, starting at 0. The ceiling is 500 milliseconds doubled for each attempt (500, 1000, 2000, 4000...), but never more than 8000.

    Return a random number of milliseconds between 0 and that ceiling, different on each call. The function only computes: it does not wait.

  3. Task 3

    retry.ts: a retry loop of your own

    Write and export askWithRetry(client: OpenAI, model: string, messages: OpenAI.ChatCompletionMessageParam[], attempts = 4): Promise<string | null> that sends the messages to that model with client.chat.completions.create and returns the reply's text.

    The check passes a client whose own retries are off. When a call fails with a rate limit (429), a server error (500 and above), a timeout or a connection error, try again, up to attempts requests in all.

    Before each new try, wait the seconds in the error's retry-after header when it has one, otherwise waitTime from your backoff.ts (in milliseconds).

    Never retry any other error, such as a 404 for a wrong model name: throw it at once. When the last attempt fails, throw that last error.

  4. Task 4 · interview question

    Interview: fallback.ts, answer within two seconds

    Write and export answer(client: OpenAI, messages: OpenAI.ChatCompletionMessageParam[], primary = 'lab-chat-slow', backup = 'lab-chat'): Promise<string | null> that returns the reply's text, always within 2 seconds.

    Ask the primary model first, with a timeout of 1 second and no retries. If it times out, cannot be reached, is rate limited or fails with a server error, ask the backup model instead and return its reply.

    When the primary answers, return its reply and do not call the backup. Do not catch other errors: a bad request is a bug, not a reason to fall back.

What you will learn

  • The four kinds of failure, and the openai error classes for each.
  • What the package already does: maxRetries and timeout, and why their defaults are not always what you want.
  • Backoff with jitter: how long to wait between attempts, and why the wait is random.
  • Falling back: a second model, so a customer still gets an answer.

More in Talking to a model

Build "Errors and retries" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant, in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.