AI engineering traps (TypeScript) · Chapter 13 of 16

Retries that hammer

Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Tools
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of AI engineering traps (TypeScript) are free. This one comes with the all labs plan.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    repro.ts: send a prompt that is too long, on purpose

    Write reproduce() in repro.ts. Make an OpenAI client that counts its requests, by handing it its own fetch (new OpenAI({ maxRetries: 0, fetch: ... }) that adds one to a counter and then calls the real fetch). Send through askWithRetry from retry.ts, with that client, a request too long for the window: model 'lab-chat' and one user message whose content is 'water ' repeated 9000 times. It fails: catch the error, and resolve to how many requests the client sent.

    The check runs your reproduce() twice: beside the original retry.ts it must resolve to 5, and beside a fixed retry.ts to 1 (a 400 is never retried). So count real requests. reproduce() must finish within 10 seconds.

  2. Task 2

    retry.ts: retry only what can succeed, and wait first

    Fix askWithRetry in retry.ts. Make at most 4 attempts in all. After a 429, any 5xx status, or a request that never got an answer (OpenAI.APIConnectionError), wait with sleep(ms) and ask again; any other error (400, 401, ...) is thrown at once, with no retry; when the 4th attempt fails too, throw its error. Wait what the error's retry-after header says, in seconds turned into milliseconds; with no such header, wait 500 ms, then 1000, then 2000.

    Keep the log line retry: attempt <n> failed for every failed attempt, and keep askWithRetry(body, client?, sleep?) as it is. The check counts the requests and every sleep, with lab-chat-flaky (a 503, then a 429 with retry-after 1, then an answer), lab-chat-down (always 503), a prompt too long for the window, a request with no API key and a connection that fails once. It also type-checks retry.ts. Waiting before every retry is not enough if a 400 is still retried.

  3. Task 3

    retry.test.ts: a test that would have caught it

    Write a vitest test file, retry.test.ts, that would have caught this retry storm. The check runs your file three times: beside the original retry.ts, where at least one test must fail; beside the fixed retry.ts, where every test must pass; and beside a wrong fix that waits with backoff but still retries a 400, where at least one test must fail.

    So test both sides, with a counting client and a fake sleep that only notes the milliseconds: a prompt too long for the window makes 1 request and no waits; 'lab-chat-flaky' (with something unique in the message) makes 3 requests with waits of 500 and 1000. Each test must finish within 10 seconds.

What you will learn

  • repro.ts: send a prompt that is too long, on purpose
  • retry.ts: retry only what can succeed, and wait first
  • retry.test.ts: a test that would have caught it

More in Tools

Build "Retries that hammer" yourself

Sunwai, a Lucknow app for housing societies, runs on a model, and Farah, its CTO, keeps assigning you tickets. Each is a trap AI engineering interviews love to ask about. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI engineering traps (TypeScript) are free. This one comes with the all labs plan.