AI engineering traps · Chapter 13 of 16

Retries that hammer

Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Tools
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    repro.py: send a prompt that is too long, on purpose

    Write reproduce() in repro.py. Make an OpenAI client that counts its requests, by handing it an httpx client with a request hook (OpenAI(max_retries=0, http_client=httpx.Client(event_hooks={"request": [sent.append]})), with sent a list). Send through ask_with_retry from retry.py, with that client, a request too long for the window: model 'lab-chat' and one user message whose content is 'water ' repeated 9000 times. It fails: catch the error, and return how many requests the client sent.

    The check runs your reproduce() twice: beside the original retry.py it must return 5, and beside a fixed retry.py 1 (a 400 is never retried). So count real requests. reproduce() must finish within 10 seconds.

  2. Task 2

    retry.py: retry only what can succeed, and wait first

    Fix ask_with_retry in retry.py. Make at most 4 attempts in all. After a 429, any 5xx status, or a request that never got an answer (openai.APIConnectionError), wait with sleep(seconds) and ask again; any other error (400, 401, ...) is raised at once, with no retry; when the 4th attempt fails too, raise its error. Wait what the error's retry-after header says, in seconds; with no such header, wait 0.5 seconds, then 1.0, then 2.0.

    Keep the printed line retry: attempt <n> failed for every failed attempt, and keep ask_with_retry(body, client=None, sleep=time.sleep) as it is. The check counts the requests and every sleep, with lab-chat-flaky (a 503, then a 429 with retry-after 1, then an answer), lab-chat-down (always 503), a prompt too long for the window, a request with no API key and a connection that fails once. Waiting before every retry is not enough if a 400 is still retried.

  3. Task 3

    test_retry.py: a test that would have caught it

    Write a pytest file, test_retry.py, that would have caught this retry storm. The check runs your file three times: beside the original retry.py, where at least one test must fail; beside the fixed retry.py, where every test must pass; and beside a wrong fix that waits with backoff but still retries a 400, where at least one test must fail.

    So test both sides, with a counting client and a fake sleep that only notes the seconds (a list's append): a prompt too long for the window makes 1 request and no waits; 'lab-chat-flaky' (with something unique in the message) makes 3 requests with waits of 0.5 and 1.0. Each test must finish within 10 seconds.

What you will learn

  • repro.py: send a prompt that is too long, on purpose
  • retry.py: retry only what can succeed, and wait first
  • test_retry.py: a test that would have caught it

More in Tools

Build "Retries that hammer" yourself

Sunwai, a Lucknow app for housing societies, runs on a model, and Farah, its CTO, keeps assigning you tickets. Each is a trap AI engineering interviews love to ask about. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.