AI engineering traps · Chapter 13 of 16
Retries that hammer
Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Tools
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
repro.py: send a prompt that is too long, on purpose
Write reproduce() in repro.py. Make an OpenAI client that counts its requests, by handing it an httpx client with a request hook (OpenAI(max_retries=0, http_client=httpx.Client(event_hooks={"request": [sent.append]})), with sent a list). Send through ask_with_retry from retry.py, with that client, a request too long for the window: model 'lab-chat' and one user message whose content is 'water ' repeated 9000 times. It fails: catch the error, and return how many requests the client sent.
The check runs your reproduce() twice: beside the original retry.py it must return 5, and beside a fixed retry.py 1 (a 400 is never retried). So count real requests. reproduce() must finish within 10 seconds.
- Task 2
retry.py: retry only what can succeed, and wait first
Fix ask_with_retry in retry.py. Make at most 4 attempts in all. After a 429, any 5xx status, or a request that never got an answer (openai.APIConnectionError), wait with sleep(seconds) and ask again; any other error (400, 401, ...) is raised at once, with no retry; when the 4th attempt fails too, raise its error. Wait what the error's retry-after header says, in seconds; with no such header, wait 0.5 seconds, then 1.0, then 2.0.
Keep the printed line retry: attempt <n> failed for every failed attempt, and keep ask_with_retry(body, client=None, sleep=time.sleep) as it is. The check counts the requests and every sleep, with lab-chat-flaky (a 503, then a 429 with retry-after 1, then an answer), lab-chat-down (always 503), a prompt too long for the window, a request with no API key and a connection that fails once. Waiting before every retry is not enough if a 400 is still retried.
- Task 3
test_retry.py: a test that would have caught it
Write a pytest file, test_retry.py, that would have caught this retry storm. The check runs your file three times: beside the original retry.py, where at least one test must fail; beside the fixed retry.py, where every test must pass; and beside a wrong fix that waits with backoff but still retries a 400, where at least one test must fail.
So test both sides, with a counting client and a fake sleep that only notes the seconds (a list's append): a prompt too long for the window makes 1 request and no waits; 'lab-chat-flaky' (with something unique in the message) makes 3 requests with waits of 0.5 and 1.0. Each test must finish within 10 seconds.
What you will learn
- repro.py: send a prompt that is too long, on purpose
- retry.py: retry only what can succeed, and wait first
- test_retry.py: a test that would have caught it
More in Tools
Build "Retries that hammer" yourself
Sunwai, a Lucknow app for housing societies, runs on a model, and Farah, its CTO, keeps assigning you tickets. Each is a trap AI engineering interviews love to ask about. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.