RAG in Python · Chapter 32 of 37

When the model is down

By the end of this chapter you can keep the assistant useful when the model is slow, rate limited or down: a timeout you choose, retries with backoff and jitter that honour Retry-After, a circuit breaker, and search results when no answer is possible. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Production
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.

Why you need this

Model providers have bad days. Requests slow down, rate limits kick in at the busiest hour, and sometimes every request fails for an hour. Kabir tried the assistant against the lab's slow model:

$ python3 kabir.py
UPI payments are settled together with your card, net banking and wallet payments, on the standard settlement timeline. [help-accept-upi-payments#4]
4.02 s

Four seconds of a spinner, and the client would have waited ten minutes. When the model is down, every merchant waits for their own failure and then sees an error page, although the search that found the right help articles worked perfectly. Nothing you have built so far decides how long to wait, when to try again, or what to show instead.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    failures.py: complete, a timeout and no hidden retries

    Kabir's merchants waited four seconds for the slow model, and the client would have waited ten minutes. Write complete(messages, model=MODEL, timeout=TIMEOUT) in failures.py: send messages to model and return the reply's text ("" when the content is None). Give up after timeout seconds, and turn the client's own retries off (max_retries 0), with client.with_options. Catch nothing: errors go out as the openai package raises them.

    complete(m, "lab-chat-slow", 0.3) raises openai.APITimeoutError in about 0.3 s (with the client's retries it takes over two seconds), and complete(m, "lab-chat-down") raises openai.InternalServerError after exactly one request to the model.

  2. Task 2

    failures.py: with_retry, backoff, jitter and Retry-After

    Write is_retryable(err): True for an openai.APIConnectionError (timeouts are one) and for an openai.APIStatusError whose status_code is 408, 409, 429 or 500 and above; False for anything else, such as a 404, a 400 or a TypeError.

    Write with_retry(fn, retries=3, base=0.2, cap=10.0, sleep=time.sleep, rand=random.random): call fn() and return what it returns. On a retryable error, wait and call fn again, at most retries times after the first try, then raise the last error; raise any other error at once. Before retry n (from 0), wait the error's retry-after header in seconds when it has one (at most cap), otherwise rand() * min(cap, base * 2 ** n), always through sleep.

    Against lab-chat-flaky with rand 0.5 the waits are [0.1, 1.0] and the answer comes back; lab-chat-down with retries 3 and rand 1 waits [0.2, 0.4, 0.8] and raises the 503; a wrong model name is sent once, with no wait.

  3. Task 3

    failures.py: CircuitBreaker, stop calling a model that is down

    The fields of CircuitBreaker(failures=3, cooldown=30.0, clock=time.monotonic) are given: state ("closed", "open" or "half-open"), count and opened_at. Write its call(fn), which returns or raises what fn did.

    Closed: call fn; a retryable failure (is_retryable) adds to count, and failures in a row open the breaker, with opened_at = clock(); a success sets count back to 0; other errors are raised without counting. Open: raise CircuitOpenError at once, without calling fn, until cooldown seconds have passed since it opened; then it is half-open, and one call goes through: a success closes it, a retryable failure opens it again from now.

    With lab-chat-down and failures 3, the fourth and fifth calls never reach the model. A 404 never opens the breaker.

  4. Task 4 · interview question

    Interview: failures.py, answer_or_search, fall back to search results

    Write answer_or_search(conn, question, breaker, model=MODEL, timeout=TIMEOUT, table="chunks"): retrieve 5 chunks, then ask the model through breaker.call with your complete(build_messages(question, hits), model, timeout). When it works, return Answer(text=the reply, sources=the 5 chunk ids).

    When it fails with a retryable error or a CircuitOpenError, return SearchResults(results=...): the first 3 chunks as Result(id, title, snippet=the first 200 characters of the chunk's text). Any other error, such as the 404 of a wrong model name, is raised. With lab-chat-down and CircuitBreaker(failures=2), three merchants all get search results and the model receives only 2 requests; a timeout falls back too.

    Arjun will ask: the provider is down for an hour, so what do merchants see, and what does it cost?

What you will learn

  • Timeouts and the client's defaults: a 600 second timeout and 2 retries you did not ask for.
  • Retries done right: only for errors that can pass, with exponential backoff, jitter, and the server's Retry-After.
  • A circuit breaker: stop calling a model that keeps failing.
  • A fallback: search results are still help.

More in Production

Build "When the model is down" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.