RAG: search that answers · Chapter 32 of 37
When the model is down
By the end of this chapter you can keep the assistant useful when the model is slow, rate limited or down: timeouts you choose, retries with backoff and jitter that honour Retry-After, a circuit breaker, and search results when no answer is possible. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Production
- 4 tasks
- In the all labs plan
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.
Why you need this
Model providers have bad days. Requests slow down, rate limits kick in at the busiest hour, and sometimes every request fails for an hour. Kabir tried his assistant against the lab's slow model:
$ node exercise.ts
UPI payments are settled together with your card, net banking and wallet payments, on the standard settlement timeline. [help-accept-upi-payments#4]
4035 ms
Four seconds of a spinner, and it would have waited ten minutes: that is the openai client's default timeout. When the model is down, every merchant waits for their own failure and then sees an error page, although the search that found the right help articles worked perfectly. Nothing you have built so far decides how long to wait, when to try again, or what to show instead.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
complete: a timeout, and no hidden retries
complete(messages, { model, timeoutMs }) sends messages to model (MODEL unless given) and returns the reply's text. It gives up after timeoutMs (TIMEOUT_MS unless given) and turns off the client's own retries with maxRetries 0; errors are thrown as the SDK throws them.
complete(m, { model: 'lab-chat-slow', timeoutMs: 300 }) rejects with OpenAI.APIConnectionTimeoutError in about 300 ms (with the client's retries it takes several seconds), and complete(m, { model: 'lab-chat-down' }) rejects with status 503 after exactly one request to the model.
- Task 2
withRetry: retry with backoff, jitter and Retry-After
isRetryable(err) is true for an OpenAI.APIConnectionError (timeouts included) and for an OpenAI.APIError with status 408, 409, 429 or 500 and above; false for anything else, such as a 404 or a TypeError.
withRetry(fn, { retries = 3, baseMs = 200, maxMs = 10000, sleep, random = Math.random }) calls fn and returns its result. On a retryable error it waits and calls fn again, at most retries times after the first attempt, then throws the last error; a non-retryable error is thrown at once. Before retry n (from 0) it waits the error's Retry-After header in seconds times 1000 when there is one (at most maxMs), otherwise random() * min(maxMs, baseMs * 2 ** n), always through sleep. Against lab-chat-flaky with random 0.5 the waits are [100, 1000]; lab-chat-down with retries 3 and random 1 waits [200, 400, 800] and throws the 503.
- Task 3
CircuitBreaker: stop calling a model that is down
new CircuitBreaker({ failures = 3, cooldownMs = 30000, now = Date.now }), with a public state of 'closed', 'open' or 'half-open'. call(fn) returns or throws what fn did. Closed: call fn; a retryable failure (isRetryable) counts, and failures in a row open the breaker; a success sets the count back to 0; other errors are thrown without counting.
Open: throw CircuitOpenError at once, without calling fn, until cooldownMs has passed since it opened, measured with now(). Then half-open: one call goes through; a success closes the breaker, a retryable failure opens it again from now(). With lab-chat-down and failures 3, the fourth call never reaches the model; a 404 never opens the breaker.
- Task 4 · interview question
Interview: answerOrSearch, fall back to search results
answerOrSearch(question, { model, timeoutMs, circuit }) retrieves 5 chunks, then asks the model through circuit.call with complete(buildMessages(question, chunks), { model, timeoutMs }). When it works, return { kind: 'answer', answer, sources: the 5 chunk ids }.
When it fails with a retryable error or a CircuitOpenError, return { kind: 'search', results }: the first 3 chunks as { id, title, snippet }, snippet being the first 200 characters of content. Any other error, such as the 404 of a wrong model name, is thrown. With lab-chat-down and a breaker of failures 2, three merchants all get search results and the model receives only 2 requests. Arjun will ask what merchants see during an hour-long outage, and what it costs.
What you will learn
- Timeouts and the SDK's defaults: a 10 minute timeout and 2 retries you did not ask for.
- Retries done right: only for errors that can pass, with exponential backoff, jitter, and the server's Retry-After.
- A circuit breaker: stop calling a model that keeps failing.
- Graceful fallback: search results are still help.
More in Production
Build "When the model is down" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.