The Vercel AI SDK · Chapter 17 of 21
Providers and fallbacks
By the end of this chapter you can choose Mali's models in one place and keep Mali answering when a provider fails. You will give the models short names of your own, look them up by a string, ask the same model through two different providers, meet the two OpenAI APIs, fall back to a second model when the first is down, and price one chat on two models. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Ready for customers
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of The Vercel AI SDK are free. This one comes with the all labs plan.
Why you need this
Mali is nearly ready for customers, and Vikram, who looks after costs and uptime, has two questions before Ananya switches it on. First: "Model names are written all over our code. When we move a job to a cheaper model, how many files do we change?" Second: "Last Diwali our payment provider went down for forty minutes. What does Mali do when its model provider goes down?"
So far every file says lab('lab-chat'). Changing a model means finding every one of them. And when the provider answers 503, the call fails: the SDK tries three times, waits about 6 seconds, then throws, and the customer sees nothing. In Diwali week, that is hundreds of customers.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
providers.ts: name the models once
Export mali, a customProvider (from 'ai') with two languageModels: 'mali-fast' is lab('lab-chat-mini') and 'mali-smart' is lab('lab-chat') (lab from ../../shared/mali.ts), and no fallbackProvider, so any other name fails with a NoSuchModelError. Export registry, createProviderRegistry({ lab, mali }), so 'mali:mali-fast' and 'lab:lab-chat-large' name models.
Under if (import.meta.main), ask Do you replace broken plants? with MALI_INSTRUCTIONS through registry.languageModel('mali:mali-fast'), then 'mali:mali-smart', and print one line each:
ID (MODEL): ANSWER
with MODEL from result.response.modelId.
- Task 2
two-apis.ts: one model, two providers
Make an OpenAI provider with createOpenAI (from @ai-sdk/openai): baseURL process.env.OPENAI_BASE_URL, apiKey process.env.OPENAI_API_KEY. Ask What does shipping cost? with MALI_INSTRUCTIONS through lab('lab-chat'), then through openai.chat('lab-chat'), and print:
openai-compatible: FIRST ANSWER
openai chat: SECOND ANSWER
same: true or false
Then ask through openai('lab-chat'), catch the error, and when APICallError.isInstance(error), print openai responses: STATUS MESSAGE (error.statusCode and error.message).
- Task 3
fallback.ts: a backup when a provider is down
Export withFallback(primary, backup): it returns wrapLanguageModel({ model: primary, middleware }) with a middleware (specificationVersion 'v4') whose wrapGenerate returns await doGenerate(), or, when that throws an APICallError with isRetryable true, backup.doGenerate(params); and whose wrapStream does the same with doStream. Any other error is thrown again.
The Check asks through withFallback(lab('lab-chat-down'), lab('lab-chat-mini')) with generateText and with streamText, and wants one 503, then one answer from mini, quickly; a working primary used alone; and a 400 thrown, not sent to the backup.
- Task 4 · interview question
Interview: cost-per-model.ts, the same chat on two models
Have one three-turn chat with Mali (MALI_INSTRUCTIONS, keeping the history) on lab('lab-chat'), then the same chat, with a new history, on lab('lab-chat-mini'): Do you take cash? then And UPI? then What if my plant arrives damaged?
Print each model's three turns added up, with lab-chat at ₹200 in and ₹800 out and lab-chat-mini at ₹15 in and ₹60 out, a million tokens, rupees with 4 decimals:
MODEL: IN in, OUT out, ₹COST
Then lab-chat-mini costs N times less, N being lab-chat's cost over mini's, with 1 decimal.
Vikram will ask: mini gives the same answers here for far less, so why not use it for everything?
What you will learn
- customProvider: your own names for models (mali-fast, mali-smart).
- createProviderRegistry: one place that turns 'provider:model' strings into models.
- @ai-sdk/openai and the two OpenAI APIs: Chat Completions and Responses.
- A fallback: a middleware that asks a second model when the first one fails.
- Cost per model: what the same chat costs on a big model and a small one.
More in Ready for customers
Build "Providers and fallbacks" yourself
HariBhari, a Bengaluru plant shop, wants Mali, a plant helper on its website. You build it end to end, from the first call to a chat that streams cards, asks before it adds to the cart and stays within budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of The Vercel AI SDK are free. This one comes with the all labs plan.