APIs with FastAPI · Chapter 24 of 26
Serving AI
By the end of this chapter you can put a language model behind a FastAPI route: ask it for structured output and validate what comes back, keep the API answering when the model fails or is slow, stream its answer to the client as it is written, and say what each request cost. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Production
- 4 tasks
- FastAPI
- In the all labs plan
The first 2 chapters of APIs with FastAPI are free. This one comes with the all labs plan.
Why you need this
Sunita has a pile of parcels that came back. Not lost, not damaged: undeliverable, because the address was typed into a WhatsApp chat by a customer in a hurry. "flat 12 gulmohar resi nr birla mandir mlv ngr jpr, ph 94140 55667". The rider reached Malviya Nagar and phoned around for an hour. Imran hears the same from merchants: their checkout has one big address box, and customers paste whatever they have.
DakiyaGo wants a route that takes that text and answers an address in parts (name, phone, house, landmark, area, city, state, pincode), so the booking can be checked before a rider leaves. No regular expression reads "mlv ngr" as Malviya Nagar; a language model does. But a model is a partner API like the ones in chapter 15, only stranger: it answers in words unless told otherwise, it is sometimes wrong, sometimes down, sometimes slow, and every call costs money. Aman: "The model is the easy part. The API around it is the work."
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
parse.py: an address from a WhatsApp message
Customers type addresses into WhatsApp in a hurry. Make an app with one route: POST /addresses/parse takes a JSON body with one field, text (1 to 500 characters; anything else answers 422 without asking the model), and sends it to lab-chat as the user's message, with your instructions as the system message, asking for structured output: an Address with name, phone, line1, landmark, area, city, state and pincode, each a string or null.
It answers {"address": {...those eight parts, exactly as the model gave them...}, "serviceable": ..., "tokens": ...}: serviceable is true only when the pincode is in /workspace/data/pincodes.json with active true, and tokens is the total tokens the model counted for the call. The check sends a served pincode, one DakiyaGo does not serve now, and an address with no pincode.
- Task 2
fallback.py: when the model fails
The same route as task 1, but the model's name comes from the environment variable DAKIYAGO_MODEL (default lab-chat), read once when the app starts, and the answer has a fourth key, source: "model" when the model answered.
When the model fails (lab-chat-down answers 503) or answers something that is not a valid address (lab-chat-sloppy fences its JSON), answer 200 anyway with source "fallback", tokens 0, and an address from a rule: pincode is the first run of exactly six digits in the text (or null), city and state are that pincode's from pincodes.json (or null), every other part null; serviceable as in task 1. Ask the model at most twice per address.
- Task 3
ask.py: stream the answer
Imran wants an assistant on the merchant dashboard that answers price questions as it types. Make an app with POST /ask taking {"question": ...} (1 to 300 characters, else 422 without asking the model). It asks lab-chat with stream=True: a system message holding DakiyaGo's rate card from /workspace/data/rate_card.json written as plain sentences (every zone with its first-slab price, its extra-slab price and its days, and the cash on delivery fee), and the question as the user's message.
It answers text/event-stream: one event per piece the model streams, data: {"text": "<the piece>"}, then a last event, data: [DONE]. The check compares your pieces with what the model streamed, asks how many days the state zone takes, and times the events over real HTTP: they must arrive as the model writes them.
- Task 4 · interview question
Interview: slow.py, when the model is slow
Last night the model took 40 seconds per answer, and every merchant's request hung, health checks too. Make an app with GET /health answering {"status": "ok"} and POST /addresses/parse as in task 1 (same body, same answer), with the model's name from DAKIYAGO_MODEL (default lab-chat).
When the model has not answered within about 2 seconds, answer 504 with a JSON detail telling the merchant to try again and a Retry-After header in seconds, after one request to the model. While a request waits for the model, GET /health must answer at once. The check runs your app with uvicorn and lab-chat-slow, which takes 4 seconds. The interviewer will ask: what does the API do when the model is slow, for this request and for everyone else's?
What you will learn
- Calling a model with the openai package, and what comes back.
- Structured output: a JSON schema from a Pydantic class, and why you validate anyway.
- When the model fails: errors, retries and a fallback.
- When the model is slow: timeouts, and keeping the event loop free.
- Streaming tokens over server-sent events, and cost per request.
More in Production
Build "Serving AI" yourself
DakiyaGo, a Jaipur courier startup, takes bookings on WhatsApp. Merchants want an API, and you build it. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of APIs with FastAPI are free. This one comes with the all labs plan.