The Vercel AI SDK · Chapter 15 of 21

Middleware

By the end of this chapter you can change what happens around every model call in one place, without touching the routes that make the calls. You will write a logging middleware that prints each call's model, tokens and time, a cache that answers a repeated question without asking the model again, a cache for streams that still streams, and you will wrap Mali's model in all of them, in the right order. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ready for customers
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of The Vercel AI SDK are free. This one comes with the all labs plan.

Why you need this

Mali is live on HariBhari's website, and two people have new questions.

Vikram, who looks after costs and uptime, wants to see every model call in the server log: which model, how many tokens, how long it took. When a bill looks high, he wants to know which calls made it. Deepa, who leads customer care, has noticed that a few questions arrive hundreds of times a day, word for word: "What time do you deliver on Sundays?", "Can I pay by UPI?". Every one is a model call HariBhari pays for, and every one gets the same answer.

You could add a console.log after every streamText in the route, and an if around it that looks in a Map first. Then chapter 13's route, chapter 14's route that saves chats, and every route after them need the same lines, and they drift apart. Farhan, your mentor, says this is what middleware is for: code that wraps the model itself, so every call through that model gets it, and the routes stay as they are.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    log.ts: log every model call

    Export logMiddleware, a LanguageModelMiddleware (specificationVersion 'v4') that prints one line for every call that reaches the model, and changes nothing else.

    wrapGenerate awaits doGenerate(), prints [MODEL] generate: IN in, OUT out, MS ms and returns the result. wrapStream passes every part of the stream on through a TransformStream, and prints [MODEL] stream: IN in, OUT out, MS ms when the finish part passes. MODEL is model.modelId; IN and OUT come from the usage (inputTokens.total and outputTokens.total).

    The Check wraps lab-chat-mini and lab-chat with your middleware, and wants one line per call with the model's own token counts, and the same answer, still streaming.

  2. Task 2

    cache.ts: answer a repeated call from a cache

    Export cacheMiddleware, a middleware whose wrapGenerate keeps each result in a Map under the key JSON.stringify({ model: model.modelId, params }), and returns the stored result when the same key comes again, without calling the model.

    The Check asks the same question twice (0 model requests the second time, the same answer), then with other instructions, then on lab-chat-mini, and wants each of those to reach the model.

  3. Task 3

    stream-cache.ts: cache a stream, replay it as a stream

    Export streamCacheMiddleware, whose wrapStream records the parts of a stream (a TransformStream that pushes each part into a list and passes it on, and stores the list under the key in flush()), and on a hit returns simulateReadableStream({ chunks, chunkDelayInMs: 5 }) instead of calling the model. The key is made as in task 2.

    The Check streams a long care guide twice: the second time 0 model requests, the same text in the same number of pieces, spread out in time, not all at once.

  4. Task 4 · interview question

    Interview: mali.ts, the order of middlewares

    Export maliModel: lab('lab-chat') wrapped with defaultSettingsMiddleware (temperature 0.3, maxOutputTokens 500), cacheMiddleware, streamCacheMiddleware and logMiddleware, in an order that makes the log print one line for every request that reaches the model and none for a cached answer.

    The Check asks three questions, one twice, and streams one answer twice: every request carries temperature 0.3 and max_tokens 500, and the log has as many lines as there were model requests.

    Vikram will ask: three questions, one asked twice. How many log lines, and why does the order decide it?

What you will learn

  • What a middleware is: a model wrapped in your code, with wrapLanguageModel.
  • Its three hooks: transformParams, wrapGenerate and wrapStream.
  • What the model is really sent (params), and the parts of a stream.
  • Logging a call and a stream, without slowing the stream down.
  • Caching a whole answer, and a stream you can replay as a stream.
  • defaultSettingsMiddleware, and why the order of middlewares matters.

More in Ready for customers

Build "Middleware" yourself

HariBhari, a Bengaluru plant shop, wants Mali, a plant helper on its website. You build it end to end, from the first call to a chat that streams cards, asks before it adds to the cart and stays within budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of The Vercel AI SDK are free. This one comes with the all labs plan.