The Vercel AI SDK · Chapter 12 of 21

Photos as attachments

By the end of this chapter a customer can attach a photo of a sick plant in Mali's chat, see it in the conversation, and get an answer about what is in it. You will make the route check every file before the model sees it, add a photo picker and thumbnails to the chat, and decide what to do with old photos as a conversation grows. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The chat UI
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of The Vercel AI SDK are free. This one comes with the all labs plan.

Why you need this

Half of what reaches Deepa's team on WhatsApp is a photo with a question: yellow leaves, brown tips, a pot that arrived in pieces. "What's wrong with it?" is easy to answer when you can see it, and impossible from words alone. Customers do not know what a Calathea is; they know what their plant looks like.

Mali's chat takes only text. Ananya wants a camera button. Models that can see (most current ones) take images as part of a message, and the AI SDK has a place for them all the way through: in useChat, in the UI message, in the route and in the request to the model. Farhan adds one condition: the route decides what reaches the model, not the browser. A photo is a file someone else sent your server.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    server.ts: the route checks every photo

    Before streamText, check every file part of every message: its mediaType is image/png, image/jpeg or image/webp, its url is a data: URL, and its data, decoded from base64, is at most 2 MB (2 * 1024 * 1024 bytes). If any file fails, answer 400 with the text Please attach a photo (PNG, JPEG or WebP) up to 2 MB. and do not call the model. Good photos go on to the model as they are.

    The Check sends a photo (its exact bytes must reach the model, and Mali's answer must stream), a PDF, a PNG 10 bytes over 2 MB, a PNG of exactly 2 MB (allowed), a text file in an earlier message, and a message with no file.

  2. Task 2

    Chat.tsx: attach a photo, and show it

    Add to chapter 9's chat a file picker (an input of type file) whose picked files are kept in state and sent with sendMessage({ text, files }); refuse a picked file that is not image/png, image/jpeg or image/webp, or is over 2 MB, by keeping nothing and showing Please attach a photo (PNG, JPEG or WebP) up to 2 MB. (cleared when a message is sent); and show every image file part of a message as an img with src part.url and alt part.filename.

    The Check attaches yellow-leaves.png (its bytes must be sent, its thumbnail shown, and Mali's answer appear), then picks a PDF and a 2 MB photo: the rule must show, and neither may be sent.

  3. Task 3 · interview question

    Interview: slim.ts, old photos in the history

    Export dropOldPhotos(messages), which returns new messages where every file part in every user message except the last becomes a text part [photo: FILENAME], leaving everything else as it was and the messages it got unchanged. Under if (import.meta.main), print the size of the given chat as model messages (JSON.stringify(await convertToModelMessages(chat)).length) before and after:

    with the old photo: N bytes

    without it: N bytes

    Vikram will ask: what does Mali lose when the old photo becomes a note, and when is that a bad trade?

What you will learn

  • File parts: how a photo travels inside a message, as a data: URL.
  • sendMessage({ text, files }): useChat reads the picked files for you.
  • What the model receives: a file part becomes an image in the request.
  • Checking files on the server: type, size, and where the bytes come from.
  • Showing photos in the chat, and refusing bad files on the page.
  • Photos in the history: what every later turn carries.

More in The chat UI

Build "Photos as attachments" yourself

HariBhari, a Bengaluru plant shop, wants Mali, a plant helper on its website. You build it end to end, from the first call to a chat that streams cards, asks before it adds to the cart and stays within budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of The Vercel AI SDK are free. This one comes with the all labs plan.