Claude and the Anthropic SDK (TypeScript) · Chapter 10 of 21

Images

By the end of this chapter you can send Claude a photo of a bill in an image block, work out what that photo costs in tokens before you send it, turn a photo into a typed record with structured output, and fix the 400 that comes from naming the wrong image type. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Tools and structure
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Claude and the Anthropic SDK (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Families do not type their bills. They photograph them: a hospital bill on the kitchen table, a pharmacy slip held up to the light, sent on WhatsApp, three or four photos a claim. Kamla has been reading those photos and typing the amounts into Parchi's sheet by hand. Every request you have sent so far was text, and a photo is not text: you cannot paste it into a string.

Claude reads images. A photo goes into the message as its own content block, beside your question, and Claude reads the bill printed in it. That is the whole idea, and it comes with three things to get right: how the block is shaped, what a photo costs (Farhan's first question: photos are big), and the image type, which the API checks against the bytes.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    read-a-photo.ts: a bill photo

    Send 'claude-opus-5-5' (max_tokens 4096, system: SYSTEM, given) one user message whose content is two blocks, in this order: an image block with source billPhoto('bill-CLM-2412.png'), then a text block, exactly: How much will the policy pay for this bill?

    Print the text of the reply's text blocks, then input tokens: N (reply.usage.input_tokens).

  2. Task 2

    image-tokens.ts: what a photo costs

    Export imageTokens(width: number, height: number): number. If the long edge is over 1568 px, scale both sides by 1568 / long edge and round each to a whole pixel. Then return width x height / 750, rounded up.

    In the main block, read pharmacy-slip-ramesh.jpg's size with size() (given) and print size: W x H, then predicted: N tokens. Send 'claude-opus-5-5' (max_tokens 1024) the photo and the text What is this?, then the same text without the photo, and print measured: M tokens, where M is the first reply's input_tokens minus the second's.

  3. Task 3

    photo-to-record.ts: a photo into a record

    Export Bill, a zod object with: kind ('hospital_bill' or 'pharmacy_slip'), patient, hospital (string or null), date, bill_no, claim_id and doctor (each string or null), items (list of { name, amount: number }), total (number).

    Export async readBill(file: string): Promise<Bill>: one client.messages.parse request to 'claude-opus-5-5' (max_tokens 4096) with an image block of billPhoto(file), then the text Read this bill into the schema., and output_config: { format: zodOutputFormat(Bill) }. Return reply.parsed_output.

    In the main block, for bill-CLM-2412.png then pharmacy-slip-ramesh.jpg print FILE: KIND, PATIENT, N items, ₹TOTAL, claim ID, doctor NAME, with inr(total), and none for a null claim id or doctor.

  4. Task 4 · interview question

    Interview: wrong-media-type.ts

    "Kamla's upload form labels every photo image/jpeg, and some of them come back 400. Why, and how do you fix it?" Export mediaType(bytes: Buffer) that returns 'image/png', 'image/jpeg', 'image/gif' or 'image/webp' from the file's first bytes (and throws for anything else).

    In the main block, send 'claude-opus-5-5' (max_tokens 1024) bill-CLM-2412.png's bytes (given) as base64 with media_type 'image/jpeg' and the text What is this?. Catch the Anthropic.BadRequestError and print refused: STATUS TYPE. Then print detected: TYPE (mediaType of the bytes), send the same request with that type, and print the reply's text.

What you will learn

  • Image blocks: { type: 'image', source: { type: 'base64', media_type, data } }, and where they go.
  • What a photo costs: width x height / 750 tokens, after big photos are shrunk.
  • Photos into records: structured output with an image in the message.
  • The media type check, and the other 400s an image can bring.
  • A photo Claude cannot read, and what it says then.

More in Tools and structure

Build "Images" yourself

Parchi, a Jaipur startup, helps families claim health insurance. You build Sahay, a claims assistant that reads bills and policies with Claude, and measure what caching, models and effort cost. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Claude and the Anthropic SDK (TypeScript) are free. This one comes with the all labs plan.