MCP server project (TypeScript) · Chapter 10 of 12

Descriptions and evals

By the end of this chapter you can say who reads a tool's description and how a model uses it to choose a tool, rewrite Parda's six descriptions for that reader, and measure the result with an eval: twelve messages members really write, the tool each one should get, and a score. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ready for every assistant
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of MCP server project (TypeScript) are free. This one comes with the all labs plan.

Why you need this

The server is safe now: a token per member, scopes, the member's yes before any payment, a trace id behind every bug. But an assistant still has to choose the right tool for what the member writes, and it chooses from the descriptions in tools/list. The developers wrote those for themselves: GET /bookings/me.

Datta's support team collected 12 messages members really write, and the tool each one should lead to (src/mcp/evals/cases.json). With the developers' descriptions, the lab model gets 7 of them right:

$ npm run evals
Passed 7 of 12 (58%)
  "Any Marathi plays on Friday?": wanted find_showtimes, got no tool
  "Can two of us sit together in the balcony for showtime 18?": wanted see_seats, got no tool
  "Confirm the seats I am holding for showtime 18": wanted book_seats, got see_seats
  "Did my tickets for Saturday get confirmed?": wanted list_my_bookings, got find_showtimes
  "What did I book last week?": wanted list_my_bookings, got no tool

"Got no tool" is worse than it looks. Asked "Any Marathi plays on Friday?", the model called nothing and answered:

Generally, that depends on the venue and the show. Please contact support for the details.

Parda has one: Ek Hota Gaon, a Marathi play at Rangdeep Studio Theatre, Fri 6 Nov, 7:30 pm. The member is sent to Datta's phone line for an answer find_showtimes already has. And "Confirm the seats I am holding" got the seat map instead of a booking, while the hold's 10 minutes run out. No code you have written so far can fix this, because the model never sees your code. It sees words.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    descriptions.ts: descriptions for the model

    Rewrite the six DESCRIPTIONS in src/mcp/descriptions.ts for the model that chooses a tool: what the tool does in the member's words, when to use it, and what it returns (a part that starts with Returns). Each has at least 15 words, no API path, no HTTP method (GET, POST) and no webhook; pay_booking's says the member approves each payment in the Parda app.

    The Check reads each description in tools/list and checks those rules.

  2. Task 2

    evals/score.ts: scoring the eval

    Write score(cases, picks) and reportText(report) in src/mcp/evals/score.ts. score throws Error("<n> cases but <m> picks") for lists of different lengths, counts a case as passed when the pick's tool equals the case's (null meaning no tool), and returns { passed, total, failures } with one line per failure, "\"<message>\": wanted <tool>, got <tool>" ("no tool" for null). reportText returns "Passed <p> of <t> (<rate>%)", the rate rounded (0 with no cases), then each failure on its own line after two spaces.

    The Check scores made-up picks: all right, two wrong, a tool where none was wanted, and lists of different lengths.

  3. Task 3 · interview question

    Interview: descriptions.ts, the eval passes

    Make npm run evals pass at least 11 of its 12 cases with the lab model, by improving your descriptions (only descriptions.ts changes). The developers' descriptions passed 7 of 12.

    The Check runs the 12 cases of evals/cases.json against your server's tools/list with lab-chat (temperature 0), and counts the tools it calls first.

    Nikhil will ask: the eval passes with the lab model. Does that prove the descriptions work for Claude or GPT? What would you do before trusting them?

What you will learn

  • Who reads a description: only the model, on every request, beside the member's message.
  • How a model chooses a tool: how the lab model scores tools, and why a real model is different.
  • Writing for that reader: what it does in the member's words, when to use it, what it returns.
  • What an eval is, and how it differs from a test.
  • Scoring an eval: passed, total, rate, and a failure line you can act on.
  • Trying a real model on your own machine, with your own key.

More in Ready for every assistant

Build "Descriptions and evals" yourself

Parda wants AI assistants like Claude Desktop to book seats for its members, safely. You build the MCP server that lets them, and only as the member who allowed it. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of MCP server project (TypeScript) are free. This one comes with the all labs plan.