MCP server project · Chapter 10 of 12
Descriptions and evals
By the end of this chapter you can say who reads a tool's description and how a model uses it to choose a tool, rewrite DabbaDrop's five descriptions for that reader, and measure the result with an eval: twelve messages members really write, the tool each one should get, and a score. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Ready for every assistant
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of MCP server project are free. This one comes with the all labs plan.
Why you need this
The server is safe now: a token per member, scopes, the member's yes before any payment, a trace id behind every bug. But an assistant still has to choose the right tool for what the member writes, and it chooses from the descriptions in tools/list. The developers wrote those for themselves: GET /orders/me.
Kunal collected 12 messages members really write and the tool each one should lead to (mcp_server/evals/cases.json). With the developers' descriptions, the lab model gets 6 of them right:
$ python -m mcp_server.evals
Passed 6 of 12 (50%)
"Do you have anything Jain in Dadar?": wanted find_dish, got no tool
"I want something without onion or garlic for lunch in Thane": wanted find_dish, got no tool
"Did my dabba for Monday get booked?": wanted list_my_orders, got no tool
"What did I order last week?": wanted list_my_orders, got no tool
"Use my wallet for order 2": wanted pay_order, got no tool
"Settle the bill for order 2": wanted pay_order, got no tool
"Got no tool" is worse than it looks. Asked "Do you have anything Jain in Dadar?", the model called nothing and answered from its own idea of tiffin services:
Most tiffin services make any dish Jain on request, at no extra cost.
Dadar Kitchen has no Jain dish at all (find_dish for Dadar and jain answers an empty list). Tara would order on a fasting day on the strength of a sentence nobody at DabbaDrop wrote. And "Use my wallet for order 2" got "Could you tell me how many dabbas you want?": the model reached for the wrong tool. No code you have written so far can fix this, because the model never sees your code. It sees words.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
descriptions.py: descriptions for the model
Rewrite the five DESCRIPTIONS in mcp_server/descriptions.py for the model that chooses a tool: what the tool does in the member's words, when to use it, and what it returns (a part that starts with Returns). Each has at least 15 words, no API path, no HTTP method (GET, POST) and no webhook; pay_order's says the member approves each payment in the DabbaDrop app.
The Check reads each description in tools/list and checks those rules.
- Task 2
evals/score.py: scoring the eval
Write score(cases, picks) and report_text(report) in mcp_server/evals/score.py. score raises ValueError("<n> cases but <m> picks") for lists of different lengths, counts a case as passed when the pick's tool equals the case's (None meaning no tool), and returns Report(passed, total, failures) with one line per failure, "\"<message>\": wanted <tool>, got <tool>" ("no tool" for None). report_text returns "Passed <p> of <t> (<rate:.0%>)" and then each failure on its own line after two spaces.
The Check scores made-up picks: all right, two wrong, a wrong tool where none was wanted, and lists of different lengths.
- Task 3 · interview question
Interview: descriptions.py, the eval passes
Make python -m mcp_server.evals pass at least 11 of its 12 cases with the lab model, by improving your descriptions (only descriptions.py changes). The developers' descriptions passed 6 of 12.
The Check runs the 12 cases of evals/cases.json against your server's tools/list with lab-chat (temperature 0), and counts the tools it calls first.
Kunal will ask: the eval passes with the lab model. Does that prove the descriptions work for Claude or GPT? What would you do before trusting them?
What you will learn
- Who reads a description: only the model, on every request, beside the member's message.
- How a model chooses a tool: how the lab model scores tools, and why a real model is different.
- Writing for that reader: what it does in the member's words, when to use it, what it returns.
- What an eval is, and how it differs from a test.
- Scoring an eval: passed, total, rate, and a failure line you can act on.
- Trying a real model on your own machine, with your own key.
More in Ready for every assistant
Build "Descriptions and evals" yourself
DabbaDrop wants AI assistants like Claude Desktop to order for its members, safely. You build the MCP server that lets them, and only as the member who allowed it. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of MCP server project are free. This one comes with the all labs plan.