Building with LLMs (TypeScript) · Chapter 8 of 24
Cost and latency
By the end of this chapter you can measure what one call and one feature cost in rupees and in milliseconds, read cached tokens on the bill, trim a prompt to what a feature needs, and decide with numbers whether a cheaper model will do. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Talking to a model
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.
Why you need this
The drafting tool has been running for a week, and Harpreet from finance has the first bill. He does not want the total: he wants to know which feature costs what, what one more customer message costs, and what next month will cost when Zoya turns the tool on for all 240 messages a day. Lata has a question of her own: customers say the bot is slow. How slow?
A function you write in TypeScript costs nothing to run again, and takes the time your computer takes. A model call is billed by the token, by someone else, and takes the provider's time. Chapter 3 priced one request. This chapter measures a whole feature, finds where its tokens go, and cuts them.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
cost.ts: what a call costs
Write and export costInr(usage: OpenAI.CompletionUsage, price: Price): number that returns what one call cost, in rupees, as a number (not rounded).
usage is a response's usage. Price is a type you write and export: rupees per million tokens, as the lab's models list reports it, for example { input: 200, cached_input: 50, output: 800 }.
Cached prompt tokens are billed at the cached_input price, the rest of the prompt tokens at the input price, and completion tokens at the output price. When the usage has no prompt token details, or no cached count, nothing was cached.
- Task 2
measure.ts: one call in rupees and milliseconds
Write and export measure(client: OpenAI, model: string, messages: OpenAI.ChatCompletionMessageParam[]) that sends the messages to that model and resolves to an object with these keys:
reply (the reply's text), promptTokens, cachedTokens, completionTokens, rupees (what the call cost) and ms (how many milliseconds the call took).
Take the model's prices from client.models.list(): each lab model has a price_inr_per_million field. Use costInr from your cost.ts for the rupees.
The check measures lab-chat-mini and lab-chat-slow (4 seconds).
- Task 3
trim.ts: only what the feature needs
SafarLine's lost-and-found bot only needs sections 6 (Luggage) and 7 (Lost and found) of the policy. Write and export sections(policy: string, numbers: number[]): string that returns only the policy's sections with those numbers, each from its ## heading to the next section, in the policy's order. It may keep the policy's title and opening lines.
Then, when the file is run, ask lab-chat the question Is my laptop safe in the hold? twice, each time with a system message of an instruction to answer only from the policy, followed by the text: first the whole policy, then only sections 6 and 7. Print three lines:
full policy: N prompt tokens
sections 6 and 7: N prompt tokens
and the second reply's text.
- Task 4 · interview question
Interview: compare.ts, should we switch to the cheaper model?
Harpreet asks whether the policy bot can move from lab-chat to lab-chat-mini. Answer with numbers. Ask both models these five questions, each with a system message of an instruction to answer only from the policy followed by the whole of /workspace/data/policy.md, at temperature 0:
Can I bring my dog?
When will my refund arrive?
How much does extra luggage cost?
When is WhatsApp support open?
Can I change the passenger's name?
Print three lines, with each model's average cost per answer times 1,000, in rupees with two decimals, and how many of the five replies are exactly the same:
lab-chat: ₹X per 1,000 answers
lab-chat-mini: ₹Y per 1,000 answers
same replies: N of 5
What you will learn
- Measuring a call: tokens, cached tokens, rupees and milliseconds.
- Where the tokens go: in a grounded answer, almost all of them are the pasted document.
- Three ways to pay less: trim the prompt, let the cache work, and a smaller model, checked on your own examples.
- What latency is made of: the first token, and every token after it.
More in Talking to a model
Build "Cost and latency" yourself
SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant, in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.