Building with LLMs · Chapter 8 of 24

Cost and latency

By the end of this chapter you can measure what one call and one feature cost in rupees and in seconds, read cached tokens on the bill, trim a prompt to what a feature needs, and decide with numbers whether a cheaper model will do. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Talking to a model
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.

Why you need this

The drafting tool has been running for a week, and Harpreet from finance has the first bill. He does not want the total: he wants to know which feature costs what, what one more customer message costs, and what next month will cost when Zoya turns the tool on for all 240 messages a day. Lata has a question of her own: customers say the bot is slow. How slow?

A function you write in Python costs nothing to run again, and takes the time your computer takes. A model call is billed by the token, by someone else, and takes the provider's time. Chapter 3 priced one request. This chapter measures a whole feature, finds where its tokens go, and cuts them.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    cost.py: what a call costs

    Write cost_inr(usage, price) that returns what one call cost, in rupees, as a number (not rounded).

    usage is a response's usage. price is a dictionary of rupees per million tokens, as the lab's models list reports it, for example {"input": 200, "cached_input": 50, "output": 800}.

    Cached prompt tokens are billed at the cached_input price, the rest of the prompt tokens at the input price, and completion tokens at the output price. When the usage has no prompt token details, or no cached count, nothing was cached.

  2. Task 2

    measure.py: one call in rupees and seconds

    Write measure(client, model, messages) that sends the messages to that model and returns a dictionary with these keys:

    reply (the reply's text), prompt_tokens, cached_tokens, completion_tokens, rupees (what the call cost) and seconds (how long the call took).

    Take the model's prices from client.models.list(): each lab model has a price_inr_per_million field. Use cost_inr from your cost.py for the rupees.

    The check measures lab-chat-mini and lab-chat-slow (4 seconds).

  3. Task 3

    trim.py: only what the feature needs

    SafarLine's lost-and-found bot only needs sections 6 (Luggage) and 7 (Lost and found) of the policy. Write sections(policy, numbers) that returns only the policy's sections with those numbers, each from its ## heading to the next section, in the policy's order. It may keep the policy's title and opening lines.

    Then, when the file is run, ask lab-chat the question Is my laptop safe in the hold? twice, each time with a system message of an instruction to answer only from the policy, followed by the text: first the whole policy, then only sections 6 and 7. Print three lines:

    full policy: N prompt tokens

    sections 6 and 7: N prompt tokens

    and the second reply's text.

  4. Task 4 · interview question

    Interview: compare.py, should we switch to the cheaper model?

    Harpreet asks whether the policy bot can move from lab-chat to lab-chat-mini. Answer with numbers. Ask both models these five questions, each with a system message of an instruction to answer only from the policy followed by the whole of /workspace/data/policy.md, at temperature 0:

    Can I bring my dog?

    When will my refund arrive?

    How much does extra luggage cost?

    When is WhatsApp support open?

    Can I change the passenger's name?

    Print three lines, with each model's average cost per answer times 1,000, in rupees with two decimals, and how many of the five replies are exactly the same:

    lab-chat: ₹X per 1,000 answers

    lab-chat-mini: ₹Y per 1,000 answers

    same replies: N of 5

What you will learn

  • Measuring a call: tokens, cached tokens, rupees and seconds.
  • Where the tokens go: in a grounded answer, almost all of them are the pasted document.
  • Three ways to pay less: trim the prompt, let the cache work, and a smaller model, checked on your own examples.
  • What latency is made of: the first token, and every token after it.

More in Talking to a model

Build "Cost and latency" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.