Build an AI gateway · Chapter 4 of 21

Tokens and rupees

By the end of this chapter you can price any call to the paisa, cache discounts included, guess a request's tokens before sending it and say how far off the guess is for each provider, add up what each of Jawab's brands spent and match it to the providers' bill, and say the most a request can cost before it is sent. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: One API over three providers
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.

Why you need this

Lakshmi runs Jawab's finance. Every month three invoices arrive, one from each provider, and none of them says which brand the money was for. Kaapi Kottai pays Jawab for its support; Nila Handlooms pays less; each brand has an AI budget for the day (lab teams: ₹6 for Kaapi, ₹2.50 for Nila, ₹27.50 across all six). Lakshmi wants three things:

  • what each brand spent today, in rupees;
  • that Darwaza's figures agree with the invoices, to the paisa, so she can trust them;
  • and later (chapter 15), that no brand goes over its budget, which means knowing what a request might cost before it is sent.

Darwaza already reads each provider's usage into one shape (chapter 2). But chapter 1's v0 priced every input token at the full price, and that is wrong as soon as a provider's cache is involved: a cached token on Nimbus costs a quarter, and Kestrel has two cache prices of its own. This chapter turns usage into rupees properly, and then turns rupees into something Lakshmi can act on.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    cost.py: what one call cost

    Write def cost_inr(target, usage) -> float, the rupees for one call at PRICES[target] (rupees per million tokens): usage["input"] at the full input price (.input), usage["cached"] at the cache-read price (Nimbus's .cached_in or Kestrel's .cache_read), usage["cache_write"] at Kestrel's .cache_write price, and usage["output"] at .output, added up and divided by 1,000,000.

    Trap: "cached" and "cache_write" may be missing (count them as 0), and Tara has no cache prices at all (they are None). Pricing cached tokens at the full price makes a cached call look up to four times dearer than the bill.

  2. Task 2

    estimate.py: tokens, before sending

    Write def estimate_tokens(messages) -> int: for each message its characters / 4 rounded up, plus 4; then 3 more for the request.

    Under if __name__ == "__main__":, send each of the day's first 20 requests (day()[:20]) with call from darwaza.py to "nimbus/nimbus-mini", "kestrel/kestrel-swift" and "tara/tara-flash". A request's error is (estimate - counted) / counted, where counted is every input token the provider reported (usage's input + cached + cache_write). Print, per provider, the average error in %, one decimal, with its sign:

    nimbus: estimate off by +X%

    kestrel: estimate off by -Y%

    tara: estimate off by +Z%

    Trap: counted must include the cached tokens: a cached request still had all of them read.

  3. Task 3

    spend.py: rupees per brand

    Write an async gateway function that awaits handle (darwaza.py) and adds "cost_inr" (cost_inr(r["model"], r["usage"]), from cost.py) to each answer, passing failures on as they are. Replay "10:00" to "12:00" through it. Add up the answered results' cost_inr and count the answers per team, and print one line per team, most rupees first:

    team: ₹X (N answers) (4 decimals)

    then:

    Darwaza's sum: ₹X (all teams, 2 decimals)

    providers billed: ₹Y (report.inr, 2 decimals)

    Trap: the two totals must agree; they do only if every answer is priced from its own usage at the providers' prices.

  4. Task 4 · interview question

    Interview: worst_case.py, the most it can cost

    Write def worst_case_inr(target, req) -> float: estimate_tokens(req["messages"]) at the full input price plus req's max_tokens (1024 when there is none) at the output price, over 1,000,000, at PRICES[target]. It sends nothing.

    Under if __name__ == "__main__":, for the day's first reply (day(features=["reply"])[0].request), print, 4 decimals, for nimbus/nimbus-large, nimbus/nimbus-mini, kestrel/kestrel-pro, kestrel/kestrel-swift and tara/tara-flash in that order:

    TARGET: at most ₹X

    then:

    cheapest worst case: TARGET

    Trap: the worst case assumes no cache discount and every allowed output token; use the request's own max_tokens when it has one.

What you will learn

  • How providers bill: per million tokens, input and output apart, and cached tokens at their own prices.
  • cost_inr(target, usage): one call, priced to the paisa.
  • Counting tokens before sending: a cheap estimate, and how wrong it is for each provider.
  • Rupees per brand: Darwaza's sums, checked against the bill.
  • The worst case: the most a request can cost, before anything is sent.

More in One API over three providers

Build "Tokens and rupees" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.