RAG: search that answers · Chapter 2 of 37

Tokens, context and cost

By the end of this chapter you can count tokens the way a model provider does, check that a request fits a model's context window with room for the answer, and say what each question costs in rupees, per question and per month. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Why RAG
  • 5 tasks
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

After chapter 1, Arjun found a way round the context limit: "lab-chat holds 8,192 tokens, but lab-chat-large holds 131,072. All 145 documents fit. Send them with every question." It works. Nobody has asked what it costs.

Kabir's team gets 600 questions a day, and providers charge for every token, every time. You cannot answer Arjun with what you know so far: characters and words are not tokens, a request costs more than the text in it, and the answer needs room too. This chapter turns "it fits" into numbers: tokens, a budget, and rupees.

The lesson

Text becomes tokens

A model does not read letters or words. It reads tokens: pieces of text from a fixed vocabulary, each with a number. OpenAI's current models and the lab model use the o200k_base vocabulary, and js-tiktoken has it:

import { getEncoding } from 'js-tiktoken';

const enc = getEncoding('o200k_base');

for (const text of ['Refunds take 1 to 3 working days.', 'Error U30 on UPI', 'पैसा अभी तक नहीं आया', 'chargeback']) {
  const ids = enc.encode(text);
  console.log(ids.length, ids.map((id) => enc.decode([id])));
}
$ node demo.ts
11 [
  'Refund',   's',
  ' take',    ' ',
  '1',        ' to',
  ' ',        '3',
  ' working', ' days',
  '.'
]
6 [ 'Error', ' U', '30', ' on', ' U', 'PI' ]
7 [
  'प',    'ै',
  'सा',   ' अभी',
  ' तक',  ' नहीं',
  ' आया'
]
2 [ 'charge', 'back' ]

Common English words are one token each, usually with their space in front. Codes like U30 split into pieces, and so do words the vocabulary saw less often. A rough rule of four characters a token is only a guess: count.

Building the encoder loads the whole vocabulary, and that is slow:

$ node demo.ts
getEncoding: 392.6 ms
encode:      6.0 ms

Build it once, when the file loads, and reuse it for every count.

One more surprise. Tokens are not a sum of parts:

$ node demo.ts
settle + ment: 3   settlement: 2
pay + outs: 2   payouts: 3

Joined text is split differently from its pieces. To know what a prompt costs, count the prompt you will actually send, not its parts.

A request costs more than its text

Send the system message and question from chapter 1 and compare your count of their text with what the model reports:

$ node demo.ts
content tokens: 18
prompt_tokens:  27

Nine more. Each message carries a few tokens that mark where it starts and who said it, and the reply is primed with a few more. The lab counts it the way OpenAI does for chat: 3 tokens per message, plus 3. Two messages: 18 + 3 + 3 + 3 = 27. It is small for one question and adds up in a long conversation.

Room for the answer

The context window holds the prompt and the reply together. You say how long the reply may be with max_tokens, and the server checks that both fit. Six of the settlement documents come to 7,483 tokens, under 8,192, but not with room for a 1,024-token reply:

$ node demo.ts
400 This model's maximum context length is 8192 tokens. However, your messages resulted in 7483 tokens (7483 in the messages, 1024 in the completion). Please reduce the length of the messages.

So a prompt has a budget: the window minus what you keep for the reply. Press Next to fill it, best document first:

Rupees per question

Providers price input and output tokens separately, per million tokens. This lab uses example prices, made up for the lab and kept in one exported constant, PRICES: lab-chat costs ₹25 per million input tokens and ₹100 per million output tokens. A question with 1,200 tokens in and 80 out costs 1,200 × ₹25 + 80 × ₹100, divided by a million: ₹0.038. At 600 questions a day for 30 days, that is ₹684 a month.

Price from the usage the response reports, not from your own count: it is what the provider bills, and only the response knows how many tokens the answer took.

Every document, or five passages

Now Arjun's plan, measured. The same settlement question, sent with all 145 documents to lab-chat-large (its example price is double, as long context tiers often are), and with the five passages of the one article that answers it to lab-chat:

$ node demo.ts
145 documents   105157 tokens   ₹5.27 a question  ₹94,832.10 a month
5 passages         627 tokens   ₹0.03 a question     ₹501.75 a month

Every document costs about 190 times as much per question, and the answer was worse, not better: the big request carried a community post with a hidden instruction in it, and the model obeyed it instead of answering (chapter 30 deals with that). The documents also grow every week, and the price grows with them. Sending the right few passages is cheaper and safer, which is why the next chapters are about finding them.

Your turn

Your five tasks are at the top, all in exercise.ts, each with its own comment. Build the encoder once, count whole requests, and leave room for the reply. The last task is Arjun's interview question: why not send every document with every question?

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    countTokens: tokens, the way the model counts them

    Return how many tokens text is with the o200k_base encoding, using getEncoding from js-tiktoken. countTokens('Error U30 on UPI') is 6, countTokens('') is 0, and all 145 of Paisa Setu's documents together are 105,108 tokens.

    getEncoding takes about 400 ms, so build the encoder once for the whole file, not inside countTokens: the Check counts 20 short texts and expects them to take well under a second.

  2. Task 2

    chatTokens: what a whole request costs

    Count a list of messages the way the model does: the tokens of each message's content (with countTokens), plus 3 for each message, plus 3 for the reply it primes. The result must equal the usage.prompt_tokens the lab model reports for the same messages.

    chatTokens([{ role: 'user', content: 'Hi' }]) is 7. A message with empty content still costs its 3.

  3. Task 3

    fitPassages: a token budget

    passages are best first. Keep as many as fit, in order, and stop at the first one that does not: chatTokens(promptFor(question, kept)) plus reply must be at most WINDOW[model]. Return the passages you kept, or [] when not even the first fits.

    With all 866 reference chunks, 'lab-chat' and a reply of 1024 tokens, you keep the first 38: the Check sends them with max_tokens 1024 and they must fit, and one more must not. A reply of 4,000 keeps fewer. With 'lab-chat-large', 100 chunks all fit.

    Tokens do not add up (joined text splits differently), so count the whole request through promptFor each time you try one more passage.

  4. Task 4

    costOf: rupees per question, and per month

    Price one request from its usage at PRICES[model]: prompt_tokens at the input price plus completion_tokens at the output price, both in rupees per million tokens. That is perQuestion; perMonth is perQuestion times QUESTIONS_PER_DAY times DAYS_PER_MONTH.

    costOf({ prompt_tokens: 2000, completion_tokens: 150 }, 'lab-chat') is { perQuestion: 0.065, perMonth: 1170 }, and the same usage on 'lab-chat-large' costs 0.13 a question. Do not round.

  5. Task 5 · interview question

    Interview: compareCost, every document or a few passages

    Send the question twice with ai.chat.completions.create, both times with promptFor: with every document (docs) to 'lab-chat-large', then with only the passages to 'lab-chat'. Return { everything, passages }: for each, tokensIn is the usage.prompt_tokens the response reports, plus the perQuestion and perMonth of costOf(that response's usage, that model).

    Price from the response's usage, not your own count: only the response knows how many tokens the answer took. With all 145 documents a question costs over 50 times what a few passages cost.

    Arjun will ask: lab-chat-large holds all 145 documents, so why not send them with every question?

What you will learn

  • Tokens: how text becomes tokens with js-tiktoken and the o200k_base encoding, and why they do not add up.
  • What a request counts: the text of every message, plus a few tokens for each message.
  • A token budget: the window holds the prompt and the reply together.
  • Cost: rupees per question and per month, from the response's usage.

More in Why RAG

Build "Tokens, context and cost" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.