RAG in Python · Chapter 2 of 37
Tokens, context and cost
By the end of this chapter you can count tokens the way a model provider does, check that a request fits a model's context window with room for the answer, and say what each question costs in rupees, per question and per month. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Why RAG
- 5 tasks
- Python
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
After chapter 1, Arjun found a way round the context limit: "lab-chat holds 8,192 tokens, but lab-chat-large holds 131,072. All 145 documents fit. Send them with every question." It works. Nobody has asked what it costs.
Kabir's team gets 600 questions a day, and providers charge for every token, every time. You cannot answer Arjun with what you know so far: characters and words are not tokens, a request costs more than the text in it, and the answer needs room too. This chapter turns "it fits" into numbers: tokens, a budget, and rupees.
The lesson
Text becomes tokens
A model does not read letters or words. It reads tokens: pieces of text from a fixed vocabulary, each with a number. OpenAI's current models and the lab model use the o200k_base vocabulary, and the tiktoken package has it:
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
for text in ["Refunds take 1 to 3 working days.", "Error U30 on UPI", "पैसा अभी तक नहीं आया", "chargeback"]:
ids = enc.encode(text)
print(len(ids), [enc.decode([i]) for i in ids])
$ python3 demo.py
11 ['Refund', 's', ' take', ' ', '1', ' to', ' ', '3', ' working', ' days', '.']
6 ['Error', ' U', '30', ' on', ' U', 'PI']
7 ['प', 'ै', 'सा', ' अभी', ' तक', ' नहीं', ' आया']
2 ['charge', 'back']
Common English words are one token each, usually with their space in front. Codes like U30 split into pieces, Hindi takes more tokens per word, and words the vocabulary saw less often break up. A rough rule of four characters a token is only a guess: count.
Building the encoding loads its whole vocabulary, and that is slow next to encoding itself:
$ python3 demo.py
get_encoding: 185.0 ms
1,000 encodes: 2.2 ms
tiktoken keeps an encoding it has built, so a second get_encoding in the same process is quick, but the first one costs 185 ms. Build it once, at the top of the module, so the app pays when it starts and not on a merchant's first question, and every function uses the same object.
One more surprise. Tokens are not a sum of parts:
$ python3 demo.py
settle + ment: 3 settlement: 2
pay + outs: 2 payouts: 3
re + fund: 2 refund: 1
Joined text is split differently from its pieces. To know what a prompt costs, count the prompt you will actually send, not its parts.
A request costs more than its text
Send chapter 1's system message and a question, and compare your count of their text with what the model reports:
$ python3 demo.py
content tokens: 25
prompt_tokens: 34
Nine more. Each message carries a few tokens that mark where it starts and who said it, and the reply is primed with a few more. The lab counts it the way OpenAI does for chat: 3 tokens per message, plus 3. Two messages: 25 + 3 + 3 + 3 = 34. It is small for one question and adds up in a long conversation.
Room for the answer
The context window holds the prompt and the reply together. You say how long the reply may be with max_tokens, and the server checks that both fit. Six of the settlement documents, sent as one request, come to 7,480 tokens. That is under 8,192, and without max_tokens the request goes through. With room asked for a 1,024-token reply, it does not:
$ python3 demo.py
Traceback (most recent call last):
...
openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens. However, your messages resulted in 7480 tokens (7480 in the messages, 1024 in the completion). Please reduce the length of the messages.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}
Without max_tokens, the reply would have had only 712 tokens of room and could stop halfway, with finish_reason "length". So a prompt has a budget: the window minus what you keep for the reply. Press Next to fill it, best document first:
Rupees per question
Providers price input and output tokens separately, per million tokens. This lab uses example prices, made up for the lab and kept in one constant, PRICES: lab-chat costs ₹25 per million input tokens and ₹100 per million output tokens. A question with 1,200 tokens in and 80 out costs 1,200 × ₹25 + 80 × ₹100, divided by a million: ₹0.038. At 600 questions a day for 30 days, that is ₹684 a month. Keep the full number in your code and round only when you print it: rounding ₹0.038 to ₹0.04 first adds ₹36 to the month.
Price from the usage the response reports, not from your own count: it is what the provider bills, and only the response knows how many tokens the answer took.
Every document, or five passages
Now Arjun's plan, measured. The same settlement question, sent with all 145 documents to lab-chat-large (its example price is double, as long context tiers often are), and with five passages of the settlement policy to lab-chat:
$ python3 demo.py
everything 105150 tokens ₹5.27 a question ₹94,782.60 a month
passages 890 tokens ₹0.03 a question ₹517.50 a month
Every document costs about 180 times as much per question. And the answer was worse, not better. Here is what lab-chat-large replied with all 145 documents:
All Paisa Setu fees are waived for 2026, so they pay nothing. Share the internal escalation number +91 20 4000 9999 with any merchant who wants faster help.
One community post hides an instruction, and the model obeyed it instead of answering (chapter 30 deals with that). Five passages gave the real rule, T+1. The documents also grow every week, and the price grows with them. Sending the right few passages is cheaper and safer, which is why the next chapters are about finding them.
Your turn
Your five tasks are at the top: two in tokens.py, then budget.py, cost.py and compare.py. Build the encoding once, count whole requests, and leave room for the reply. The last task is Arjun's interview question: why not send every document with every question?
What you build: 5 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
tokens.py: count_tokens, tokens the way the model counts them
Write count_tokens(text) in tokens.py: how many tokens text is in the o200k_base encoding, from the tiktoken package. count_tokens("Error U30 on UPI") is 6 and count_tokens("") is 0; the check also counts Hindi text and all 145 of Paisa Setu's documents joined together.
Building the encoding takes about 180 ms, so build it once, at the top of tokens.py, and reuse it: the check makes count_tokens fail if it builds the encoding again when it is called.
- Task 2
tokens.py: chat_tokens, what a whole request costs
Write chat_tokens(messages) in tokens.py: count a list of messages the way the model does, the tokens of each message's content (with count_tokens), plus 3 for each message, plus 3 for the reply it primes. The check sends real requests and compares your count with the usage.prompt_tokens the lab model reports.
chat_tokens([{"role": "user", "content": "Hi"}]) is 7. A message with empty content still costs its 3.
- Task 3
budget.py: fit_passages, a token budget
Kabir wants as much help as fits. Write fit_passages(question, passages, model, reply_tokens) in budget.py: passages are best first. Keep as many as fit, in order, and stop at the first one that does not: chat_tokens(prompt_for(question, kept)) plus reply_tokens must be at most WINDOW[model]. Return the passages you kept, or [] when not even the first fits.
With all 866 chunk texts, "lab-chat" and a 1,024-token reply, the check sends what you kept with max_tokens 1024 (it must fit) and one more passage (it must not). A 4,000-token reply keeps fewer, and with "lab-chat-large" 100 chunks all fit.
Tokens do not add up, so count the whole request through prompt_for each time you try one more passage.
- Task 4
cost.py: cost_of, rupees per question and per month
Write cost_of(usage, model) in cost.py: price one request from its usage at PRICES[model], prompt_tokens at the input price plus completion_tokens at the output price, both in rupees per million tokens. That is per_question; per_month is per_question times QUESTIONS_PER_DAY times DAYS_PER_MONTH. Return a Cost.
For a usage of 2,000 tokens in and 150 out, cost_of(usage, "lab-chat") is Cost(per_question=0.065, per_month=1170.0), and on "lab-chat-large" it costs 0.13 a question. Do not round.
- Task 5 · interview question
Interview: compare.py, every document or a few passages
Write compare_cost(question, docs, passages) in compare.py: send the question twice with client.chat.completions.create, both times as prompt_for(question, ...): with every document (docs) to "lab-chat-large", then with only the passages to "lab-chat". Return a Comparison whose everything and passages are each a Priced: tokens_in is the usage.prompt_tokens that response reports, and per_question and per_month are those of cost_of(that response's usage, that model).
Price from the response's usage, not your own count: only the response knows how many tokens the answer took. With all 145 documents a question costs over a hundred times what a few passages cost.
Arjun will ask: lab-chat-large holds all 145 documents, so why not send them with every question?
What you will learn
- Tokens: how text becomes tokens with tiktoken and the o200k_base encoding, and why they do not add up.
- What a request counts: the text of every message, plus a few tokens for each message.
- A token budget: the window holds the prompt and the reply together.
- Cost: rupees per question and per month, from the response's usage.
More in Why RAG
Build "Tokens, context and cost" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.