Building with LLMs (TypeScript) · Chapter 3 of 24
Tokens and context
By the end of this chapter you can count the tokens of any text and of a whole request before you send it, explain what a context window is and what happens when a request does not fit, and turn token counts into rupees. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Talking to a model
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Harpreet from finance has one question about the support bot: what will it cost? Providers do not charge per message or per word. They charge per token, for what you send and for what the model writes, and every chat turn resends the whole conversation (chapter 2). Lata has a question too: she pasted SafarLine's whole bookings file into a request so the bot could look things up, and the API refused it.
Both answers are the same unit. A string's length counts characters, which is not what the model reads, not what it can hold, and not what you pay for. This chapter counts what is.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
languages.ts: characters and tokens
Harpreet suspects Hinglish and Hindi messages cost more. Read /workspace/data/messages.jsonl (one JSON object per line) and, for each language in this order: en, hinglish, hi, print the average number of characters (the text's length) and the average number of tokens (o200k_base) of the messages' text, both with one decimal, like this:
en: 12.3 characters, 4.5 tokens
Print exactly three lines.
- Task 2
count-request.ts: predict prompt_tokens
Export a function from count-request.ts:
countRequest(messages: { role: string; content: string }[]): number
It returns the number of prompt tokens the API will count for a request to lab-chat with those messages, without sending anything.
The API adds a fixed number of tokens for every message and a fixed number for the whole request. Find both numbers by experiment: send requests with one and with two messages and compare usage.prompt_tokens with the tokens of their contents.
The check compares your function with the real prompt_tokens for several requests.
- Task 3
too-long.ts: past the context window
Lata wants the bot to answer from the whole bookings file. Send lab-chat a system message holding an instruction to answer from the bookings below followed by the whole text of /workspace/data/bookings.json, and the user message: How many bookings are for Goa?
The request is too big: catch the API's error and print lab-chat, a colon, a space and the error's code.
Then send the same messages to lab-chat-large and print lab-chat-large: N prompt tokens (N from the response's usage), then input cost: ₹X, the input cost at lab-chat-large's ₹1000 per million input tokens, with two decimals:
lab-chat: ...
lab-chat-large: ... prompt tokens
input cost: ₹...
- Task 4 · interview question
Interview: estimate.ts, what the queue would cost
Harpreet asks: if the bot answered every message in /workspace/data/messages.jsonl, each in its own request with a system message holding exactly the text of /workspace/data/policy.md (nothing else) and a user message holding exactly the customer's text, how many prompt tokens would that be in total, and what would they cost on lab-chat at ₹200 per million input tokens?
Answer without calling the model at all: count. Print exactly:
prompt tokens: N
input cost: ₹X
with N the total and X in rupees with two decimals.
What you will learn
- What a token is, and how to count them with js-tiktoken.
- Why the same message costs more in Hinglish or Hindi than in English.
- Counting a whole request, the way the API does.
- The context window: the most a model can read and write at once, and the error when you go over it.
- Price per token in ₹, for input and output.
More in Talking to a model
Build "Tokens and context" yourself
SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant, in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.