Fine-tuning basics (TypeScript) · Chapter 7 of 20
Tokens, limits and cost
By the end of this chapter you can count an example the way the model reads it, keep only the examples a model can train on, work out what a fine-tuning job will read and what it will cost before you start it, and give a rare code more weight in training. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The data
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.
Why you need this
The data is clean, deduplicated, split by customer and in the chat format. Before Irfan presses start on a job, two people want answers.
Rashmi asks what it will cost. Providers do not bill by examples or by megabytes: they bill by tokens, and so far you have counted neither. Mangala has a different worry. Of the 1,000 training examples, 94 are leaks. A model learns most from what it sees most, and leaks are one example in ten.
There is a third problem nobody has asked about yet: some of Chulha's messages are too long for chota to read at all. A job given one of them does not skip it quietly. It fails.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
tokens.ts: count an example as chota reads it
Write and export tokensOf(e): how many of chota's tokens the example e is. Count the whole conversation the way chota-base reads it: chatIds(e.messages).ids (from ../../shared/chota.ts), which leaves the system message out and adds <user>, <bot> and <end>.
Under if (import.meta.main), count every example of examples('train') and print one line:
train: N examples, T tokens, longest L, average A (T added up, L the largest, A = T / N with one decimal)
- Task 2
fits.ts: drop what chota cannot read
Write and export fits(e): true when the example is at most 32 tokens (tokensOf, from ./tokens.ts), chota-base's limit.
Under if (import.meta.main), turn every row of dedupedRows() (./pipeline.ts, given) into an example with toChat(r.text, r.tag) and print:
kept K of N, dropped D
then one line for each dropped example, in the rows' order:
TOKENS tokens: TEXT
- Task 3
cost.ts: what a job trains, and what it costs
Write and export trainingCost(set, epochs, rupeesPerMillion): { tokens, rupees }. A job reads every example once per epoch, so tokens = every example's tokensOf added up, times epochs; rupees = tokens × rupeesPerMillion / 1,000,000.
Under if (import.meta.main), read chota-base's training price from the API (client.models.retrieve('chota-base'): price_inr_per_million.training) and print one line for each of 1, 2, 3 and 4 epochs on examples('train'):
1 epoch: T tokens, ₹R
2 epochs: T tokens, ₹R (rupees with two decimals)
- Task 4 · interview question
Interview: oversample.ts, more leaks for the model to see
Mangala: "Only one example in ten is a leak. Can the model see more of them, and what does that cost?" Write and export oversample(set, tag, times): a new list, in order, where every example tagged tag is repeated times times and every other example appears once.
Under if (import.meta.main), print two lines, for examples('train') and for oversample(train, 'LEAK', 2):
before: N examples, LEAK L (S%), 3 epochs T tokens, ₹R
after: N examples, LEAK L (S%), 3 epochs T tokens, ₹R
(S is LEAK's share of N with one decimal; T and ₹R from trainingCost at ₹250 per million, two decimals)
What you will learn
- Tokens: the pieces a model reads, and why the same message is a different number of tokens for different models.
- The chat template: what the model actually reads, special pieces and all.
- The window: the most tokens a model reads at once, and what to do with examples that do not fit.
- Trained tokens: what a job reads (tokens times epochs), and what that costs.
- Class balance: why a rare code is learned worst, and oversampling, the cheapest way to give it weight.
More in The data
Build "Tokens, limits and cost" yourself
Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.