Fine-tuning basics · Chapter 7 of 20

Tokens, limits and cost

By the end of this chapter you can count an example the way the model reads it, keep only the examples a model can train on, work out what a fine-tuning job will read and what it will cost before you start it, and give a rare code more weight in training. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The data
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.

Why you need this

The data is clean, deduplicated, split by customer and in the chat format. Before Irfan presses start on a job, two people want answers.

Rashmi asks what it will cost. Providers do not bill by examples or by megabytes: they bill by tokens, and so far you have counted neither. Mangala has a different worry. Of the 1,000 training examples, 94 are leaks. A model learns most from what it sees most, and leaks are one example in ten.

There is a third problem nobody has asked about yet: some of Chulha's messages are too long for chota to read at all. A job given one of them does not skip it quietly. It fails.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    tokens.py: count an example as chota reads it

    Write tokens_of(e): how many of chota's tokens the example e is. Count the whole conversation the way chota-base reads it: chat_ids(e["messages"]).ids (from chota), which leaves the system message out and adds <user>, <bot> and <end>.

    Under if __name__ == "__main__":, count every example of examples("train") and print one line:

    train: N examples, T tokens, longest L, average A (T added up, L the largest, A = T / N with one decimal)

  2. Task 2

    fits.py: drop what chota cannot read

    Write fits(e): True when the example is at most 32 tokens (tokens_of, from tokens.py), chota-base's limit.

    Under if __name__ == "__main__":, turn every row of deduped_rows() (pipeline.py, given) into an example with to_chat(r["text"], r["tag"]) and print:

    kept K of N, dropped D

    then one line for each dropped example, in the rows' order:

    TOKENS tokens: TEXT

  3. Task 3

    cost.py: what a job trains, and what it costs

    Write training_cost(set_, epochs, rupees_per_million): a dict {"tokens", "rupees"}. A job reads every example once per epoch, so tokens = every example's tokens_of added up, times epochs; rupees = tokens × rupees_per_million / 1,000,000.

    Under if __name__ == "__main__":, read chota-base's training price from the API (client.models.retrieve("chota-base").price_inr_per_million["training"]) and print one line for each of 1, 2, 3 and 4 epochs on examples("train"):

    1 epoch: T tokens, ₹R

    2 epochs: T tokens, ₹R (rupees with two decimals)

  4. Task 4 · interview question

    Interview: oversample.py, more leaks for the model to see

    Mangala: "Only one example in ten is a leak. Can the model see more of them, and what does that cost?" Write oversample(set_, tag, times): a new list, in order, where every example tagged tag is repeated times times and every other example appears once.

    Under if __name__ == "__main__":, print two lines, for examples("train") and for oversample(train, "LEAK", 2):

    before: N examples, LEAK L (S%), 3 epochs T tokens, ₹R

    after: N examples, LEAK L (S%), 3 epochs T tokens, ₹R

    (S is LEAK's share of N with one decimal; T and ₹R from training_cost at ₹250 per million, two decimals)

What you will learn

  • Tokens: the pieces a model reads, and why the same message is a different number of tokens for different models.
  • The chat template: what the model actually reads, special pieces and all.
  • The window: the most tokens a model reads at once, and what to do with examples that do not fit.
  • Trained tokens: what a job reads (tokens times epochs), and what that costs.
  • Class balance: why a rare code is learned worst, and oversampling, the cheapest way to give it weight.

More in The data

Build "Tokens, limits and cost" yourself

Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.