Fine-tuning basics · Chapter 1 of 20

Meet Chulha, and two ways to tag a message

By the end of this chapter you can say what fine-tuning is and what it is not, call a model through an OpenAI-style API from Python, measure a model on a held-out set with three numbers instead of one, and compare a prompted big model with a small fine-tuned one on quality and on cost. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Should you fine-tune?
  • 4 tasks
  • Python
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

Chulha, in Nagpur, runs WhatsApp booking and support for forty LPG gas agencies across Vidarbha. Customers write the way they talk: "cylinder kab aayega", "gas cha vaas yetoy", "delivery boy ne 50 rupaye extra maange". About 30,000 messages a month. Each one needs one of six codes so it lands in the right queue: REFILL, DELIVERY, LEAK, CONNECTION, PAYMENT or OTHER.

Mangala, who leads the desk, wrote the tagging guide and has tagged more messages than anyone. One code matters more than the rest: a LEAK that lands in the billing queue is a kitchen full of gas that nobody calls back about. Rashmi, Chulha's founder, wants the tagging done by a model so her agents can spend their time answering. Her first plan is the obvious one: rent a big general model and tell it the six codes.

Irfan, Chulha's ML engineer, will work with you. Before anyone argues about models, he wants numbers. Here is the first one: that big model, asked the obvious way, tags 60.5% of the desk's validation messages right and catches 13 of 24 leaks. Last week Irfan also fine-tuned a tiny model on Mangala's tagged messages, as an experiment. It tags 90.8% right, every reply is a code, it costs a fraction of the big one per message, and it still misses 8 of 24 leaks.

So fine-tuning helped a lot, and it is not done. This lab is about the space between those two facts: what fine-tuning really is, when it beats prompting and when it does not, how to prepare the data, how to run a job, what happens inside one, and how to prove the result is good enough for Mangala.

The lesson

Chulha's data

Mangala's desk exported two months of tagged messages, and Irfan cleaned them. They are in /workspace/chulha (read-only), split three ways: the split names are the ones every fine-tuning project uses.

  • train.jsonl: 1,000 messages to learn from.
  • val.jsonl: 195 messages to check on while you work (validation).
  • test.jsonl: 186 messages to score on at the end, and never to learn from.

lab data in the terminal counts them by tag:

train 1000  REFILL 284  DELIVERY 263  LEAK 94  CONNECTION 123  PAYMENT 190  OTHER 46
val    195  REFILL 56  DELIVERY 49  LEAK 24  CONNECTION 22  PAYMENT 37  OTHER 7
test   186  REFILL 55  DELIVERY 57  LEAK 18  CONNECTION 28  PAYMENT 19  OTHER 9

Each line is one conversation in the chat format fine-tuning APIs expect: a system message, the customer's message, and the reply the model should give. Chapter 6 is about this format; for now, read one:

{"messages":[{"role":"system","content":"Tag this Chulha customer message with one code: REFILL, DELIVERY, LEAK, CONNECTION, PAYMENT or OTHER. Reply with the code only."},{"role":"user","content":"regulator se sss awaaz aa rahi hai"},{"role":"assistant","content":"LEAK"}]}

chulha.py (given, in /workspace/shared) reads them: examples("test") is the list, message_of(e) the customer's words, tag_of(e) the right code, and PROMPT the system message:

from chulha import PROMPT, examples, message_of, tag_of

What fine-tuning is

A language model is trained once, on a lot of text, to predict what comes next. That training sets its weights: the numbers inside it. After that you can change what it does in two very different ways.

  • Prompting changes the input. You write instructions and examples in the request, and the model reads them every time. Nothing inside it changes.
  • Fine-tuning changes the weights. You show the model many examples of your exact job (a message in, the right code out) and train it a little more, so the job is built in. After that a short request is enough.

Fine-tuning is good at behaviour: a fixed format, a narrow task done the same way every time, a house style, rules that are too many or too subtle to write in a prompt. It is poor at facts that change: a model does not reliably learn this month's prices from a training file, and next month they are wrong anyway. Chapter 2 measures both sides.

The lab's API

The lab has no internet, so it brings its own model provider at $OPENAI_BASE_URL, with the same API as OpenAI's. The official openai package for Python works unchanged. lab models lists what it sells:

lab-chat                                       window 128000  ₹250 in, ₹1000 out (per million tokens)
chota-base                                     window     32  ₹2 in, ₹8 out, ₹250 to train (per million tokens)
ft:chota-base:chulha:tags-v1:c0ffee01          window     32  ₹4 in, ₹16 out (per million tokens)

(trimmed: the ready model's checkpoints are listed too)

  • lab-chat is the big general model Rashmi wants to rent. In this lab it is a predictable stand-in, not a neural network: it reads what you send and answers the same way every time, the way a general model does with words it knows, which is what lets this lab measure prompting exactly.
  • chota-base is a real tiny transformer, 45,152 numbers, trained by Chulha on two years of customers' messages to guess the next word as someone types. It is the model you fine-tune. Chapter 10 opens it up.
  • ft:chota-base:chulha:tags-v1:c0ffee01 is Irfan's experiment: chota-base after a fine-tuning job on train.jsonl. The name says what it is: fine-tuned, from chota-base, by Chulha, the suffix he chose, an id.

OPENAI_API_KEY is set to a dummy key. Point the two variables at a real provider with your own key, on a machine with internet, and the same code calls a real model.

Asking a model to tag

A tagging request is the system prompt and one customer message:

from openai import OpenAI

client = OpenAI()
r = client.chat.completions.create(
    model="lab-chat",
    messages=[
        {"role": "system", "content": PROMPT},
        {"role": "user", "content": "regulator se sss awaaz aa rahi hai"},
    ],
)
print(r.choices[0].message.content, r.usage.prompt_tokens, r.usage.completion_tokens)
CONNECTION 51 2

OpenAI() reads OPENAI_BASE_URL and OPENAI_API_KEY, which the lab has set. The reply's text is in its first choice; usage says how many tokens went in (the prompt) and came out (the reply).

A hissing regulator is a leak. Mangala's guide says so, but the prompt did not, and a general model files "regulator" under equipment. The same message to Irfan's fine-tuned model:

LEAK 9 2

It learned the rule from Mangala's examples. Look at the tokens too: 51 against 9. chota-base's chat template drops the system message (it has no system role, like some open models), and the fine-tuned model does not need it: the job is in its weights. Every token is paid for.

The plain chota-base, before any fine-tuning, answers with nothing at all: it was trained to finish customers' messages, never to reply to one.

Format is a number too

The prompt says "Reply with the code only". Leave that sentence out and lab-chat explains itself:

The customer wants to book a refill, so the code is REFILL.

A person reads that fine. The code that routes messages to queues does not: it compares the reply with six exact strings. So Irfan counts format on its own: the share of replies that are exactly one of the six codes. Even with the sentence in, lab-chat adds a few words to about one reply in ten. A fine-tuned model replies the way its examples did, every time.

Three numbers

"It is 90% accurate" is one number, and Mangala does not trust one number. Irfan scores every model on the same held-out messages, the same way:

  • Accuracy: replies exactly equal to the right code.
  • Format: replies that are one of the six codes at all.
  • LEAK caught: of the messages that are leaks, how many were tagged LEAK.

Five validation messages through lab-chat, one at a time:

On the validation set, with the prompt above:

ModelAccuracyFormatLEAK caught
lab-chat60.5%89.2%13/24
tags-v1 (fine-tuned chota)90.8%100.0%16/24

The test set is for the final word, so your tasks score on it, once each. Why held out: a model scored on messages it trained on looks better than it is (it may remember them). The test messages also come from customers the training set never saw: chapter 5 is about why that matters.

What a message costs

Cost is tokens times price. A request's usage says how many tokens went in and came out, and lab models (or client.models.retrieve(id)) gives the prices in rupees per million tokens. For the 195 validation messages, lab-chat read 10,085 tokens and wrote 549:

(10,085 × ₹250 + 549 × ₹1000) / 1,000,000 = ₹3.07, or ₹15.74 per 1,000 messages

At 30,000 messages a month that is ₹472. Not much, until the prompt grows: chapter 2 adds Mangala's guide and examples to it, and the price grows with every word. The fine-tuned model reads 9 to 15 tokens per message at ₹4 per million.

Your turn

Tag one test message with lab-chat in task 1. Write score_model in task 2 and score lab-chat on the test set. Put the two ways side by side with their cost in task 3. In the interview question, Mangala asks the one thing the accuracy hides: which leaks did each of them miss?

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    first_tag.py: ask lab-chat for a tag

    Take the first message of the test set (examples("test")[0], from chulha) and send it to the model "lab-chat" with PROMPT as the system message and the customer's words (message_of) as the user message. Print four lines, in this order:

    message: MESSAGE

    lab-chat: REPLY (the reply's text)

    expected: TAG (tag_of: the right code)

    tokens: P in, C out (the reply's usage: prompt_tokens and completion_tokens)

  2. Task 2

    score.py: three numbers for a model

    Write score_model(model, set_): send every example of set_ to model, one request each, with PROMPT as the system message and message_of(e) as the user message. Return a dict with accuracy, format, leaks_caught, leaks, prompt_tokens and completion_tokens:

    accuracy: the share of replies exactly equal to tag_of(e) (0 to 1); format: the share of replies that are exactly one of the six TAGS; leaks: how many examples are tagged LEAK; leaks_caught: how many of those got the reply LEAK; prompt_tokens and completion_tokens: the usage of every request added up.

    Under if __name__ == "__main__": score "lab-chat" on examples("test") and print one line:

    lab-chat: accuracy A%, format F%, LEAK C/L (percentages with one decimal, like 64.5%)

  3. Task 3

    two_ways.py: prompted or fine-tuned?

    Score "lab-chat" and the ready fine-tuned model "ft:chota-base:chulha:tags-v1:c0ffee01" on examples("test") with your score_model (from score import score_model). For each, work out what 1,000 messages cost: the model's prices come from client.models.retrieve(model_id).price_inr_per_million, a dict {"input", "output"} (rupees per million tokens); the set costs (prompt_tokens × input + completion_tokens × output) / 1,000,000 rupees. Print one line per model, lab-chat first:

    lab-chat: accuracy A%, format F%, LEAK C/L, ₹R per 1,000 messages

    fine-tuned: accuracy A%, format F%, LEAK C/L, ₹R per 1,000 messages

    (percentages with one decimal, rupees with two)

  4. Task 4 · interview question

    Interview: leaks_missed.py, which leaks got away?

    Mangala asks: "95% sounds good. Which leaks did it miss?" Send every test example whose tag is LEAK to "lab-chat", then to "ft:chota-base:chulha:tags-v1:c0ffee01" (system PROMPT, user message_of). For each model, in that order, print one line per leak it did not tag LEAK:

    NAME: MESSAGE -> REPLY (NAME is lab-chat or fine-tuned)

    then NAME missed M of L leaks

What you will learn

  • What fine-tuning is: more training of a model that is already trained, on examples of your exact job.
  • The lab's API: the same API as OpenAI's, with a big general model and a tiny one you can fine-tune.
  • A held-out set: messages the model never learned from, so its score means something.
  • Three numbers: accuracy, format, and the LEAK messages caught.
  • Cost per message: tokens times price, and why a small fine-tuned model is cheap.

More in Should you fine-tune?

Build "Meet Chulha, and two ways to tag a message" yourself

Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.