Fine-tuning basics (TypeScript) · Chapter 2 of 20

Prompt, few-shot, RAG or fine-tune?

By the end of this chapter you can improve a prompted model the three usual ways (rules in the prompt, examples in the prompt, and facts retrieved into the prompt), measure each one on accuracy, format, leaks caught and rupees a month, and say plainly when fine-tuning is the right tool and when it is not. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Should you fine-tune?
  • 4 tasks
  • TypeScript
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

In chapter 1 the big model, asked the obvious way, tagged 60.5% of the validation messages right and caught 13 of 24 leaks. Irfan's fine-tuned chota did far better and was cheaper. So should Chulha fine-tune?

Not so fast, says Irfan. That prompt was one line. Nobody told lab-chat Mangala's rules, and nobody showed it a single tagged message. A fair comparison gives prompting its best shot first: prompting is quicker to change, needs no training data, and is often good enough. Fine-tuning costs a dataset, a job and a model to look after. You should only pay that when you know what prompting cannot do.

Rashmi has her own question, the one she asks about everything: "what does it cost a month?" A longer prompt is read, and paid for, with every one of 30,000 messages.

The lesson

Tokens, once more

Models read tokens: pieces of words. "regulator se sss awaaz aa rahi hai" is a handful of them. A request's usage counts them, and the price is per million. Everything in the request counts: the system message, every example, the customer's words. So every line you add to a prompt is paid for again on every message.

The guide in the prompt

Mangala's guide (guide.md in Chulha's data, 181 words) has the six codes and six rules the desk learned the hard way, like "A regulator or pipe that makes a hissing sound or smells is LEAK". Put it after the instructions in the system message:

const guide = readText('guide.md');
const messages = [
  { role: 'system', content: `${PROMPT}\n\n${guide}` },
  { role: 'user', content: 'regulator se sss awaaz aa rahi hai' },
];
without the guide: CONNECTION  (51 prompt tokens)
with the guide:    LEAK        (299 prompt tokens)

The rule turned a missed leak into a caught one. It also made the request six times longer. On the 195 validation messages, the guide lifts lab-chat from 60.5% to 79.5% and from 13 to 22 of 24 leaks. Format does not move (89.2%): the guide says nothing new about how to reply.

A reminder about lab-chat: it is the lab's predictable stand-in for a big model, not a neural network. It applies a rule when the rule's words are in the prompt, and it knows English and Hindi words but no Marathi. Real models are fuzzier, but they follow the same pattern: rules you write help, and words they have rarely seen stay hard.

Few-shot: show, do not tell

Instead of describing the job, show it. Few-shot prompting puts tagged examples in the request, so the model can copy the pattern. The cleanest way is as earlier turns of the conversation: each example is a user message (what a customer wrote) and an assistant message (the code), as if the model had already answered them.

[
  { role: 'system', content: PROMPT },
  { role: 'user', content: 'hello cylinder khatam ho gaya naya bhejo' },
  { role: 'assistant', content: 'REFILL' },
  { role: 'user', content: 'rasta band hai kya, delivery nhi aayi' },
  { role: 'assistant', content: 'DELIVERY' },
  // ... one or more examples for each code ...
  { role: 'user', content: 'regulator se sss awaaz aa rahi hai' },
]

Two things happen. The model sees what the answers look like, so it replies with a bare code every time: format goes to 100%. And it sees which messages get which code: lab-chat copies the tag of the example that shares the most words with the new message (models do something fuzzier, called in-context learning, with the same effect).

On the validation set, k examples per tag:

Few-shotAccuracyFormatLEAK caughtPrompt tokens per message
2 per tag (12 examples)72.3%100.0%14/24267
5 per tag (30)74.4%100.0%16/24594
10 per tag (60)84.6%100.0%18/241,108

More examples, better tags, longer requests. The examples help only the messages that look like one of them; the rest fall back to general word sense.

Both together

The guide and the examples do different jobs, so they add up:

PromptAccuracyFormatLEAK caughtPrompt tokens per message
guide + 2 per tag89.7%100.0%24/24515
guide + 10 per tag93.3%100.0%24/241,356

That is a strong prompt: every leak caught on the validation set, every reply a code. Irfan's fine-tuned chota scored 90.8% and caught 16 of 24. So far prompting wins on the number Mangala reads first.

What no prompt reached

Some messages beat every prompt here. Marathi for "it smells of gas":

gas cha vaas yetoy
  zero-shot      OTHER
  guide          OTHER
  few-shot 10    OTHER
  fine-tuned     LEAK

No rule in the guide has these words, and no example shares enough of them. The fine-tuned model learned from Mangala's tagged messages, which include Marathi ones. That is the core of what fine-tuning is for: behaviour you can show with many examples but cannot write down, built into the weights so the request stays short.

RAG: facts that change

Customers also ask prices: "laxmi gas ka rate kitna hai". A refill's price changes every month. lab-chat answers from what it learned, which is a month old:

Laxmi Gas: a 14.2 kg refill is Rs 880.

That was October's price. November's is ₹905. Fine-tuning would not fix this: a model does not reliably learn a number from a training file, and next month the number is wrong again. The fix is retrieval: before the request, look up the fact in your own data and put it in the prompt.

function retrieve(question: string): string  // one line from prices.json
// system: 'Laxmi Gas: ₹905 per 14.2 kg refill (November 2026)'
Laxmi Gas: a 14.2 kg refill is Rs 905 now.

That is RAG (retrieval-augmented generation) at its smallest: one lookup, one line of context. The RAG lab does it properly, with search over many documents. And the fine-tuned tagger? Asked the price, it replies PAYMENT. It was trained to tag, and it does exactly that, nothing more.

What a month costs

Each request's usage times the model's prices, per message, times 30,000. On the validation set:

SystemAccuracyLEAK caught₹ a month
lab-chat, zero-shot60.5%13/24₹472
lab-chat, guide79.5%22/24₹2,333
lab-chat, 10 per tag84.6%18/24₹8,365
lab-chat, guide + 10 per tag93.3%24/24₹10,226
fine-tuned chota90.8%16/24₹2

The best prompt costs about 5,000 times as much a month as the fine-tuned model, because it sends 1,356 tokens to a big model to tag a message of ten. Irfan's training job cost ₹9.80, once: 39,207 tokens trained at ₹250 per million (chapter 7 works this out).

So when do you fine-tune?

  • Prompt first when it is good enough: it is quick, and you change it in a minute.
  • Retrieve facts that change: prices, stock, a customer's booking.
  • Fine-tune for behaviour and format at volume: a narrow job done the same way every time, rules too many or too subtle to write, a language the big model is weak in, and when a long prompt costs too much per message.

For Chulha today, the measurements say: the best prompt catches every leak and costs ₹10,226 a month; the fine-tuned model costs ₹2 and misses leaks. The rest of this lab is about the data and the training that make a fine-tuned model good enough for Mangala.

Your turn

Put the guide in the prompt in task 1 and few-shot examples in task 2, each measured on the test set. Retrieve this month's price in task 3. In the interview question, Rashmi wants a decision: four systems, their numbers, their cost a month, and the one Mangala's rule picks.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    with-guide.ts: put the guide in the prompt

    Score 'lab-chat' on examples('test') with Mangala's guide in the prompt: two messages a request, a system message that is PROMPT, a blank line, then the whole of readText('guide.md'), and the user message. Use the given scoreModel (score.ts) with its third argument, which builds each request's messages. Print one line:

    guide: accuracy A%, format F%, LEAK C/L, P prompt tokens per message

    (P: the prompt tokens of every request added up, divided by the number of test messages, one decimal)

  2. Task 2

    few-shot.ts: show lab-chat examples

    Export shots(k): for each tag in TAGS order, the first k examples of examples('train') with that tag, each as two messages, a user message (the customer's words) then an assistant message (the tag). shots(2) is 24 messages.

    Under if (import.meta.main), for k = 2, 5 and 10, score 'lab-chat' on examples('test') with each request built as the system message PROMPT, then shots(k), then the user message. Print one line per k:

    few-shot K per tag: accuracy A%, format F%, LEAK C/L, P prompt tokens per message

  3. Task 3

    rag.ts: retrieve this month's price

    Export retrieve(question): find the agency the question names (its name without " Gas", like "laxmi" or "shree ram", as a whole word of the question, in any case) and return one line with its November price from prices.json (months['2026-11']), like Laxmi Gas: ₹905 per 14.2 kg refill (November 2026). No agency named: return ''.

    Under if (import.meta.main), ask 'aaj sai gas ka rate kya hai' three times: lab-chat with the question alone, lab-chat with retrieve(question) as the system message, and 'ft:chota-base:chulha:tags-v1:c0ffee01' with the question alone. Print:

    without context: REPLY

    with retrieval: REPLY

    fine-tuned: REPLY

  4. Task 4 · interview question

    Interview: monthly.ts, which one and what does it cost?

    Rashmi: "Which one, and what does it cost a month?" Score four systems on examples('test'), in this order: zero-shot (lab-chat, system PROMPT), guide (lab-chat, the system message of task 1), guide + 10 per tag (lab-chat, the guide's system message, then shots(10), then the message), fine-tuned ('ft:chota-base:chulha:tags-v1:c0ffee01', system PROMPT). A month is 30,000 messages: the set's cost (promptTokens × input + completionTokens × output) / 1,000,000 rupees, with the prices from client.models.retrieve(id) (price_inr_per_million), per message, times 30,000, rounded to whole rupees. Print one line per system:

    NAME: accuracy A%, LEAK C/L, ₹R a month

    then pick: NAME, Mangala's rule: of the systems that caught every leak, the most accurate (the cheaper on a tie).

What you will learn

  • Rules in the prompt: Mangala's guide as part of the system message.
  • Few-shot: tagged examples sent as earlier turns, and why the model copies them.
  • RAG (retrieval-augmented generation): look up a fact that changes, and put it in the prompt.
  • What each costs: prompt tokens per message, and rupees a month.
  • When to fine-tune: behaviour and format; not facts that change.

More in Should you fine-tune?

Build "Prompt, few-shot, RAG or fine-tune?" yourself

Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.