Fine-tuning basics (TypeScript) · Chapter 10 of 20
Inside chota-base
By the end of this chapter you can say what a language model is made of and count it, get its prediction for the next piece of text as probabilities, see exactly which predictions fine-tuning trains, and explain from the numbers why chota-base, before fine-tuning, answers a tagging request with nothing. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Inside
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.
Why you need this
So far a fine-tuning job has been a box: files in, a model id out, some losses in between. Irfan wants to stop guessing about what the box does. The next chapters change chota's weights with your own code, and you cannot change something you have never looked at.
There is also Mangala's question from chapter 1, still unanswered. chota-base read two years of Chulha's customers' messages. Ask it to tag one and it says nothing at all. Why?
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
count.ts: what the 45,152 numbers are
Write and export paramGroups(model): add up the sizes (w.data.length) of model.params by name: embeddings ('tok', 'pos'), attention (names ending .wq, .wk, .wv, .wo), feedForward (.fc_w, .fc_b, .proj_w, .proj_b), layerNorm (every other name with 'ln' in it), and total (every weight). Return { embeddings, attention, feedForward, layerNorm, total }; work from the names, as the check also tries a model of another shape.
Under if (import.meta.main), print one line per group for chota-base, in that order:
embeddings: N
- Task 2
next-token.ts: what chota expects next
Write and export nextTokens(model, text, k): run forward(model, [[USER, ...encode(text)]]), take the last position's scores (logits.cols of them), turn them into probabilities with softmax (exp(score - max) divided by the sum over the row), and return the k most likely, most likely first, as { piece: pieces([id])[0], p }.
Under if (import.meta.main), print the 5 most likely pieces after 'delivery' for chota-base, one per line:
PIECE P% (one decimal)
- Task 3
template.ts: where the loss is taken
Write and export trainedPairs(messages): with row = chatIds(messages), every position t where makeBatch([row]).weights[0][t] is 1, as the pair [pieces(row.ids)[t], pieces(row.ids)[t + 1]].
Under if (import.meta.main), for user 'cylinder kab aayega' and assistant 'DELIVERY', print the pieces joined with spaces, then one line per pair:
PIECE -> NEXT
- Task 4 · interview question
Interview: why-silent.ts, why does chota-base say nothing?
Mangala: "Irfan says chota-base read two years of our customers' messages. Why does it answer nothing?" For each of the first 3 test messages, take promptIds(messageOf(e)), the probabilities of the next piece at its last position, and print the most likely:
MESSAGE: after <bot>, PIECE P% (one decimal)
Then answer every test message with generateMany(model, messages, 4) and print:
chota-base replied with nothing to N of M test messages
What you will learn
- Weights: what chota's 45,152 numbers are, by kind.
- The next piece: scores for every piece in the vocabulary (logits), turned into probabilities (softmax).
- Greedy decoding: how a reply is made one piece at a time.
- What is trained: the chat template's marks, and which predictions the loss counts.
- The loss: how wrong a prediction is, as one number, from the probability of the right piece.
More in Inside
Build "Inside chota-base" yourself
Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.