Fine-tuning basics (TypeScript) · Chapter 5 of 20
Splitting without leaks
By the end of this chapter you can split a dataset into train, validation and test so that the test really is unseen: by customer, with a hash that never changes its mind, checked by a leak detector you wrote. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The data
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Chapter 4 made sure no message is in two places. Irfan's next question is about people: "Our test score should say how the model will do next month. Next month's messages come from customers it has never seen. Do ours?"
A customer writes in their own way. Their messages share their spelling, their area ("koradi", "sadar"), their booking ids, and sometimes the same complaint twice with a word changed. If one of those messages is in training and another in test, the test is a little open-book again: the model has met this customer. And the score you report is for "customers we already know", which is not the question Rashmi is asking.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
split-of.ts: the same customer, always the same split
Export fnv1a(s): start with h = 0x811c9dc5; for each character, h = h XOR its code (s.charCodeAt(i)), then h = Math.imul(h, 0x01000193) >>> 0; return h >>> 0.
Export splitOf(phone): the bucket fnv1a(phone) % 20; 0 to 13 is 'train', 14 to 16 'val', 17 to 19 'test'.
- Task 2
split.ts: split the rows by customer
Export splitRows(rows) returning { train, val, test }: each row goes to the split splitOf(row.phone) names, keeping the rows in the order they came.
Under if (import.meta.main), split readyRows() (given in pipeline.ts: the export cleaned, deduplicated and cut to what chota can read) and print:
train N, val N, test N
Your three lists must hold exactly the messages of the given train.jsonl, val.jsonl and test.jsonl, in order.
- Task 3
find-leaks.ts: a leak detector
Export findLeaks(splits) for { train, val, test } of rows, returning { customers, messages }: customers is how many phone numbers have rows in more than one split; messages is how many dedupeKey(text)s (from pipeline.ts) have rows in more than one split. Each phone or key counts once, however many splits it is in; one twice in the same split is no leak.
- Task 4 · interview question
Interview: three-splits.ts, which split leaks?
Irfan: "A colleague split by row number. What leaks, and does deduplicating first fix it?" A split by row number puts row i in train when i % 20 is below 14, in val when below 17, else in test. Run findLeaks on three splits, in this order, and print one line each:
split before dedupe: C customers and M messages in more than one split (cleanRows() by row number)
split by row number: ... (readyRows() by row number)
split by customer: ... (splitRows(readyRows()))
What you will learn
- Three splits: train to learn from, validation to choose with, test to report once.
- Group splits: keep everything of one customer on one side.
- A stable hash: FNV-1a of the phone number, so a customer always lands in the same split.
- A leak detector: customers and messages that sit in two splits, counted.
More in The data
Build "Splitting without leaks" yourself
Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.