Fine-tuning basics · Chapter 5 of 20
Splitting without leaks
By the end of this chapter you can split a dataset into train, validation and test so that the test really is unseen: by customer, with a hash that never changes its mind, checked by a leak detector you wrote. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The data
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.
Why you need this
Chapter 4 made sure no message is in two places. Irfan's next question is about people: "Our test score should say how the model will do next month. Next month's messages come from customers it has never seen. Do ours?"
A customer writes in their own way. Their messages share their spelling, their area ("koradi", "sadar"), their booking ids, and sometimes the same complaint twice with a word changed. If one of those messages is in training and another in test, the test is a little open-book again: the model has met this customer. And the score you report is for "customers we already know", which is not the question Rashmi is asking.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
split_of.py: the same customer, always the same split
Write fnv1a(s): start with h = 0x811C9DC5; for each character, h = h XOR its code (h ^= ord(ch)), then h = (h * 0x01000193) & 0xFFFFFFFF; return h.
Write split_of(phone): the bucket fnv1a(phone) % 20; 0 to 13 is "train", 14 to 16 "val", 17 to 19 "test".
- Task 2
split.py: split the rows by customer
Write split_rows(rows) returning a dict {"train", "val", "test"} of lists: each row goes to the split split_of(row["phone"]) names, keeping the rows in the order they came.
Under if __name__ == "__main__":, split ready_rows() (given in pipeline.py: the export cleaned, deduplicated and cut to what chota can read) and print:
train N, val N, test N
Your three lists must hold exactly the messages of the given train.jsonl, val.jsonl and test.jsonl, in order.
- Task 3
find_leaks.py: a leak detector
Write find_leaks(splits) for a dict {"train", "val", "test"} of rows, returning a dict {"customers", "messages"}: customers is how many phone numbers have rows in more than one split; messages is how many dedupe_key(text)s (from pipeline.py) have rows in more than one split. Each phone or key counts once, however many splits it is in; one twice in the same split is no leak.
- Task 4 · interview question
Interview: three_splits.py, which split leaks?
Irfan: "A colleague split by row number. What leaks, and does deduplicating first fix it?" A split by row number puts row i in train when i % 20 is below 14, in val when below 17, else in test. Run find_leaks on three splits, in this order, and print one line each:
split before dedupe: C customers and M messages in more than one split (clean_rows() by row number)
split by row number: ... (ready_rows() by row number)
split by customer: ... (split_rows(ready_rows()))
What you will learn
- Three splits: train to learn from, validation to choose with, test to report once.
- Group splits: keep everything of one customer on one side.
- A stable hash: FNV-1a of the phone number, so a customer always lands in the same split.
- A leak detector: customers and messages that sit in two splits, counted.
More in The data
Build "Splitting without leaks" yourself
Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.