Fine-tuning basics · Chapter 5 of 20

Splitting without leaks

By the end of this chapter you can split a dataset into train, validation and test so that the test really is unseen: by customer, with a hash that never changes its mind, checked by a leak detector you wrote. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The data
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.

Why you need this

Chapter 4 made sure no message is in two places. Irfan's next question is about people: "Our test score should say how the model will do next month. Next month's messages come from customers it has never seen. Do ours?"

A customer writes in their own way. Their messages share their spelling, their area ("koradi", "sadar"), their booking ids, and sometimes the same complaint twice with a word changed. If one of those messages is in training and another in test, the test is a little open-book again: the model has met this customer. And the score you report is for "customers we already know", which is not the question Rashmi is asking.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    split_of.py: the same customer, always the same split

    Write fnv1a(s): start with h = 0x811C9DC5; for each character, h = h XOR its code (h ^= ord(ch)), then h = (h * 0x01000193) & 0xFFFFFFFF; return h.

    Write split_of(phone): the bucket fnv1a(phone) % 20; 0 to 13 is "train", 14 to 16 "val", 17 to 19 "test".

  2. Task 2

    split.py: split the rows by customer

    Write split_rows(rows) returning a dict {"train", "val", "test"} of lists: each row goes to the split split_of(row["phone"]) names, keeping the rows in the order they came.

    Under if __name__ == "__main__":, split ready_rows() (given in pipeline.py: the export cleaned, deduplicated and cut to what chota can read) and print:

    train N, val N, test N

    Your three lists must hold exactly the messages of the given train.jsonl, val.jsonl and test.jsonl, in order.

  3. Task 3

    find_leaks.py: a leak detector

    Write find_leaks(splits) for a dict {"train", "val", "test"} of rows, returning a dict {"customers", "messages"}: customers is how many phone numbers have rows in more than one split; messages is how many dedupe_key(text)s (from pipeline.py) have rows in more than one split. Each phone or key counts once, however many splits it is in; one twice in the same split is no leak.

  4. Task 4 · interview question

    Interview: three_splits.py, which split leaks?

    Irfan: "A colleague split by row number. What leaks, and does deduplicating first fix it?" A split by row number puts row i in train when i % 20 is below 14, in val when below 17, else in test. Run find_leaks on three splits, in this order, and print one line each:

    split before dedupe: C customers and M messages in more than one split (clean_rows() by row number)

    split by row number: ... (ready_rows() by row number)

    split by customer: ... (split_rows(ready_rows()))

What you will learn

  • Three splits: train to learn from, validation to choose with, test to report once.
  • Group splits: keep everything of one customer on one side.
  • A stable hash: FNV-1a of the phone number, so a customer always lands in the same split.
  • A leak detector: customers and messages that sit in two splits, counted.

More in The data

Build "Splitting without leaks" yourself

Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.