Fine-tuning basics (TypeScript) · Chapter 5 of 20

Splitting without leaks

By the end of this chapter you can split a dataset into train, validation and test so that the test really is unseen: by customer, with a hash that never changes its mind, checked by a leak detector you wrote. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The data
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Chapter 4 made sure no message is in two places. Irfan's next question is about people: "Our test score should say how the model will do next month. Next month's messages come from customers it has never seen. Do ours?"

A customer writes in their own way. Their messages share their spelling, their area ("koradi", "sadar"), their booking ids, and sometimes the same complaint twice with a word changed. If one of those messages is in training and another in test, the test is a little open-book again: the model has met this customer. And the score you report is for "customers we already know", which is not the question Rashmi is asking.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    split-of.ts: the same customer, always the same split

    Export fnv1a(s): start with h = 0x811c9dc5; for each character, h = h XOR its code (s.charCodeAt(i)), then h = Math.imul(h, 0x01000193) >>> 0; return h >>> 0.

    Export splitOf(phone): the bucket fnv1a(phone) % 20; 0 to 13 is 'train', 14 to 16 'val', 17 to 19 'test'.

  2. Task 2

    split.ts: split the rows by customer

    Export splitRows(rows) returning { train, val, test }: each row goes to the split splitOf(row.phone) names, keeping the rows in the order they came.

    Under if (import.meta.main), split readyRows() (given in pipeline.ts: the export cleaned, deduplicated and cut to what chota can read) and print:

    train N, val N, test N

    Your three lists must hold exactly the messages of the given train.jsonl, val.jsonl and test.jsonl, in order.

  3. Task 3

    find-leaks.ts: a leak detector

    Export findLeaks(splits) for { train, val, test } of rows, returning { customers, messages }: customers is how many phone numbers have rows in more than one split; messages is how many dedupeKey(text)s (from pipeline.ts) have rows in more than one split. Each phone or key counts once, however many splits it is in; one twice in the same split is no leak.

  4. Task 4 · interview question

    Interview: three-splits.ts, which split leaks?

    Irfan: "A colleague split by row number. What leaks, and does deduplicating first fix it?" A split by row number puts row i in train when i % 20 is below 14, in val when below 17, else in test. Run findLeaks on three splits, in this order, and print one line each:

    split before dedupe: C customers and M messages in more than one split (cleanRows() by row number)

    split by row number: ... (readyRows() by row number)

    split by customer: ... (splitRows(readyRows()))

What you will learn

  • Three splits: train to learn from, validation to choose with, test to report once.
  • Group splits: keep everything of one customer on one side.
  • A stable hash: FNV-1a of the phone number, so a customer always lands in the same split.
  • A leak detector: customers and messages that sit in two splits, counted.

More in The data

Build "Splitting without leaks" yourself

Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.