Fine-tuning basics · Chapter 18 of 20

Evaluating before and after

By the end of this chapter you can turn a model's replies into a confusion matrix, read recall and precision for each tag from it, score free-text replies with a model as a judge (and know where a judge goes wrong), and compare a prompted and a fine-tuned system honestly on the same held-out messages. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Proof
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.

Why you need this

Rashmi has one number from every chapter so far, and she likes the fine-tuned one: "95% against 91%. The fine-tuned model is better. Done?"

Mangala does not answer with a number. She takes the desk's printout of the replies and starts making piles: the real leaks here, and next to each one, what the model called it. "I do not care about 95. I care that a hissing cylinder went to the refill queue."

Accuracy averages six codes into one number, and the six are not equally common or equally costly. You already counted LEAK caught on its own in chapter 1. This chapter makes that idea complete: every tag, both kinds of mistake, and a way to score replies that are not exact codes at all. Then you put four systems side by side, once, on the test set.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    confusion.py: a confusion matrix

    Write COLUMNS (the six TAGS, then "not a code") and confusion_matrix(expected, got): a dict with a row for each of the six TAGS, each row a dict with a count for each of COLUMNS, every cell starting at 0, and each message adding 1 to row expected[i], column got[i] (or "not a code" when the reply is not exactly one of the six codes).

    Under if __name__ == "__main__":, get the fine-tuned model's replies on examples("test") with replies_of (replies.py, given) and print one line per tag, in TAGS order, with the cells above 0 in COLUMNS order:

    LEAK: REFILL 6, DELIVERY 2, LEAK 16 (the validation set's line; yours is the test set's)

  2. Task 2

    per_tag.py: recall and precision for each tag

    Write per_tag(m): for each of the six TAGS, a dict {"precision", "recall", "support"}: support is the row added up ("not a code" too); recall is m[tag][tag] / support; precision is m[tag][tag] divided by the column (every row's count for that tag). Shares 0 to 1, and 0 when the divisor is 0.

    Under if __name__ == "__main__":, the fine-tuned model on examples("test"), one line per tag in TAGS order:

    TAG: recall R%, precision P%, N messages (one decimal)

  3. Task 3

    judge.py: a judge for replies that are not exact

    Write judge(message, output, expected): one request to "lab-chat" with a system message asking it to judge (or grade) a tagger's output and reply in JSON, and a user message of three lines, MESSAGE: ..., OUTPUT: ..., EXPECTED: ...; return the reply parsed, a dict {"verdict", "reason"}.

    Under if __name__ == "__main__":, tag examples("test") with "lab-chat" and the system message PROMPT.replace(" Reply with the code only.", ""), judge every reply against tag_of(e), and print:

    exact: X%

    judged: Y%

  4. Task 4 · interview question

    Interview: before_after.py, is 95.2% better than 90.9%?

    Score four systems on examples("test"), in this order, with replies_of: zero-shot ("lab-chat", plain), best prompt ("lab-chat", best_prompt), chota-base ("chota-base", plain), fine-tuned ("ft:chota-base:chulha:tags-v1:c0ffee01", plain). Print a line each:

    NAME: accuracy A%, format F%, LEAK recall L%, worst recall W% (TAG) (worst: the lowest of the six recalls, the first in TAGS order on a tie)

    then, message by message:

    fine-tuned against best prompt: both right N, only fine-tuned F, only best prompt P, both wrong B

What you will learn

  • A confusion matrix: a row for what the message was, a column for what the model said.
  • Recall and precision per tag: missed ones, and false alarms.
  • A judge: a model that scores replies exact match cannot, and how to check the judge.
  • Before and after: four systems on the same messages, and where two of them really differ.

More in Proof

Build "Evaluating before and after" yourself

Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.