AI answer grader project · Chapter 10 of 12

The release gate

By the end of this chapter you can measure any prompt or model on a fixed set of answers the teachers agreed on, find the group of answers a change made worse, and decide in code, by rules written before you look, whether that change may grade Shikhar's students. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Teachers and releases
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI answer grader project are free. This one comes with the all labs plan.

Why you need this

The grader works. Prompt 3 agrees with the agreed marks on 77% of the 200 answers, and anything doubtful goes to Anjali's queue. Now people want to change it. Karthik has a shorter prompt, prompt 4, that reads about 70 fewer tokens on every answer: 3,000 answers a week makes that 210,000 tokens a week. The provider has a new, cheaper model, lab-chat-v2, and a smaller one, lab-chat-mini.

Each change sounds harmless. How would you know? So far you have chapter 5's calibration, which you read by eye, and chapter 6's bias report, which you also read by eye. Nobody runs them on a Friday evening before a change goes live. And a change that is a little better on average can be much worse for one group of students: the ones who write long answers, say. On Saturday night 3,000 students get marks, and nobody notices until a parent calls.

Kavita's rule is simple: nothing grades a student until it has been measured against the teachers, and it may not be worse. This chapter turns that rule into code.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    gate.py: the grader on the gold set

    In app/release/gate.py write score_rows(rows): "mae" (the mean of abs(score - gold_total)), "exact" (the share of rows with score == gold_total) and "point_agreement" (over every row's gold points, the share where row.points.get(id) equals the gold marks), each rounded to 3 places.

    The Check scores four GoldRows of its own, then GET /release/check?prompt=3 as Anjali: the grader in use scores mae 0.325, exact 0.75 and point_agreement 0.917 on the 40 gold answers.

  2. Task 2

    gate.py: slice by slice

    In app/release/gate.py write slice_mae(rows): for each slice of SLICES in order, the mean absolute error of the rows its function keeps, rounded to 3 places, leaving out a slice with no rows.

    The Check slices rows of its own (one slice empty), then GET /release/check?prompt=4: the shorter prompt's long answers are 1.667 marks off, against 0.0 for the prompt in use.

  3. Task 3 · interview question

    Interview: gate.py, may it be released?

    In app/release/gate.py write regressions(current, candidate): the reasons, in order: "mean absolute error <c> is worse than <b>" when the candidate's mae is more than settings.max_mae_increase above the current one's; "exact agreement <c> is lower than <b>" when its exact is more than settings.max_exact_drop below; then "<slice> answers: mean absolute error <c> is worse than <b>" for each slice more than max_mae_increase above. [] when it may be released.

    The Check tries dicts of its own (on the line, just over it, a slice the current grader has no rows in), then GET /release/check for prompt 4 (blocked), lab-chat-v2 (blocked) and lab-chat-mini (released: the same grades for less).

    Karthik will ask: the new model is cheaper and its mean absolute error is only 0.175 marks worse. Why not release it and keep an eye on it?

What you will learn

  • A gold set: 40 answers marked point by point by both teachers and agreed, kept fixed, so two measurements can be compared.
  • Three numbers on it: mean absolute error, exact agreement, and point agreement (are the right points given the marks?).
  • Slices: the same error measured on one group at a time (long answers, short ones, physics, chemistry).
  • A release gate: rules decided before you look, with a small allowance, that say yes or no and give the reasons.
  • Prompt 4: what dropping two sentences does to long answers.

More in Teachers and releases

Build The release gate yourself

Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI answer grader project are free. This one comes with the all labs plan.