AI answer grader project (TypeScript) · Chapter 10 of 12
The release gate
By the end of this chapter you can measure any prompt or model on a fixed set of answers the teachers agreed on, find the group of answers a change made worse, and decide in code, by rules written before you look, whether that change may grade Shikhar's students. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Teachers and releases
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of AI answer grader project (TypeScript) are free. This one comes with the all labs plan.
Why you need this
The grader works. Prompt 3 agrees with the agreed marks on 77% of the 200 answers, and anything doubtful goes to Anjali's queue. Now people want to change it. Karthik has a shorter prompt, prompt 4, that reads about 70 fewer tokens on every answer: 3,000 answers a week makes that 210,000 tokens a week. The provider has a new, cheaper model, lab-chat-v2, and a smaller one, lab-chat-mini.
Each change sounds harmless. How would you know? So far you have chapter 5's calibration and chapter 6's bias report, and you read both by eye. Nobody runs them on a Friday evening before a change goes live. And a change that is a little better on average can be much worse for one group of students: the ones who write long answers, say. On Saturday night 3,000 students get marks, and nobody notices until a parent calls.
Kavita's rule is simple: nothing grades a student until it has been measured against the teachers, and it may not be worse. This chapter turns that rule into code.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
gate.ts: the grader on the gold set
In src/release/gate.ts write scoreRows(rows): mae (the mean of Math.abs(score - goldTotal)), exact (the share of rows with score === goldTotal) and pointAgreement (over every row's gold points, the share where row.points[id] equals the gold marks), each rounded to 3 places.
The Check scores four GoldRows of its own and one whose grader left out a point, then the 40 gold answers graded with prompt 3, the grader in use: mae 0.325, exact 0.75 and pointAgreement 0.917.
- Task 2
gate.ts: slice by slice
In src/release/gate.ts write sliceMae(rows): for each slice of SLICES in order, the mean absolute error of the rows its function keeps, rounded to 3 places, leaving out a slice with no rows.
The Check slices rows of its own (some slices empty), then the gold set graded with prompt 4 and prompt 3: the shorter prompt's long answers are 1.667 marks off, against 0 for the prompt in use.
- Task 3 · interview question
Interview: gate.ts, may it be released?
In src/release/gate.ts write regressions(current, candidate): the reasons, in order: "mean absolute error <c> is worse than <b>" when the candidate's mae is more than config.maxMaeIncrease above the current one's; "exact agreement <c> is lower than <b>" when its exact is more than config.maxExactDrop below; then "<slice> answers: mean absolute error <c> is worse than <b>" for each slice more than maxMaeIncrease above. [] when it may be released.
The Check tries objects of its own (on the line, just over it, a slice the current grader has no rows in, worse everywhere), then GET /release/check for prompt 4 (blocked), lab-chat-v2 (blocked) and lab-chat-mini (released: the same grades for less).
Karthik will ask: the new model is cheaper and its mean absolute error is only 0.175 marks worse. Why not release it and keep an eye on it?
What you will learn
- A gold set: 40 answers marked point by point by both teachers and agreed, kept fixed, so two measurements can be compared.
- Three numbers on it: mean absolute error, exact agreement, and point agreement (did the right points get the marks?).
- Slices: the same error measured on one group at a time (long answers, short ones, physics, chemistry).
- A release gate: rules decided before you look, with a small allowance, that say yes or no and give the reasons.
- Prompt 4: what dropping two sentences does to long answers.
More in Teachers and releases
Build The release gate yourself
Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI answer grader project (TypeScript) are free. This one comes with the all labs plan.