AI answer grader project (TypeScript) · Chapter 5 of 12
Calibration
By the end of this chapter you can measure how closely two markers agree on the same answers, with four numbers: exact agreement, agreement within one mark, mean absolute error and Cohen's kappa. You can read GET /calibration, which grades all 200 answers and puts the grader beside the two teachers, and say in numbers how good the grader is, and how good it could ever hope to be. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Can it be trusted?
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of AI answer grader project (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Kavita's third wish was proof: before the grader marks anyone, show that it is as good as a second teacher. Chapter 4 made each grade safer. It did not say how good the grades are. A grade with no problems can still be wrong: A019 got 5 where both teachers gave 3, and nothing complained.
Shikhar already has what you need to measure it. Last month Anjali and Rakesh marked the same 200 answers separately, then agreed on a mark for each. Every answer in data/answers.json carries all three beside its text, and the given answersById() (in src/data.ts) reads them:
console.log(answersById().get('A200'));
{
id: 'A200',
questionId: 'chem-mole',
studentId: 'S2075',
text: "One mole is a fixed number of particles, the way a dozen always means twelve. Avogadro's number, about 6.02 x 10^23, is the number of particles in one mole.",
teacherA: 4,
teacherB: 4,
agreed: 4
}
Comparing a grader with people on answers whose marks are known is called calibration. The code so far grades one answer at a time and has no way to say "on these 200 answers, it is this far from the teachers". This chapter writes that measure.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
metrics.ts: how often two markers agree
In src/calibrate/metrics.ts write exactAgreement(a, b): the share (0 to 1) of positions where a and b hold the same score. Then withinOne(a, b): the share where they differ by at most one mark.
The Check compares Anjali's and Rakesh's marks on six answers of its own (3 the same, 5 within one), two equal lists, two lists that never agree and one with scores two apart.
- Task 2
metrics.ts: how far apart, on average
In src/calibrate/metrics.ts write meanAbsError(a, b): the average of Math.abs(a[i] - b[i]) over the positions. Once the other metrics are written, call GET /calibration as Anjali.
The Check tries lists of its own; once every metric is written it reads GET /calibration too: on all 200 answers the two teachers are 0.26 marks apart on average, and the grader (prompt 1) is 0.715 marks from the marks they agreed on.
- Task 3 · interview question
Interview: metrics.ts, agreement beyond chance
In src/calibrate/metrics.ts write cohenKappa(a, b): (observed - expected) / (1 - expected), with observed the exact agreement and expected the sum, over every score seen in either list, of (share of a with that score) times (share of b with that score); 1 when expected is 1.
The Check works out kappa for lists where agreement is only chance (0), perfect (1), a worked example (1/3), one constant list and a grader that always says 2, then reads GET /calibration: the teachers' kappa is 0.697 and the grader's 0.442.
Karthik will ask: the grader gives exactly the teachers' score on 54% of the answers. Is it 54% good? What would a grader that always says 2 score?
What you will learn
- Calibration: grading answers whose marks are known and comparing.
- Exact agreement and within one: how often two markers give the same score, or nearly.
- Mean absolute error: how many marks apart, on average.
- Cohen's kappa: agreement beyond what luck alone would give.
- The ceiling: two teachers against each other, the most a grader can hope for.
- Reading the report: by question, and the five worst answers.
More in Can it be trusted?
Build "Calibration" yourself
Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI answer grader project (TypeScript) are free. This one comes with the all labs plan.