AI answer grader project · Chapter 5 of 12
Calibration
By the end of this chapter you can measure how closely two markers agree on the same answers, with four numbers: exact agreement, agreement within one mark, mean absolute error and Cohen's kappa. You can read GET /calibration, which grades all 200 answers and puts the grader beside the two teachers, and say in numbers how good the grader is, and how good it could ever hope to be. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Can it be trusted?
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of AI answer grader project are free. This one comes with the all labs plan.
Why you need this
Kavita's third wish was proof: before the grader marks anyone, show that it is as good as a second teacher. Chapter 4 made each grade safer. It did not say how good the grades are. A grade with no problems can still be wrong: A019 got 5 where both teachers gave 3, and nothing complained.
Shikhar already has what you need to measure it. Last month Anjali and Rakesh marked the same 200 answers separately, then agreed on a mark for each. Every answer in data/answers.json carries all three, beside its text and its student, and the given answers_by_id() reads them:
>>> r = answers_by_id()["A200"]
>>> r.id, r.question_id, r.teacher_a, r.teacher_b, r.agreed
('A200', 'chem-mole', 4, 4, 4)
Comparing a grader with people on answers whose marks are known is called calibration. The code so far grades one answer at a time and has no way to say "on these 200 answers, it is this far from the teachers". This chapter writes that measure.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
metrics.py: how often two markers agree
In app/calibrate/metrics.py write exact_agreement(a, b): the share (0 to 1) of positions where a and b hold the same score. Then within_one(a, b): the share where they differ by at most one mark.
The Check compares Anjali's and Rakesh's marks on six answers of its own (3 the same, 5 within one), two equal lists and two lists that never agree.
- Task 2
metrics.py: how far apart, on average
In app/calibrate/metrics.py write mean_abs_error(a, b): the average of abs(x - y) over the pairs. Then call GET /calibration as Anjali.
The Check tries lists of its own, then reads GET /calibration: on all 200 answers the two teachers are 0.26 marks apart on average, and the grader (prompt 1) is 0.715 marks from the marks they agreed on.
- Task 3 · interview question
Interview: metrics.py, agreement beyond chance
In app/calibrate/metrics.py write cohen_kappa(a, b): (observed - expected) / (1 - expected), with observed the exact agreement and expected the sum, over every score seen, of (share of a with that score) times (share of b with that score); 1.0 when expected is 1.
The Check works out kappa for lists where agreement is only chance (0), perfect (1), a worked example (1/3) and one constant list, then reads GET /calibration: the teachers' kappa is 0.697 and the grader's 0.442.
Karthik will ask: the grader gives exactly the teachers' score on 54% of the answers. Is it 54% good? What would a grader that always says 2 score?
What you will learn
- Calibration: grading answers whose marks are known and comparing.
- Exact agreement and within one: how often two markers give the same score, or nearly.
- Mean absolute error: how many marks apart, on average.
- Cohen's kappa: agreement beyond what luck alone would give.
- The ceiling: two teachers against each other, the most a grader can hope for.
- Reading the report: by question, and the five worst answers.
More in Can it be trusted?
Build "Calibration" yourself
Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI answer grader project are free. This one comes with the all labs plan.