AI answer grader project · Chapter 7 of 12
Consistency
By the end of this chapter you can grade the same answer several times on purpose, measure how far the runs spread, and turn the runs into one steady grade with a vote, point by point. You will also know what temperature and a seed do inside the model, and why, for this grader, one run at temperature 0 is still the better choice. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Can it be trusted?
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of AI answer grader project are free. This one comes with the all labs plan.
Why you need this
Kavita's rule is simple: two students who write the same words get the same marks. A teacher can be tired on a Sunday, but she will not give the same sentence 1 mark at ten o'clock and 4 marks at eleven.
A model can. Take A004, a phy-ice answer that Anjali and Rakesh both marked 3 of 5:
In ice the molecules are arranged in a cage-like structure held by hydrogen bonds, which leaves empty space between them. It floats because the upward buoyant force equals the weight of the water it displaces.
Graded five times at temperature 1, with nothing else changed, it got 1, 2, 2, 2 and 4. Same answer, same rubric, same prompt. If Shikhar's 3,000 answers a week were graded that way, the mark a student gets would depend partly on luck.
The grader so far grades once and believes what comes back. It has no way to see that a grade is unsteady, and no way to make an unsteady one steadier. This chapter adds both: a way to measure the wobble, and a vote that takes it out.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
consistency.py: the same answer, again and again
In app/grading/consistency.py write repeat(ai, question, answer, *, runs=5, temperature=1.0, version=None, model=None): for run i from 0, grade_once(ai, question, answer, version=version, model=model, temperature=temperature, seed=i), settled with settle(reply.grade, question, answer). Return the Grades in run order.
The Check repeats a phy-ice answer three times at temperature 1 and reads the request log (seeds 0, 1, 2 and temperature 1), compares each Grade with its own settle of the model's reply, then repeats at temperature 0.
- Task 2
consistency.py: how far the runs spread
In app/grading/consistency.py write spread(grades): the highest score minus the lowest.
The Check tries Grades of its own, then GET /consistency as Anjali: answer A004 graded 5 times at temperature 1 gets scores 1, 2, 2, 2 and 4 (spread 3); at temperature 0 every run of A013 gets 5 (spread 0).
- Task 3 · interview question
Interview: consistency.py, the vote
In app/grading/consistency.py write vote(grades, question): a Grade whose points are, for every rubric point in rubric order, the median_low of the runs' marks; score their sum; out_of question.marks; problems every run's problems once each, in the order first seen, plus Problem("total", "runs_disagree", "<n> runs gave scores from <lo> to <hi>") when the spread is settings.max_spread (2) or more; evidence and reasons the first run's.
The Check votes on Grades of its own (an odd and an even number of runs, a repeated problem, a spread of 2), then GET /consistency for A004 at temperature 1: the vote is 2 with a runs_disagree problem.
Karthik will ask: temperature 0 gives the same grade every time. Why would anyone grade an answer five times and vote?
What you will learn
- Temperature, from inside: how a model picks each word, and why 0 always gives the same pick while 1 samples.
- A seed: how it fixes the dice, so a run can be repeated exactly.
- The spread: the highest score minus the lowest, a plain measure of how much a grade moves.
- The median and median_low: why a vote takes the middle mark and not the average.
- Measuring before paying: one run at temperature 0 against a vote of five at temperature 1, on all 200 answers.
More in Can it be trusted?
Build "Consistency" yourself
Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI answer grader project are free. This one comes with the all labs plan.