AI answer grader project (TypeScript) · Chapter 7 of 12

Consistency

By the end of this chapter you can grade the same answer several times on purpose, measure how far the runs spread, and turn the runs into one steady grade with a vote, point by point. You will also know what temperature and a seed do inside the model, and why, for this grader, one run at temperature 0 is still the better choice. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Can it be trusted?
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of AI answer grader project (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Kavita's rule is simple: two students who write the same words get the same marks. A teacher can be tired on a Sunday, but she will not give the same sentence 1 mark at ten o'clock and 4 marks at eleven.

A model can. Take A004, a phy-ice answer that Anjali and Rakesh both marked 3 of 5:

In ice the molecules are arranged in a cage-like structure held by hydrogen bonds, which leaves empty space between them. It floats because the upward buoyant force equals the weight of the water it displaces.

Graded five times at temperature 1, with nothing else changed, it got 1, 2, 2, 2 and 4. Same answer, same rubric, same prompt. If Shikhar's 3,000 answers a week were graded that way, the mark a student gets would depend partly on luck.

The grader so far grades once and believes what comes back. It cannot see that a grade is unsteady, and it cannot make an unsteady one steadier. This chapter adds both: a way to measure the wobble, and a vote that takes most of it out.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    consistency.ts: the same answer, again and again

    In src/grading/consistency.ts write repeat(ai, question, answer, { runs = 5, temperature = 1, version, model }): for run i from 0, await gradeOnce(ai, question, answer, { version, model, temperature, seed: i }), settled with settle(reply.grade, question, answer). Return the Grades in run order.

    The Check repeats a phy-ice answer three times at temperature 1 and reads the request log (seeds 0, 1, 2 and temperature 1), compares each Grade with its own settle of the model's reply, then repeats at temperature 0 and with no options.

  2. Task 2

    consistency.ts: how far the runs spread

    In src/grading/consistency.ts write spread(grades): the highest score minus the lowest.

    The Check tries Grades of its own; once repeat and vote are written it calls GET /consistency as Anjali too: answer A004 graded 5 times at temperature 1 gets scores 1, 2, 2, 2 and 4 (spread 3); at temperature 0 every run of A013 gets 5 (spread 0).

  3. Task 3 · interview question

    Interview: consistency.ts, the vote

    In src/grading/consistency.ts write vote(grades, question): a Grade whose points are, for every rubric point in rubric order, the medianLow (given) of the runs' marks; score their sum; outOf question.marks; problems every run's problems once each (the same point, code and message), in the order first seen, plus { point: "total", code: "runs_disagree", message: "<n> runs gave scores from <lo> to <hi>" } when the spread is config.maxSpread (2) or more; evidence and reasons the first run's.

    The Check votes on Grades of its own (an odd and an even number of runs, a repeated problem, a spread of 2), then GET /consistency for A004 at temperature 1: the vote is 2 with a runs_disagree problem.

    Karthik will ask: temperature 0 gives the same grade every time. Why would anyone grade an answer five times and vote?

What you will learn

  • Temperature, from inside: how a model picks each word, and why 0 always gives the same pick while 1 samples.
  • A seed: how it fixes the dice, so a run can be repeated exactly.
  • The spread: the highest score minus the lowest, a plain measure of how much a grade moves.
  • The median and medianLow: why a vote takes the middle mark and not the average.
  • Why two objects with the same fields are not ===, and how to find the same problem twice anyway.
  • Measuring before paying: one run at temperature 0 against a vote of five at temperature 1, on all 200 answers.

More in Can it be trusted?

Build "Consistency" yourself

Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI answer grader project (TypeScript) are free. This one comes with the all labs plan.