AI answer grader project (TypeScript) · Chapter 4 of 12

Checking the grade

By the end of this chapter you can check every grade the model gives before anyone sees it: that each quote is really in the student's answer, that each point's marks are within what the point is worth, that every rubric point is graded once, and that the total adds up. Then you settle the grade the app keeps: marks kept in range, the score your own sum, and every problem listed for a teacher. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: A first grader
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of AI answer grader project (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Chapter 3's grader sends one answer to the model and hands its grade straight back. Kavita wants marks on Saturday night for about 3,000 answers a week, so nobody will read most of those grades before the students do. Whatever the model writes, Aditi sees.

Here is what the lab model wrote for three of the week's real answers (data/answers.json):

  • A013 is a long, full answer to phy-ice. The model gave p1 3 marks. p1 is worth 2. Its total was 6 out of 5.
  • A003 is a full answer with two spelling slips ("teh", "becuase"). The model quoted it with the slips corrected. Those are not the words the student wrote.
  • A019 answers phy-satellite. The model's points add up to 5, but its total says 4.

Over all 200 answers, chapter 3's grader with prompt 1 makes 27 quotes that are not in the answer, 19 wrong totals and 12 marks out of range: 48 answers have at least one problem. Model graders really do this: they add up carelessly, they tidy up what they quote, and a long answer can push a mark past the point's maximum. And every one of those replies was valid JSON that readGrade accepted. The schema is not enough: this chapter checks what the grade says.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    verify.ts: is the quote in the answer?

    In src/grading/verify.ts write evidenceFound(evidence, answer): squash both (the given squash keeps only the words, lower case, one space apart) and return true when " " + quote + " " is inside " " + answer + " "; an empty quote is never found.

    The Check tries quotes against one answer: the same words in other case and spacing (found), part of a sentence (found), half a word ("less den": not found), words not in the answer, the empty quote and one of only dots (not found), and a quote that corrects the answer's spelling slip (not found).

  2. Task 2

    verify.ts: every problem with a grade

    In src/grading/verify.ts write checkGrade(grade, question, answer): for each of the model's points in order, "unknown_point" (not in the rubric) or "duplicate_point" (seen already), else "out_of_range" (awarded not 0 to the point's marks), "no_evidence" (marks but a blank quote) or "evidence_not_found" (a quote evidenceFound refuses); then "missing_point" for each rubric point not graded; then { point: "total", code: "total_mismatch" } when the total is not the sum of the awarded marks. Each Problem has a message too.

    The Check builds grades of its own with each problem and a clean one (no problems), then grades three of the week's answers with the lab model: a long full answer (3 marks on a 2-mark point), an answer with spelling slips (quoted with the slips corrected) and one whose total leaves out a point.

  3. Task 3 · interview question

    Interview: verify.ts, the grade the app keeps

    In src/grading/verify.ts write settle(grade, question, answer): a Grade with points (every rubric point's marks, in rubric order: the model's first entry for it kept between 0 and the point's marks, 0 when left out), score (the sum of those, never the model's total), outOf (question.marks), problems (checkGrade) and evidence and reasons (the entry's, "" when left out).

    The Check settles grades of its own (3 marks on a 2-mark point, a missing point, a wrong total, a point graded twice, -2 marks), then POST /grade with the long full answer: score 5 of 5, an out_of_range problem and needsReview.

    Karthik will ask: when a quote is not in the answer, why send the grade to a teacher instead of giving that point 0?

What you will learn

  • What strict does not promise: the right shape, not the right marks.
  • A quote as a claim: why evidence lets code check a grade, and how to look for a quote fairly (squash, whole words only).
  • Problems with codes: unknown_point, duplicate_point, out_of_range, no_evidence, evidence_not_found, missing_point, total_mismatch, each with a sentence a teacher can read.
  • Settling a grade: walk the rubric, keep marks in range, add them up yourself.
  • needsReview: which problems send a grade to a teacher.

More in A first grader

Build "Checking the grade" yourself

Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI answer grader project (TypeScript) are free. This one comes with the all labs plan.