AI answer grader project · Chapter 2 of 12

Rubrics as data

By the end of this chapter you can describe a rubric as Pydantic classes, with a limit on every field and a rule that the points add up, and load a whole file of questions so that every problem in it is listed in one go, before anyone grades with it. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: A first grader
  • 3 tasks
  • Python
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

The grader will mark 3,000 answers a week against the rubrics in a JSON file, and the teachers write those rubrics. Next week's draft, data/rubrics-draft.json, has four new questions from the teachers. In chapter 1 a question was a plain dict, read by raw_questions(), which reads anything. Here is chapter 1's keyword grader on the draft, with a full answer to phy-lens and any answer to chem-ph:

{'points': {'p1': 2, 'p2': 2, 'p3': 2}, 'total': 6}
{'points': {'p1': 0, 'p3': 0}, 'total': 0}

6 marks on a 5-mark question: phy-lens's points are worth 2, 2 and

  1. And chem-ph has two points called p1, so the dict kept only the second one, and a point worth 2 marks vanished without a word. No error, no warning. Aditi would see "6 of 5" on Saturday night, or lose two marks she earned, and nobody would know why until a student complained.

A rubric is data the grader trusts. So it must be checked the moment it is read, and a teacher fixing it must see every problem at once.

The lesson

Tour of the codebase

New or changed since chapter 1:

app/rubric/models.py              yours: RubricPoint, Question (Tasks 1, 2),
                                  load_questions (Task 3); RubricError and
                                  Question.point given
app/modules/questions/router.py   given: GET /questions, GET /questions/{id},
                                  GET /questions/check-draft (teachers)
app/main.py                       changed: includes the questions router
app/baseline.py                   given now: chapter 1's keyword grader
data/rubrics-draft.json           next week's four draft questions

RubricError holds a list of lines, problems, one per problem. Question.point("p2") finds a point by its id (chapter 4 uses it). GET /questions lists the 12 questions, GET /questions/{id} shows one with its rubric (404 for an id that is not there), and GET /questions/check-draft loads the draft and answers {"ok": false, "problems": [...]} or {"ok": true, ...}.

All three call your load_questions, so before Task 3, "The week's questions" answers 500 and the terminal ends with:

  File "/tmp/w/02-rubrics-as-data-starter/app/rubric/models.py", line 92, in load_questions
    raise NotImplementedError("Task 3: load_questions is not written yet")
NotImplementedError: Task 3: load_questions is not written yet

A class for each part

Pydantic turns a class with type hints into a checker. You give it a dict (from JSON), and it either gives back an object whose fields are all the right type, or raises a ValidationError that says what is wrong. Here is a made-up example, not a rubric: a slot in Shikhar's timetable.

class Slot(BaseModel):
    model_config = ConfigDict(extra="forbid")

    room: str = Field(pattern=r"^R[0-9]{3}$")
    batch: Literal["JEE-27A", "JEE-27B", "NEET-27A"]
    start: int = Field(ge=8, le=20)
    end: int = Field(ge=9, le=21)
    topics: list[str] = Field(default=[], max_length=3)

Read each line as "the type says what kind of value, Field says which values are allowed":

  • pattern: the string must match this regular expression. ^ and $ mean the whole string, so "R204" passes and "204" does not.
  • min_length and max_length: on a string they count characters; on a list they count items. A list with min_length=1 cannot be empty.
  • ge and le: greater than or equal, less than or equal. The range includes both ends.
  • Literal[...]: only the values listed.
  • A default (= [], or default= inside Field): the key may be left out of the JSON. A field with no default must be there.
  • extra="forbid": a key the class does not know is an error. A teacher who types "weight" instead of "marks" hears about it, instead of the key being quietly ignored.

A class can hold a list of another class, slots: list[Slot], and Pydantic checks every item.

Give it a slot with three things wrong, and you get all three:

3 validation errors for Slot
room
  String should match pattern '^R[0-9]{3}$' [type=string_pattern_mismatch, input_value='204', input_type=str]
    For further information visit https://errors.pydantic.dev/2.14/v/string_pattern_mismatch
batch
  Input should be 'JEE-27A', 'JEE-27B' or 'NEET-27A' [type=literal_error, input_value='JEE-28', input_type=str]
    For further information visit https://errors.pydantic.dev/2.14/v/literal_error
teacher
  Extra inputs are not permitted [type=extra_forbidden, input_value='Rakesh', input_type=str]
    For further information visit https://errors.pydantic.dev/2.14/v/extra_forbidden

The rubric's two classes are built the same way: RubricPoint (id, text, marks, keywords, tip) and Question (id, subject, text, marks, and a list of RubricPoint). The goal lists every limit. One of them is easy to miss: keywords has a default, because a point may have none. Leave it without one, and the Check, which takes the keywords off a point, says:

a point without keywords raised an error: ValidationError: 1 validation error for Question

A rule about two fields

A field's limit sees one value. "The points add up to the question's marks" is about two fields together, so it cannot be a Field limit. It needs a model validator, a method that runs after every field is checked:

    @model_validator(mode="after")
    def ends_after_start(self) -> "Slot":
        if self.end <= self.start:
            raise ValueError(f"the slot ends at {self.end}, before it starts at {self.start}")
        return self

mode="after" means the method gets the finished object as self, with every field already the right type, so you can do arithmetic on them. To refuse, raise ValueError with a sentence; Pydantic turns it into a ValidationError. To accept, return self. If any field failed its own limit, the validator does not run at all.

The rubric's rule has two parts: point ids must be unique (chem-ph's two p1s), and the marks must add up (phy-lens's 6 of 5). The sentences are in the goal, word for word.

Where and what: errors()

A ValidationError is one exception with a list inside. e.errors() gives one dict per problem; two keys matter:

  • loc: where, as a tuple of keys and list positions.
  • msg: what, in a sentence.

For a day of the timetable, Day (a date and a list of slots), with the second slot ending at 22, and then a day with no slots:

('slots', 1, 'end') Input should be less than or equal to 21
('slots',) List should have at least 1 item after validation, not 0

('slots', 1, 'end') reads "the slots list, item 1 (counting from 0), its end". Joined with dots it is slots.1.end, which a teacher can follow in the file. And the model validator's error, on a slot from 11 to 9:

() Value error, the slot ends at 9, before it starts at 11

Two things to see. Its loc is empty, because the rule is about the whole object, not one field. And Pydantic put "Value error, " before your sentence. A teacher does not need that; the task takes it off. Leave it on, and the Check says:

a line still has Pydantic's "Value error, " in it: take it off the message

Every problem at once

Question.model_validate(row) checks one question and raises at the end of that question. A file has many. If load_questions lets the first ValidationError escape, the teacher fixes one question, saves, loads again, finds the next, and so on: one round for each broken question. So the loop catches the error for each question, turns each of its errors() into a line, and carries on with the next question. Only after the last one does it decide: any lines, raise RubricError(lines); none, return the questions as a dict by id.

Two problems no single question can see: a question with no id (the line then names it by its position, "question 4"), and two questions with the same id. Stop at the first broken question, and the Check shows how much is missing:

the problems are ["phy-bad: rubric.1.text: String should have at least 10 characters", "phy-bad: rubric.2.marks: Input should be less than or equal to 5"]; the task wants 5 lines starting ["phy-bad: rubric.1.text: ", "phy-bad: rubric.2.marks: ", "phy-six: the points add up to 6", "question 4: id: ",

With all three tasks done, "Check next week's draft rubrics" answers:

{"ok":false,"problems":["phy-lens: the points add up to 6 marks, but the question is worth 5","chem-ph: point ids must be unique: p1 is used more than once","phy-echo: rubric.1.text: String should have at least 10 characters"]}
[200]

Three problems in one answer: phy-lens and chem-ph from the start of this chapter, and phy-echo, whose second point has no text at all. That last one matters for the grader in chapter 3: a rubric point with no text gives the model nothing to compare the answer with.

Your turn

  • Task 1, the fields: every field of RubricPoint and Question with its limit, and keywords defaulting to []. Keep extra="forbid".
  • Task 2, points_add_up: unique ids, then marks that add up, each with the goal's sentence; return self.
  • Task 3, load_questions: every problem in every question as a line, "Value error, " taken off, repeated ids too; RubricError or the questions by id. The trap: stopping at the first broken question.

Karthik will ask: the grader would fail anyway on a broken rubric. Why check every rubric when it is loaded, and why list every problem instead of stopping at the first?

Do the tasks in order: Task 2's check needs Task 1's fields, and Task 3 validates with your classes. Start the app with python -m app, send "The week's questions" (as Aditi), "One question and its rubric", "A question that is not there" (404) and "Check next week's draft rubrics" (as Anjali), then press Check.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    models.py: the fields of a rubric

    In app/rubric/models.py give RubricPoint its fields: id (pattern ^p[1-9]$), text (at least 10 characters), marks (1 to 5), keywords (a list of strings, default []) and tip (at least 1 character). Give Question its fields: id (pattern ^[a-z]+-[a-z]+$), subject (one of "physics", "chemistry", "biology", "maths"), text (at least 10 characters), marks (1 to 10) and rubric (a list of 1 to 6 RubricPoint). Keep extra="forbid".

    The Check validates phy-ice from data/questions.json, then breaks it one field at a time (an id "P1", a 3-character text, marks 0 and 6, no tip, an extra "weight", a subject "history", no rubric points) and expects a ValidationError on that field each time.

  2. Task 2

    models.py: the points add up

    In Question write points_add_up, a model_validator(mode="after"): raise ValueError("point ids must be unique: p1 is used more than once") when ids repeat (every repeated id, sorted, joined with ", "), and ValueError("the points add up to 6 marks, but the question is worth 5") when the points' marks do not sum to the question's marks. Return self.

    The Check validates a question with two p1s, one whose points add up to 6 of 5, and the 12 questions of data/questions.json (all fine).

  3. Task 3 · interview question

    Interview: models.py, every problem at once

    In app/rubric/models.py write load_questions(path): validate every question in the JSON file and collect one line per error, for every question: "<id>: <loc>: <msg>" (loc joined with ".", like rubric.1.text) or "<id>: <msg>" when loc is empty, without Pydantic's "Value error, " prefix; <id> is the row's id or "question N" (from 1) when it has none; add "<id>: this id is used by two questions" for a repeated id. Raise RubricError(lines) if there are any, else return the questions as a dict by id.

    The Check loads a file of its own with five questions (one fine, one with two broken fields, one whose points add up to 6 of 5, one with no id, one with the first one's id), then GET /questions and GET /questions/check-draft as Anjali (3 problems in next week's draft).

    Karthik will ask: the grader would fail anyway on a broken rubric. Why check every rubric when it is loaded, and why list every problem instead of stopping at the first?

What you will learn

  • A Pydantic model for each part of a rubric: fields with types.
  • Field limits: pattern, min_length, ge and le, and limits on a list; Literal for a fixed set of values; a default.
  • extra="forbid": a key the class does not know is an error.
  • A model validator: a rule about two fields together.
  • ValidationError.errors(): where (loc) and what (msg), for every broken field.
  • Every problem at once, across a whole file.

More in A first grader

Build "Rubrics as data" yourself

Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.