AI answer grader project · Chapter 2 of 12
Rubrics as data
By the end of this chapter you can describe a rubric as Pydantic classes, with a limit on every field and a rule that the points add up, and load a whole file of questions so that every problem in it is listed in one go, before anyone grades with it. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: A first grader
- 3 tasks
- Python
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
The grader will mark 3,000 answers a week against the rubrics in a JSON file, and the teachers write those rubrics. Next week's draft, data/rubrics-draft.json, has four new questions from the teachers. In chapter 1 a question was a plain dict, read by raw_questions(), which reads anything. Here is chapter 1's keyword grader on the draft, with a full answer to phy-lens and any answer to chem-ph:
{'points': {'p1': 2, 'p2': 2, 'p3': 2}, 'total': 6}
{'points': {'p1': 0, 'p3': 0}, 'total': 0}
6 marks on a 5-mark question: phy-lens's points are worth 2, 2 and
- And chem-ph has two points called p1, so the dict kept only the second one, and a point worth 2 marks vanished without a word. No error, no warning. Aditi would see "6 of 5" on Saturday night, or lose two marks she earned, and nobody would know why until a student complained.
A rubric is data the grader trusts. So it must be checked the moment it is read, and a teacher fixing it must see every problem at once.
The lesson
Tour of the codebase
New or changed since chapter 1:
app/rubric/models.py yours: RubricPoint, Question (Tasks 1, 2),
load_questions (Task 3); RubricError and
Question.point given
app/modules/questions/router.py given: GET /questions, GET /questions/{id},
GET /questions/check-draft (teachers)
app/main.py changed: includes the questions router
app/baseline.py given now: chapter 1's keyword grader
data/rubrics-draft.json next week's four draft questions
RubricError holds a list of lines, problems, one per problem. Question.point("p2") finds a point by its id (chapter 4 uses it). GET /questions lists the 12 questions, GET /questions/{id} shows one with its rubric (404 for an id that is not there), and GET /questions/check-draft loads the draft and answers {"ok": false, "problems": [...]} or {"ok": true, ...}.
All three call your load_questions, so before Task 3, "The week's questions" answers 500 and the terminal ends with:
File "/tmp/w/02-rubrics-as-data-starter/app/rubric/models.py", line 92, in load_questions
raise NotImplementedError("Task 3: load_questions is not written yet")
NotImplementedError: Task 3: load_questions is not written yet
A class for each part
Pydantic turns a class with type hints into a checker. You give it a dict (from JSON), and it either gives back an object whose fields are all the right type, or raises a ValidationError that says what is wrong. Here is a made-up example, not a rubric: a slot in Shikhar's timetable.
class Slot(BaseModel):
model_config = ConfigDict(extra="forbid")
room: str = Field(pattern=r"^R[0-9]{3}$")
batch: Literal["JEE-27A", "JEE-27B", "NEET-27A"]
start: int = Field(ge=8, le=20)
end: int = Field(ge=9, le=21)
topics: list[str] = Field(default=[], max_length=3)
Read each line as "the type says what kind of value, Field says which values are allowed":
pattern: the string must match this regular expression.^and$mean the whole string, so "R204" passes and "204" does not.min_lengthandmax_length: on a string they count characters; on a list they count items. A list withmin_length=1cannot be empty.geandle: greater than or equal, less than or equal. The range includes both ends.Literal[...]: only the values listed.- A default (
= [], ordefault=insideField): the key may be left out of the JSON. A field with no default must be there. extra="forbid": a key the class does not know is an error. A teacher who types"weight"instead of"marks"hears about it, instead of the key being quietly ignored.
A class can hold a list of another class, slots: list[Slot], and Pydantic checks every item.
Give it a slot with three things wrong, and you get all three:
3 validation errors for Slot
room
String should match pattern '^R[0-9]{3}$' [type=string_pattern_mismatch, input_value='204', input_type=str]
For further information visit https://errors.pydantic.dev/2.14/v/string_pattern_mismatch
batch
Input should be 'JEE-27A', 'JEE-27B' or 'NEET-27A' [type=literal_error, input_value='JEE-28', input_type=str]
For further information visit https://errors.pydantic.dev/2.14/v/literal_error
teacher
Extra inputs are not permitted [type=extra_forbidden, input_value='Rakesh', input_type=str]
For further information visit https://errors.pydantic.dev/2.14/v/extra_forbidden
The rubric's two classes are built the same way: RubricPoint (id, text, marks, keywords, tip) and Question (id, subject, text, marks, and a list of RubricPoint). The goal lists every limit. One of them is easy to miss: keywords has a default, because a point may have none. Leave it without one, and the Check, which takes the keywords off a point, says:
a point without keywords raised an error: ValidationError: 1 validation error for Question
A rule about two fields
A field's limit sees one value. "The points add up to the question's marks" is about two fields together, so it cannot be a Field limit. It needs a model validator, a method that runs after every field is checked:
@model_validator(mode="after")
def ends_after_start(self) -> "Slot":
if self.end <= self.start:
raise ValueError(f"the slot ends at {self.end}, before it starts at {self.start}")
return self
mode="after" means the method gets the finished object as self, with every field already the right type, so you can do arithmetic on them. To refuse, raise ValueError with a sentence; Pydantic turns it into a ValidationError. To accept, return self. If any field failed its own limit, the validator does not run at all.
The rubric's rule has two parts: point ids must be unique (chem-ph's two p1s), and the marks must add up (phy-lens's 6 of 5). The sentences are in the goal, word for word.
Where and what: errors()
A ValidationError is one exception with a list inside. e.errors() gives one dict per problem; two keys matter:
loc: where, as a tuple of keys and list positions.msg: what, in a sentence.
For a day of the timetable, Day (a date and a list of slots), with the second slot ending at 22, and then a day with no slots:
('slots', 1, 'end') Input should be less than or equal to 21
('slots',) List should have at least 1 item after validation, not 0
('slots', 1, 'end') reads "the slots list, item 1 (counting from 0), its end". Joined with dots it is slots.1.end, which a teacher can follow in the file. And the model validator's error, on a slot from 11 to 9:
() Value error, the slot ends at 9, before it starts at 11
Two things to see. Its loc is empty, because the rule is about the whole object, not one field. And Pydantic put "Value error, " before your sentence. A teacher does not need that; the task takes it off. Leave it on, and the Check says:
a line still has Pydantic's "Value error, " in it: take it off the message
Every problem at once
Question.model_validate(row) checks one question and raises at the end of that question. A file has many. If load_questions lets the first ValidationError escape, the teacher fixes one question, saves, loads again, finds the next, and so on: one round for each broken question. So the loop catches the error for each question, turns each of its errors() into a line, and carries on with the next question. Only after the last one does it decide: any lines, raise RubricError(lines); none, return the questions as a dict by id.
Two problems no single question can see: a question with no id (the line then names it by its position, "question 4"), and two questions with the same id. Stop at the first broken question, and the Check shows how much is missing:
the problems are ["phy-bad: rubric.1.text: String should have at least 10 characters", "phy-bad: rubric.2.marks: Input should be less than or equal to 5"]; the task wants 5 lines starting ["phy-bad: rubric.1.text: ", "phy-bad: rubric.2.marks: ", "phy-six: the points add up to 6", "question 4: id: ",
With all three tasks done, "Check next week's draft rubrics" answers:
{"ok":false,"problems":["phy-lens: the points add up to 6 marks, but the question is worth 5","chem-ph: point ids must be unique: p1 is used more than once","phy-echo: rubric.1.text: String should have at least 10 characters"]}
[200]
Three problems in one answer: phy-lens and chem-ph from the start of this chapter, and phy-echo, whose second point has no text at all. That last one matters for the grader in chapter 3: a rubric point with no text gives the model nothing to compare the answer with.
Your turn
- Task 1, the fields: every field of
RubricPointandQuestionwith its limit, andkeywordsdefaulting to[]. Keepextra="forbid". - Task 2, points_add_up: unique ids, then marks that add up, each with the goal's sentence; return
self. - Task 3, load_questions: every problem in every question as a line, "Value error, " taken off, repeated ids too;
RubricErroror the questions by id. The trap: stopping at the first broken question.
Karthik will ask: the grader would fail anyway on a broken rubric. Why check every rubric when it is loaded, and why list every problem instead of stopping at the first?
Do the tasks in order: Task 2's check needs Task 1's fields, and Task 3 validates with your classes. Start the app with python -m app, send "The week's questions" (as Aditi), "One question and its rubric", "A question that is not there" (404) and "Check next week's draft rubrics" (as Anjali), then press Check.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
models.py: the fields of a rubric
In app/rubric/models.py give RubricPoint its fields: id (pattern ^p[1-9]$), text (at least 10 characters), marks (1 to 5), keywords (a list of strings, default []) and tip (at least 1 character). Give Question its fields: id (pattern ^[a-z]+-[a-z]+$), subject (one of "physics", "chemistry", "biology", "maths"), text (at least 10 characters), marks (1 to 10) and rubric (a list of 1 to 6 RubricPoint). Keep extra="forbid".
The Check validates phy-ice from data/questions.json, then breaks it one field at a time (an id "P1", a 3-character text, marks 0 and 6, no tip, an extra "weight", a subject "history", no rubric points) and expects a ValidationError on that field each time.
- Task 2
models.py: the points add up
In Question write points_add_up, a model_validator(mode="after"): raise ValueError("point ids must be unique: p1 is used more than once") when ids repeat (every repeated id, sorted, joined with ", "), and ValueError("the points add up to 6 marks, but the question is worth 5") when the points' marks do not sum to the question's marks. Return self.
The Check validates a question with two p1s, one whose points add up to 6 of 5, and the 12 questions of data/questions.json (all fine).
- Task 3 · interview question
Interview: models.py, every problem at once
In app/rubric/models.py write load_questions(path): validate every question in the JSON file and collect one line per error, for every question: "<id>: <loc>: <msg>" (loc joined with ".", like rubric.1.text) or "<id>: <msg>" when loc is empty, without Pydantic's "Value error, " prefix; <id> is the row's id or "question N" (from 1) when it has none; add "<id>: this id is used by two questions" for a repeated id. Raise RubricError(lines) if there are any, else return the questions as a dict by id.
The Check loads a file of its own with five questions (one fine, one with two broken fields, one whose points add up to 6 of 5, one with no id, one with the first one's id), then GET /questions and GET /questions/check-draft as Anjali (3 problems in next week's draft).
Karthik will ask: the grader would fail anyway on a broken rubric. Why check every rubric when it is loaded, and why list every problem instead of stopping at the first?
What you will learn
- A Pydantic model for each part of a rubric: fields with types.
- Field limits: pattern, min_length, ge and le, and limits on a list; Literal for a fixed set of values; a default.
- extra="forbid": a key the class does not know is an error.
- A model validator: a rule about two fields together.
- ValidationError.errors(): where (loc) and what (msg), for every broken field.
- Every problem at once, across a whole file.
More in A first grader
Build "Rubrics as data" yourself
Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.