AI answer grader project (TypeScript) · Chapter 2 of 12
Rubrics as data
By the end of this chapter you can describe a rubric as zod schemas, with a limit on every field and a rule that the points add up, and load a whole file of questions so that every problem in it is listed in one go, before anyone grades with it. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: A first grader
- 3 tasks
- TypeScript
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
The grader will mark 3,000 answers a week against the rubrics in a JSON file, and the teachers write those rubrics. Next week's draft, data/rubrics-draft.json, has four new questions from them. In chapter 1 a question was whatever JSON.parse gave back, and TypeScript's RawQuestion type believed it: types vanish when the code runs, so nothing checked the file. Here is chapter 1's keyword grader on the draft, with a full answer to phy-lens and "Lemon juice has citric acid." for chem-ph:
{ points: { p1: 2, p2: 2, p3: 2 }, total: 6 }
{ points: { p1: 0, p3: 0 }, total: 0 }
6 marks on a 5-mark question: phy-lens's points are worth 2, 2 and
- And chem-ph has two points called p1, so the object kept only the second one, and the citric acid point (2 marks) vanished without a word. No error, no warning. Aditi would see "6 of 5" on Saturday night, or lose two marks she earned, and nobody would know why until a student complained.
A rubric is data the grader trusts. So it must be checked the moment it is read, and a teacher fixing it must see every problem at once.
The lesson
Tour of the codebase
New or changed since chapter 1:
src/rubric/models.ts yours: RubricPoint, Question (Task 1), pointsAddUp (Task 2),
loadQuestions (Task 3); pointOf and RubricError given
src/modules/questions/questions.routes.ts given: GET /questions, GET /questions/:id,
GET /questions/check-draft (teachers)
src/app.ts changed: uses the questions router
src/baseline.ts given now: chapter 1's keyword grader
data/rubrics-draft.json next week's four draft questions
RubricError is an Error that holds problems, a list of lines, one per problem. pointOf(question, 'p2') finds a point by its id (chapter 4 uses it). GET /questions lists the 12 questions, GET /questions/:id shows one with its rubric (404 for an id that is not there), and GET /questions/check-draft loads the draft and answers {"ok": false, "problems": [...]} or {"ok": true, ...}.
All three call your loadQuestions, so before Task 3, "The week's questions" answers 500 and the terminal running npm run dev says:
Error: Task 3: loadQuestions is not written yet
at loadQuestions (/workspace/chapters/w-w1/src/rubric/models.ts:99:9)
at <anonymous> (/workspace/chapters/w-w1/src/modules/questions/questions.routes.ts:11:35)
...
A zod schema
zod turns a description of a value into a checker that runs. You give it a value (from JSON), and it either gives back the value with every field the right type, or tells you what is wrong. Here is a made-up example, not a rubric: a slot in Shikhar's timetable.
const Slot = z
.object({
room: z.string().regex(/^R[0-9]{3}$/),
batch: z.enum(['JEE-27A', 'JEE-27B', 'NEET-27A']),
start: z.number().int().min(8).max(20),
end: z.number().int().min(9).max(21),
topics: z.array(z.string()).max(3).default([]),
})
.strict();
Read each line as "the first call says what kind of value, each method after it adds a rule":
z.object({...})lists the keys;z.string()is any text,z.number()any number..regex(): the text must match the pattern.^and$mean the whole text, so "R204" passes and "204" does not..min()and.max(): on a string they count characters, on a number they bound the value (both ends included), on an array they count items. An array with.min(1)cannot be empty..int(): a whole number. JSON has one number type, so without it 1.5 marks gets through.z.enum([...]): only the values listed.z.array(Other): every item is checked withOther's schema, so one schema holds another..default([]): the key may be left out; the value is then[]. Every other key must be there..strict(): a key the schema does not know is an error.
Why .strict()? Without it, zod quietly drops keys it does not know. A teacher who types "weight" instead of "marks":
> z.object({ marks: z.number() }).safeParse({ marks: 2, weight: 2 })
{ success: true, data: { marks: 2 } }
safeParse and its issues
Slot.safeParse(value) never throws. It returns { success: true, data } or { success: false, error }. A good slot, with topics filled in by the default:
{
success: true,
data: { room: 'R204', batch: 'JEE-27A', start: 9, end: 11, topics: [] }
}
A slot with three things wrong gives all three in error.issues:
[
{
origin: 'string',
code: 'invalid_format',
format: 'regex',
pattern: '/^R[0-9]{3}$/',
path: [ 'room' ],
message: 'Invalid string: must match pattern /^R[0-9]{3}$/'
},
{
code: 'invalid_value',
values: [ 'JEE-27A', 'JEE-27B', 'NEET-27A' ],
path: [ 'batch' ],
message: 'Invalid option: expected one of "JEE-27A"|"JEE-27B"|"NEET-27A"'
},
{
code: 'unrecognized_keys',
keys: [ 'teacher' ],
path: [],
message: 'Unrecognized key: "teacher"'
}
]
Two keys of an issue matter: path (where) and message (what). Inside an array the path grows. For a made-up Day (a date and slots: z.array(Slot).min(1)), with the second slot ending at 22, and then a day with no slots:
for (const i of Day.safeParse(day).error!.issues) console.log(i.path, i.message);
[ 'slots', 1, 'end' ] Too big: expected number to be <=21
[ 'slots' ] Too small: expected array to have >=1 items
[ 'slots', 1, 'end' ] reads "the slots array, item 1 (counting from 0), its end". path.join('.') gives slots.1.end, which a teacher can follow in the file. And the extra key's path is empty: the problem is the whole object, not one field.
There is also Slot.parse(value), which throws a ZodError instead. Its message is the whole issues array as one JSON string, written for a developer, not for a teacher.
The rubric's two schemas are built the same way: RubricPoint (id, text, marks, keywords, tip) and Question (id, subject, text, marks, and an array of RubricPoint). The goal lists every limit. One is easy to miss: keywords has a default, because a point may have none. Leave it out, and the Check, which takes the keywords off a point, says:
a point without keywords was refused (Invalid input: expected array, received undefined); keywords defaults to []
A rule about two fields
A field's rule sees one value. "The points add up to the question's marks" is about two fields together, so it cannot be a field's rule. It needs .superRefine(fn): zod calls fn(value, ctx) with the parsed object, and you report what is wrong with ctx.addIssue():
function endsAfterStart(s: { start: number; end: number }, ctx: z.RefinementCtx) {
if (s.end <= s.start) {
ctx.addIssue({ code: 'custom', message: `the slot ends at ${s.end}, before it starts at ${s.start}` });
}
}
const Slot2 = Slot.superRefine(endsAfterStart);
A slot from 11 to 9:
[
{
code: 'custom',
message: 'the slot ends at 9, before it starts at 11',
path: []
}
]
code: 'custom'marks it as your own rule; the message is your sentence, word for word. Its path is empty: it is about the object.addIssuedoes not throw. It records the issue and the checking carries on, so one object can break two rules and get two issues. Return nothing.- When it runs: once every field has the right type. A room "R2" (the right type, the wrong pattern) still gets the rule's issue beside its own; a start of
'11'(text, not a number) gets onlyInvalid input: expected number, received string.
Throw an Error inside instead, and zod lets it escape, so safeParse throws after all. The Check says:
two points with the id p1: Question.safeParse threw point ids must be unique: p1 is used more than once; zod reports problems as issues, it does not throw
The rubric's rule has two parts: point ids must be unique (chem-ph's two p1s), and the marks must add up (phy-lens's 6 of 5). The sentences are in the goal, word for word. To find the repeated ids, remember that indexOf gives the first place a value appears: an id seen at any later place is a repeat. A Set keeps each value once, and sort() makes the message the same every time. Skip those two, and a rubric that repeats p3 and then p2 gives:
a rubric that repeats p3 and then p2 gave "point ids must be unique: p3, p2 is used more than once | the points add up to 8 marks, but the question is worth 10"; the task wants "point ids must be unique: p2, p3 is used more than once" (each repeated id once, sorted)
Every problem at once
Question.safeParse(row) checks one question. A file has many. If loadQuestions gives up at the first broken question, the teacher fixes one, saves, loads again, finds the next, and so on: one round for each broken question. So the loop checks every row, turns each issue of each failed row into a line, and carries on with the next row. Only after the last one does it decide: any lines, throw RubricError(lines); none, return the questions as an object by id.
Two problems no single question can see: a question with no id (the line then names it by its position, "question 4"), and two questions with the same id. Stop at the first broken question, and the Check shows how much is missing:
the problems are ["phy-bad: rubric.0.id: Invalid string: must match pattern /^p[1-9]$/","phy-bad: rubric.1.text: Too small: expected string to have >=10 characters"]; the task wants 5 lines starting ["phy-bad: rubric.0.id: ","phy-bad: rubric.1.text: ","phy-six: the points add up to 6","question 4: id: ","phy-ice: this id is used by two questions"]
The other trap is error.message: one line holding the whole JSON array, "phy-bad: [\n {\n \"origin\": \"string\", .... A teacher needs one line per issue, each with its path joined by dots.
With all three tasks done, "Check next week's draft rubrics" answers:
{"ok":false,"problems":["phy-lens: the points add up to 6 marks, but the question is worth 5","chem-ph: point ids must be unique: p1 is used more than once","phy-echo: rubric.1.text: Too small: expected string to have >=10 characters"]}
Three problems in one answer: phy-lens and chem-ph from the start of this chapter, and phy-echo, whose second point has no text at all. That last one matters in chapter 3: a rubric point with no text gives the model nothing to compare the answer with.
Your turn
- Task 1, the fields: every field of
RubricPointandQuestionwith its limit,.int()on the marks, andkeywordsdefaulting to[]. Keep.strict(). - Task 2, pointsAddUp: unique ids, then marks that add up, each an issue with the goal's sentence. The traps: throwing instead of
ctx.addIssue, and repeated ids that are neither unique nor sorted. - Task 3, loadQuestions: every issue of every question as a line, "question N" for a row with no id, repeated ids too;
RubricErroror the questions by id. The traps: stopping at the first broken question, anderror.message.
Karthik will ask: the grader would fail anyway on a broken rubric. Why check every rubric when it is loaded, and why list every problem instead of stopping at the first?
Do the tasks in order: Task 2's check needs Task 1's fields, and Task 3 uses both. Start the app with npm run dev, send "The week's questions" and "One question and its rubric" (as Aditi), "A question that is not there" (404) and "Check next week's draft rubrics" (as Anjali), then press Check.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
models.ts: the fields of a rubric
In src/rubric/models.ts give RubricPoint its fields with zod: id (a string matching /^p[1-9]$/), text (at least 10 characters), marks (a whole number from 1 to 5), keywords (an array of strings, default []) and tip (at least 1 character). Give Question its fields: id (matching /^[a-z]+-[a-z]+$/), subject (one of "physics", "chemistry", "biology", "maths"), text (at least 10 characters), marks (a whole number from 1 to 10) and rubric (an array of 1 to 6 RubricPoint). Keep .strict() on both.
The Check reads phy-ice from data/questions.json, then breaks it one field at a time (an id "P1", a 3-character text, marks 0, 6 and 1.5, no tip, an extra "weight", a subject "history", no rubric points, seven points) and expects an issue at that field each time.
- Task 2
models.ts: the points add up
In src/rubric/models.ts write pointsAddUp(q, ctx), which Question runs with .superRefine: add an issue with ctx.addIssue({ code: "custom", message }) saying "point ids must be unique: p1 is used more than once" when ids repeat (every repeated id once, sorted, joined with ", "), and one saying "the points add up to 6 marks, but the question is worth 5" when the points' marks do not sum to the question's marks.
The Check validates a question with two p1s, one that repeats p3 and p2, one whose points add up to 6 of 5, and the 12 questions of data/questions.json (all fine).
- Task 3 · interview question
Interview: models.ts, every problem at once
In src/rubric/models.ts write loadQuestions(path): Question.safeParse every question in the JSON file and collect one line per issue, for every question: "<id>: <path>: <message>" (the issue's path joined with ".", like rubric.1.text) or "<id>: <message>" when the path is empty; <id> is the row's id or "question N" (from 1) when it has none; add "<id>: this id is used by two questions" for a repeated id. Throw new RubricError(lines) if there are any, else return the questions as an object by id.
The Check loads a file of its own with five questions (one fine, one with two broken fields, one whose points add up to 6 of 5, one with no id, one with the first one's id), then GET /questions and GET /questions/check-draft as Anjali (3 problems in next week's draft).
Karthik will ask: the grader would fail anyway on a broken rubric. Why check every rubric when it is loaded, and why list every problem instead of stopping at the first?
What you will learn
- A zod schema for each part of a rubric: fields with types.
- Limits: .regex(), .min() and .max() on text, numbers and arrays, .int(), z.enum() for a fixed set of values, and .default().
- .strict(): a key the schema does not know is an error.
- .superRefine(): a rule about two fields together, reported with ctx.addIssue().
- safeParse and its issues: where (path) and what (message).
- Every problem at once, across a whole file.
More in A first grader
Build "Rubrics as data" yourself
Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.