AI answer grader project (TypeScript) · Chapter 1 of 12

The problem and the plan

By the end of this chapter you know what Shikhar Classes' answer grader must do and how the next eleven chapters build it, you can find your way around the lab and the codebase, you can read a rubric, and you have measured how far keyword matching gets on 200 real answers that two teachers marked, and where it goes wrong. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: A first grader
  • 3 tasks
  • TypeScript
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

Shikhar Classes is a JEE and NEET coaching institute in Talwandi, Kota. Every Saturday each batch writes short "explain it" answers: why ice floats, why a satellite does not fall, why a catalyst speeds a reaction, what a mole is, explained as you would to a Class 9 student. Kavita, the founder (a former IIT chemistry teacher), started them because a student who can explain an idea in their own words keeps it.

The teachers mark them by hand: about 3,000 answers a week, five marks each. It takes most of Sunday, and the marks reach the students on Wednesday, when the class has moved on. Kavita wants marks on Saturday night, the doubtful answers in front of a teacher, and proof that the grader is as good as a second teacher before it marks anyone.

What does "as good as a second teacher" mean? Last month Anjali (chemistry) and Rakesh (physics) marked the same 200 answers separately. They gave the same mark to 150 of 200 (75%), and were more than a mark apart on only two. Those 200 answers, with both teachers' marks and the mark they agreed on afterwards, are the project's data/answers.json. Aditi, a student in JEE-27A, sends answers and reads her marks. Karthik, Shikhar's staff AI engineer, builds the grader with you, one chapter at a time.

Before anyone brings a language model into this, Karthik wants a number to beat. The simplest grader there is looks for keywords. It is free, fast and gives the same marks every time. How close does it get to 150?

The lesson

The plan

The grader is an Express API in TypeScript. Over twelve chapters it grows like this:

1  the problem and a keyword baseline     7  the same answer graded many times
2  rubrics as checked data                8  answers that talk to the grader
3  asking a model for a grade             9  a review queue in Postgres
4  checking the model's grade in code     10 a release gate on a gold set
5  measuring it against the teachers      11 tests with a fake model
6  testing it for bias                    12 cost, batches and feedback

Every chapter starts from Karthik's finished code of the chapter before, so you always work in a project that runs. You write the chapter's files; the rest is given.

The lab

The lab is a small Linux machine with Node 24, Postgres and a lab model, a stand-in for a language model that answers with the same API as OpenAI's (chapter 3 calls it). The terminal opens in the chapter's folder. Start the app there:

$ npm run dev

> shikhar-grader@1.0.0 dev
> tsx watch src/server.ts

Shikhar answer grader listening on http://localhost:3101

(This run used another port; yours says 3000.) tsx runs TypeScript directly, with no build step, and watch restarts the server every time you save a file. Keep this terminal in view: when a route fails, its error and stack trace land here.

The request pane sends saved requests to port 3000. Each request runs as a person: "user" is Aditi (a student), "admin" is Anjali (a teacher), and a request with no person sends no login. The pane logs in with POST /auth/login and keeps the cookie for you. This chapter's saved requests:

Who am I? (Aditi)
{"user":{"email":"aditi@shikhar.example","name":"Aditi Jain","role":"student","studentId":"S2004"}}
status 200

The baseline as a student
{"error":{"code":"teachers_only","message":"Only a teacher can do this"}}
status 403

The baseline without logging in
{"error":{"code":"not_logged_in","message":"Log in first: POST /auth/login"}}
status 401

Every route except /health needs a login, and the reports are for teachers only. One more command matters from chapter 3 on: lab requests 1 prints the last request the lab model received and what it answered. Nothing calls the model yet, so today it has nothing of yours to show.

Tour of the codebase

This is chapter 1, so everything is new. One file is yours.

package.json                          the packages; npm run dev
src/server.ts                         listens on port 3000
src/app.ts                            createApp(): the Express app and its routers
src/config.ts                         every setting: the model, the data files, limits
src/lib/errors.ts                     AppError: a status, a code and a sentence
src/middleware/errors.ts              every error leaves as { "error": { code, message } }
src/auth/users.ts                     the four people: Anjali, Rakesh (teachers), Aditi, Yash (students)
src/auth/session.ts                   login with a cookie; requireUser, requireTeacher
src/ai.ts                             the client for the model (chapter 3 uses it)
src/data.ts                           Answer, loadAnswers, answersById, rawQuestions
src/modules/health/health.routes.ts   GET /health
src/modules/auth/auth.routes.ts       POST /auth/login, GET /auth/me
src/modules/baseline/baseline.routes.ts  GET /baseline (teachers only)
src/baseline.ts                       yours: keywordMarks, keywordGrade, biggestMisses
data/questions.json                   the 12 questions and their rubrics
data/answers.json                     the 200 answers and both teachers' marks

Each route group has its own router under src/modules/. A route that needs a person runs requireUser first (the 401), and a teachers' route runs requireTeacher after it (the 403). The data files use snake_case keys (question_id, teacher_a); loadAnswers renames them once, so the rest of the app reads questionId and teacherA.

Before you write anything, "The keyword baseline" answers 500 {"error":{"code":"internal","message":"Something went wrong on our side"}}, and the terminal says why:

Error: Task 2: keywordGrade is not written yet
    at keywordGrade (/workspace/chapters/w-w1/src/baseline.ts:58:9)
    at <anonymous> (/workspace/chapters/w-w1/src/modules/baseline/baseline.routes.ts:12:70)
...

What a rubric is

A rubric is the list of points a full answer makes, each worth some marks. The points of a question add up to its marks. Here is the first point of phy-ice, "Why does ice float on water?", as rawQuestions() reads it:

{
  id: 'p1',
  text: 'States that ice is less dense than liquid water.',
  marks: 2,
  keywords: [ 'density' ],
  tip: 'Compare the density of ice with the density of liquid water, not just their weight.'
}

p2 (2 marks) explains why: hydrogen bonds hold the molecules in an open hexagonal structure. p3 (1 mark) uses buoyancy. 2 + 2 + 1 = 5. Every question has three points worth 2, 2 and 1.

A rubric is how a teacher marks fairly: not "this feels like a 3", but "it makes point 1, half of point 2, and not point 3". Each point also has keywords (for today's baseline) and a tip (chapter 12's feedback to the student). In this chapter a question is a plain object from the file; chapter 2 checks it.

Whole words, not pieces of words

The keyword grader asks, for each point: is any of its keywords in the answer? The given contains(text, phrase) answers that for whole words, ignoring case:

console.log(contains('Ice is less dense than water.', 'dense'));
console.log(contains('Ice has a lower density.', 'dense'));
console.log(contains('HYDROGEN BOND', 'hydrogen bond'));
console.log(contains('hydrogen bonding', 'hydrogen bond'));
true
false
true
false

Inside, it lowercases both and builds a regular expression with two lookarounds: (?<![a-z0-9]) before the phrase and (?![a-z0-9]) after it say "no letter or digit right here". So a match cannot start or end in the middle of a word.

Why not a string's includes? Because it finds pieces of words. A doubt-desk filter for questions about the mole:

const doubts = ['What is a mole in chemistry?', 'Why do water molecules stick together?'];
console.log(doubts.filter((d) => d.toLowerCase().includes('mole')));
console.log(doubts.filter((d) => contains(d, 'mole')));
[
  'What is a mole in chemistry?',
  'Why do water molecules stick together?'
]
[ 'What is a mole in chemistry?' ]

"mole" is inside "molecules". A keyword grader built on includes gives "Hydrogen bonds" the marks for "hydrogen bond", and the Check says:

keywordMarks for "Hydrogen bonds hold the molecules." gave 2; the task wants 0 (no keyword is in it as a whole word, so 0)

A point's marks, then a whole grade

The keyword grader is all or nothing per point: a keyword is there, the point gets all its marks; none is, it gets 0. There is no half mark, because a keyword cannot tell you how well the student explained.

A whole answer's grade is every point's marks, in the rubric's order, and their total. An array's some() answers "is at least one of these true?" in one line. A JavaScript object keeps its keys in the order you add them, so adding one key per point keeps the rubric's order. Object.values() gives the marks as an array, and reduce() adds them. On four real answers (the finished code; the teachers' agreed mark on the right; A023 answers phy-satellite, the others phy-ice):

A005 { points: { p1: 0, p2: 0, p3: 0 }, total: 0 } teachers 4
A023 { points: { p1: 2, p2: 0, p3: 0 }, total: 2 } teachers 0
A014 { points: { p1: 2, p2: 2, p3: 0 }, total: 4 } teachers 4
A008 { points: { p1: 2, p2: 2, p3: 1 }, total: 5 } teachers 0

One right, three wrong, and wrong in both directions.

A baseline

A baseline is the score of the simplest method you can think of, measured on real data. It is the number every later idea must beat. If a language model, with all its cost, cannot beat keywords, nobody needs it.

GET /baseline (given) grades all 200 answers with your keywordGrade, compares each total with the teachers' agreed mark, and shows the five answers it gets most wrong. With your tasks done, "The keyword baseline" answers:

{
  "answers": 200,
  "sameAsTeachers": 69,
  "offBy2OrMore": 82,
  "agreement": 0.345,
  "biggestMisses": [
...

69 of 200. The second teacher matched 150. And 82 answers are two or more marks off, out of five.

To find the biggest misses you need a gap for each answer: the keyword total minus the teachers' mark. +5 means keywords gave 5 where the teachers gave 0; -4 means keywords gave 4 too few. Both are big misses, so you sort by the size of the gap (Math.abs), and keep the sign so a reader knows which way it went.

sort() takes a compare function (a, b) => number: negative puts a first, positive puts b first, 0 says they tie. With no function at all it compares as text, so [10, 9, 1].sort() gives [ 1, 10, 9 ]. To break a tie, join two comparisons with ||: when the first gives 0, the second decides. Results sorted by score, highest first, then by name:

const scores: [string, number][] = [['Yash', 3], ['Aditi', 5], ['Isha', 5], ['Kabir', 1]];
console.log([...scores].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])));
[ [ 'Aditi', 5 ], [ 'Isha', 5 ], [ 'Yash', 3 ], [ 'Kabir', 1 ] ]

Two traps. sort() changes the array it is called on, which is why the example copies scores first. And a compare on the gap itself, not its size, sorts every negative gap to the bottom. The Check shows what it got:

biggestMisses(rows, 3) gave [{"answerId":"A002","keyword":5,"teachers":0,"gap":5},{"answerId":"A007","keyword":5,"teachers":1,"gap":4},{"answerId":"A010","keyword":3,"teachers":3,"gap":0}]; the task wants A002 (gap 5), then A001 and A004 (gap -4, by id), each { answerId, keyword, teachers, gap }

Why keyword matching is not enough

The real misses, further down the same answer:

    {
      "answerId": "A008",
      "keyword": 5,
      "teachers": 0,
      "gap": 5,
      "text": "Density, hydrogen bonds, hexagonal lattice, buoyancy, Archimedes, lighter, expands."
    },
    {
      "answerId": "A069",
      "keyword": 5,
      "teachers": 0,
      "gap": 5,
      "text": "Rayleigh scattering, wavelength, blue, shorter, scattered, sunset, red."
    },
...
    {
      "answerId": "A098",
      "keyword": 0,
      "teachers": 5,
      "gap": -5,
      "text": "Lightning and thunder are produced at the same moment. Sound moves through air at only about 340 m/s, while light is almost a million times faster. Counting the seconds between the flash and the sound tells you how far away the storm is: about 3 seconds for each kilometre."
    },
...

The other two are A092, a long thunder answer the teachers gave 5 (keywords 0), and A135, another list of words (keywords 5, teachers 0).

Three kinds of mistake, and you have seen each one:

  • A list of words is not an explanation. A008 explains nothing and gets full marks. A student who learns this writes word lists forever.
  • The right words in a wrong sentence. A023 says "there is no gravity in space, so nothing pulls the satellite down" and gets 2 marks for the word "gravity". The teachers gave 0.
  • A right answer in other words. A005 says ice is "less dense" and that "nine tenths of an iceberg stays under the water". No keyword matches, so 0. A098 explains thunder fully, and gets 0. Most misses are this kind: the keywords gave too little on 116 answers and too much on 15.

What the grader needs is to read the meaning of a sentence and compare it with each rubric point. That is what a language model can do. It also brings new problems (it can be fooled, it can be unfair, it can be sure and wrong), which is why most of this project is about checking and measuring it.

Your turn

  • Task 1, keywordMarks: the point's marks when any keyword is in the text as a whole word (use contains), else 0.
  • Task 2, keywordGrade: every point's marks in the rubric's order, and their total. Then "The keyword baseline" says 69.
  • Task 3, biggestMisses: a gap for each row, sorted by its size (biggest first, then by answer id), the first n. The traps: sorting by the gap itself, which puts every negative gap last, and sorting the rows you were given instead of a new array.

Karthik will ask: the keyword grader is fast, free and always gives the same marks. Why not just add more keywords?

Start the app with npm run dev, send "Who am I? (Aditi)", "The keyword baseline" (as Anjali), "The baseline as a student" (403) and "The baseline without logging in" (401), then press Check. Task 2's check needs Task 1, and Task 3's check calls the route, which needs Task 2: do them in order.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    baseline.ts: a point's marks from its keywords

    In src/baseline.ts write keywordMarks(point, text): the point's marks when the text contains any of the point's keywords (use the given contains(text, keyword): whole words, case ignored), else 0.

    The Check tries a point worth 2 marks with the keywords ["density", "hydrogen bond"] on six texts: "Ice has a lower DENSITY than water." and "Hydrogen bond holds the molecules." (2); "Hydrogen bonds hold the molecules." (0: bonds is not bond), "Ice is less dense than water." (0: dense is not density), "hydrogen bonding" and "" (0).

  2. Task 2

    baseline.ts: grade a whole answer

    In src/baseline.ts write keywordGrade(question, text): { points: {...}, total: n } where points maps every rubric point's id (in rubric order) to keywordMarks(point, text), and total is their sum.

    The Check grades three answers to phy-ice (all its points, none, and the keyword list "Density, hydrogen bonds, hexagonal lattice, buoyancy, Archimedes."), then calls GET /baseline as Anjali.

  3. Task 3 · interview question

    Interview: baseline.ts, where keywords go wrong

    In src/baseline.ts write biggestMisses(rows, n = 5): rows are [answerId, keywordTotal, agreed]. Make { answerId, keyword, teachers, gap } for each, with gap = keyword - teachers, sort by Math.abs(gap) biggest first and then by answerId, and return the first n.

    The Check sorts six rows of its own (ties, a negative gap, n = 3), then calls GET /baseline: its biggest miss is A008, a list of keywords.

    Karthik will ask: the keyword grader is fast, free and always gives the same marks. Why not just add more keywords?

What you will learn

  • The plan: what the grader does, chapter by chapter.
  • The lab: npm run dev, the request pane and lab requests.
  • A rubric: the points a full answer makes, each with its marks.
  • Whole-word matching, and why "dense" is not "density".
  • A baseline: a simple method's score, measured against the teachers.
  • Why keyword matching is not enough, from its real misses.

More in A first grader

Build The problem and the plan yourself

Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.