AI answer grader project · Chapter 1 of 12
The problem and the plan
By the end of this chapter you know what Shikhar Classes' answer grader must do and how the next eleven chapters build it, you can read a rubric, and you have measured how far keyword matching gets on 200 real answers that two teachers marked, and where it goes wrong. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: A first grader
- 3 tasks
- Python
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
Shikhar Classes is a JEE and NEET coaching institute in Talwandi, Kota. Every Saturday each batch writes short "explain it" answers: why ice floats, why a satellite does not fall, why a catalyst speeds a reaction, what a mole is, explained as you would to a Class 9 student. Kavita, the founder (a former IIT chemistry teacher), started them because a student who can explain an idea in their own words keeps it.
The teachers mark them by hand: about 3,000 answers a week, five marks each. It takes most of Sunday, and the marks reach the students on Wednesday, when the topic is gone. Kavita wants marks on Saturday night, the doubtful answers in front of a teacher, and proof that the grader is as good as a second teacher before it marks anyone.
What does "as good as a second teacher" mean? Last month Anjali (chemistry) and Rakesh (physics) marked the same 200 answers separately. They gave the same mark to 150 of 200 (75%), and were never more than a mark apart on all but two. Those 200 answers, with both teachers' marks and the mark they agreed on afterwards, are the project's data/answers.json. Aditi, a student in JEE-27A, sends answers and reads her marks; she is the student you log in as. Karthik, Shikhar's staff AI engineer, builds the grader with you, one chapter at a time.
Before anyone brings a language model into this, Karthik wants a number to beat. The simplest grader there is looks for keywords. It is free, fast and gives the same marks every time. How close does it get to 150?
The lesson
The plan
The grader is a FastAPI app in Python. Over twelve chapters it grows like this:
1 the problem and a keyword baseline 7 the same answer graded many times
2 rubrics as checked data 8 answers that talk to the grader
3 asking a model for a grade 9 a review queue in Postgres
4 checking the model's grade in code 10 a release gate on a gold set
5 measuring it against the teachers 11 tests with a fake model
6 testing it for bias 12 cost, batches and feedback
Every chapter starts from Karthik's finished code of the chapter before, so you always work in a project that runs. You write the chapter's files; the rest is given.
Tour of the codebase
This is chapter 1, so everything is new. One file is yours.
pyproject.toml the packages and their versions
app/__main__.py python -m app: the server on port 8000, with reload
app/main.py create_app(): the routers and the error handler
app/config.py settings: the model, the data files, limits (later chapters use them)
app/lib/errors.py AppError: a status, a code and a sentence
app/auth/users.py the four people: Anjali, Rakesh (teachers), Aditi, Yash (students)
app/auth/session.py login with a cookie; current_user, require_teacher
app/ai.py the client for the model (chapter 3 uses it)
app/data.py Answer, load_answers, raw_questions
app/modules/health/router.py GET /health
app/modules/auth/router.py POST /auth/login, GET /auth/me
app/modules/baseline/router.py GET /baseline (teachers only)
app/baseline.py yours: keyword_marks, keyword_grade, biggest_misses
data/questions.json the 12 questions and their rubrics
data/answers.json the 200 answers and both teachers' marks
Every route except /health needs a login. The request pane logs you in as Aditi ("user") or Anjali ("admin"); a request with no login gets 401, and Aditi asking for the baseline gets 403:
{"error":{"code":"teachers_only","message":"Only a teacher can do this"}}
[403]
Start the app with python -m app. Before you write anything, "The keyword baseline" answers Internal Server Error [500], and the terminal shows why:
File "/tmp/w/01-the-problem-and-the-plan-starter/app/baseline.py", line 55, in keyword_grade
raise NotImplementedError("Task 2: keyword_grade is not written yet")
NotImplementedError: Task 2: keyword_grade is not written yet
What a rubric is
A rubric is the list of points a full answer makes, each worth some marks. The points of a question add up to its marks. Here is phy-ice, "Why does ice float on water?", from data/questions.json:
{
"id": "p1",
"text": "States that ice is less dense than liquid water.",
"marks": 2,
"keywords": ["density"],
"tip": "Compare the density of ice with the density of liquid water, not just their weight."
}
p2 (2 marks) explains why: hydrogen bonds hold the molecules in an open hexagonal structure. p3 (1 mark) uses buoyancy. 2 + 2 + 1 = 5. Every question has three points worth 2, 2 and 1.
A rubric is how a teacher marks fairly: not "this feels like a 3", but "it makes point 1, half of point 2, and not point 3". The grader will mark the same way. Each point also has keywords (for today's baseline) and a tip (chapter 12's feedback to the student).
In this chapter a question is a plain dict, read by raw_questions(). Chapter 2 turns it into checked objects.
Whole words, not pieces of words
The keyword grader asks, for each point: is any of its keywords in the answer? The given contains(text, phrase) answers that for whole words, ignoring case:
print(contains("Ice is less dense than water.", "dense"))
print(contains("Ice has a lower density.", "dense"))
print(contains("HYDROGEN BOND", "hydrogen bond"))
print(contains("hydrogen bonding", "hydrogen bond"))
True
False
True
False
Inside, it lowercases both and uses a regular expression with two lookarounds: (?<![a-z0-9]) before the phrase and (?![a-z0-9]) after it say "no letter or digit right here". So the match cannot start or end in the middle of a word.
Why not Python's in? Because "hydrogen bond" in "hydrogen bonds" is True: in finds pieces of words. A keyword grader built on in gives marks to "bonds" when the keyword is "bond", and the Check says:
keyword_marks for "Hydrogen bonds hold the molecules." gave 2; the task wants 0 (no keyword is in it as a whole word, so 0)
The Check is strict about whole words on purpose: you are measuring a method, so it must do exactly what it says.
A point's marks, then a whole grade
The keyword grader is all or nothing per point: a keyword is there, the point gets all its marks; none is, it gets 0. There is no half mark, because a keyword cannot tell you how well the student explained.
A whole answer's grade is every point's marks, in the rubric's order, and their total. any() answers "is at least one of these true?" in one line, a dict comprehension builds {id: marks}, and sum() over the dict's values gives the total. On four real answers (the finished code; the teachers' agreed mark on the right; A023 answers phy-satellite, the others phy-ice):
A005 {'points': {'p1': 0, 'p2': 0, 'p3': 0}, 'total': 0} teachers 4
A023 {'points': {'p1': 2, 'p2': 0, 'p3': 0}, 'total': 2} teachers 0
A014 {'points': {'p1': 2, 'p2': 2, 'p3': 0}, 'total': 4} teachers 4
A008 {'points': {'p1': 2, 'p2': 2, 'p3': 1}, 'total': 5} teachers 0
One right, three wrong, and wrong in both directions.
A baseline
A baseline is the score of the simplest method you can think of, measured on real data. It is the number every later idea must beat. If a language model, with all its cost, cannot beat keywords, nobody needs it.
GET /baseline (given) grades all 200 answers with your keyword_grade, compares each total with the teachers' agreed mark, and shows the five answers it gets most wrong. With your tasks done, "The keyword baseline" answers:
{
"answers": 200,
"same_as_teachers": 69,
"off_by_2_or_more": 82,
"agreement": 0.345,
"biggest_misses": [
...
69 of 200. The second teacher matched 150. And 82 answers are two or more marks off, out of five.
To find the biggest misses you need a gap for each answer: the keyword total minus the teachers' mark. +5 means keywords gave 5 where the teachers gave 0; -4 means keywords gave 4 too few. Both are big misses, so you sort by the size of the gap (abs()), and keep the sign so a reader knows which way it went. A sort key can be a tuple: Python compares the first items, and only on a tie the second. To sort test results by score, highest first, then by name, the key is (-score, name). A wrong key sorts the -4s to the bottom, and the Check shows you what it got:
biggest_misses(rows, 3) gave [{"answer_id": "A002", "keyword": 5, "teachers": 0, "gap": 5}, {"answer_id": "A007", "keyword": 5, "teachers": 1, "gap": 4}, {"answer_id": "A010", "keyword": 3, "teachers": 3, "gap": 0}]; the task wants A002 (gap 5), then A001 and A004 (gap -4, by id)
Why keyword matching is not enough
The real misses, further down the same answer:
{
"answer_id": "A008",
"keyword": 5,
"teachers": 0,
"gap": 5,
"text": "Density, hydrogen bonds, hexagonal lattice, buoyancy, Archimedes, lighter, expands."
},
{
"answer_id": "A069",
"keyword": 5,
"teachers": 0,
"gap": 5,
"text": "Rayleigh scattering, wavelength, blue, shorter, scattered, sunset, red."
},
...
{
"answer_id": "A098",
"keyword": 0,
"teachers": 5,
"gap": -5,
"text": "Lightning and thunder are produced at the same moment. Sound moves through air at only about 340 m/s, while light is almost a million times faster. Counting the seconds between the flash and the sound tells you how far away the storm is: about 3 seconds for each kilometre."
},
...
The other two are A092, a long thunder answer the teachers gave 5 (keywords 0), and A135, another list of words (keywords 5, teachers 0).
Three kinds of mistake, and you have seen each one:
- A list of words is not an explanation. A008 explains nothing and gets full marks. A student who learns this writes word lists forever.
- The right words in a wrong sentence. A023 says "there is no gravity in space" and gets 2 marks for the word "gravity". The teachers gave 0.
- A right answer in other words. A005 says ice is "less dense" and that "nine tenths of an iceberg stays under the water". No keyword matches, so 0. A098 explains thunder fully, and gets 0. Most misses are this kind: the keywords gave too little on 116 answers and too much on 15.
What the grader needs is to read the meaning of a sentence and compare it with each rubric point. That is what a language model can do. It also brings new problems (it can be fooled, it can be unfair, it can be sure and wrong), which is why most of this lab is about checking and measuring it.
Your turn
- Task 1, keyword_marks: the point's marks when any keyword is in the text as a whole word (use
contains), else 0. - Task 2, keyword_grade: every point's marks in the rubric's order, and their total. Then "The keyword baseline" says 69.
- Task 3, biggest_misses: a gap for each row, sorted by its size (biggest first, then by answer id), the first n. The trap: sorting by the gap itself, which puts every negative gap last.
Karthik will ask: the keyword grader is fast, free and always gives the same marks. Why not just add more keywords?
Each task's check stands alone (it uses its own copies of the other two functions), so you can pass them in any order. Start the app with python -m app, send "Who am I? (Aditi)", "The keyword baseline", and "The baseline as a student" (403), then press Check.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
baseline.py: a point's marks from its keywords
In app/baseline.py write keyword_marks(point, text): the point's "marks" when the text contains any of the point's "keywords" (use the given contains(text, keyword): whole words, case ignored), else 0.
The Check tries a point worth 2 marks with the keywords ["density", "hydrogen bond"] on six texts: "Ice has a lower DENSITY than water." and "Hydrogen bond holds the molecules." (2); "Hydrogen bonds hold the molecules." (0: bonds is not bond), "Ice is less dense than water." (0: dense is not density), "hydrogen bonding" and "" (0).
- Task 2
baseline.py: grade a whole answer
In app/baseline.py write keyword_grade(question, text): {"points": {...}, "total": n} where points maps every rubric point's id (in rubric order) to keyword_marks(point, text), and total is their sum.
The Check grades three answers to phy-ice (all its points, none, and the keyword list "Density, hydrogen bonds, hexagonal lattice, buoyancy, Archimedes."), then calls GET /baseline as Anjali.
- Task 3 · interview question
Interview: baseline.py, where keywords go wrong
In app/baseline.py write biggest_misses(rows, n=5): rows are (answer_id, keyword_total, agreed). Make {"answer_id", "keyword", "teachers", "gap"} for each, with gap = keyword - teachers, sort by abs(gap) biggest first and then by answer_id, and return the first n.
The Check sorts six rows of its own (ties, a negative gap, n=3), then calls GET /baseline: the keyword grader agrees with the teachers on 69 of 200 answers.
Karthik will ask: the keyword grader is fast, free and always gives the same marks. Why not just add more keywords?
What you will learn
- The plan: what the grader does, chapter by chapter.
- A rubric: the points a full answer makes, each with its marks.
- Whole-word matching, and why "dense" is not "density".
- A baseline: a simple method's score, measured against the teachers.
- Why keyword matching is not enough, from its real misses.
More in A first grader
Build The problem and the plan yourself
Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.