AI answer grader project · Chapter 9 of 12
The review queue
By the end of this chapter you can take a student's answer in, grade it, and store the answer and its grade in Postgres in one transaction; list the grades that wait for a teacher, oldest first; and keep a teacher's marks as a new row beside the grader's, which are never changed. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Teachers and releases
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of AI answer grader project are free. This one comes with the all labs plan.
Why you need this
Kavita asked for three things: marks on Saturday night, the doubtful ones in front of a teacher, and proof the grader is as good as a second teacher. The grader can now mark, check and flag an answer. Then it forgets it. POST /grade answers and keeps nothing: no record that Aditi sent an answer, no list for Anjali, nowhere to write the right marks.
How much would land on the teachers? Prompt 3 on all 200 answers:
need a teacher: 30 of 200
{'evidence_not_found': 24, 'total_mismatch': 15, 'instructions_in_answer': 6}
(A wrong total alone does not need a teacher: the code adds the marks up itself.) 30 of 200 is 15%. Of 3,000 answers a week that is about 450 for the teachers instead of all 3,000, and the other 2,550 reach the students on Saturday night. But only if the 450 are kept somewhere, in order, until a teacher gets to them, and only if what the teacher decides is kept too. That needs a database.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
service.py: an answer in, a grade stored
In app/review/service.py write submit(db, ai, user, question, text): grade_answer first; then in one transaction add a Submission (question_id, student_id = user.student_id or "", author_email, text), flush, and a GradeRow for it (prompt_version, model, score, out_of, points, evidence, problems as a list of dicts, status "review" when the grade needs review else "final", input_tokens, output_tokens), commit once, and return answer_view(sub, row, None, user).
The Check sends two answers as Aditi with POST /answers (a clean one: 201 and final; A006's text: 201 and review, with no score shown to her) and reads the rows in Postgres.
- Task 2
service.py: the teachers' queue
In app/review/service.py write review_queue(db): every GradeRow with status "review", oldest first (by its id), joined to its Submission, as {"grade_id", "submission_id", "question_id", "student_id", "text", "score", "points", "problems"}.
The Check sends three answers (two that need review, one clean), then GET /review as Anjali: the two, oldest first. As Aditi it is 403.
- Task 3 · interview question
Interview: service.py, the teacher's marks
In app/review/service.py write override(db, teacher, grade_id, points, note, questions): 404 for no such grade; the points must name exactly the rubric's ids, each from 0 to its marks, else AppError(422, "bad_points", ...); add an Override (grade_id, teacher_email, points in rubric order, score their sum, note, agrees = points equal the grade's), set the grade's status to "final" without changing its marks, commit, return answer_view(sub, row, override, teacher).
The Check overrides a review grade as Anjali (and tries a missing point, 3 marks on a 2-mark point and grade 999), then reads the rows: the grader's marks are as it gave them, the teacher's are beside them, and Aditi now sees the teacher's score.
Karthik will ask: why keep the grader's marks once a teacher has set the right ones? Why not just update the grade?
What you will learn
- The tables of migration 0001: submissions, grades, overrides.
- Grade first, then the transaction: why the model is called before anything touches the database.
- flush() and one commit(): an id inside the transaction, and all rows saved together or none.
- The review queue: a join, a filter and an order, served oldest first.
- An override is a new row: why the teacher's marks go beside the grader's and never over them.
More in Teachers and releases
Build The review queue yourself
Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI answer grader project are free. This one comes with the all labs plan.