AI answer grader project · Chapter 12 of 12
Cost and feedback
By the end of this chapter you can say in rupees what grading one answer and a week of answers costs, grade a whole class's answers a few at a time within the provider's limit, and give each student feedback that says, point by point, what they got and what to do next. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Ready for the institute
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of AI answer grader project are free. This one comes with the all labs plan.
Why you need this
The grader is measured, tested and gated. Before Kavita switches it on for every batch, three questions are left.
What does it cost? About 3,000 answers a week. Kavita has a budget, and "a few tokens an answer" is not a number she can plan with. She wants rupees: per answer, and per week.
How long does Saturday night take? POST /answers grades one answer while the student waits. Anjali wants to grade a whole class's answers to one question at once. One after another, each one waits for the model's reply before the next is sent: 17 answers on lab-chat-limited take 8.73 seconds. A real model can take a few seconds a reply, and 3,000 answers one at a time would run for hours.
What does Aditi read? Today a final grade gives her a number: "score": 3, "out_of": 5. She cannot tell which point she missed or what to write next time, which was the reason Kavita started these answers in the first place. The grade has the model's reasons, but they were written for the code and for a teacher, not for a student.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
usage.py: what grading costs
In app/cost/usage.py write cost_inr(usage): (input x settings.price_input + output x settings.price_output) / 1,000,000 as a Decimal rounded to 4 places. Then week_estimate(usages, answers_per_week): answers_graded, avg_input_tokens and avg_output_tokens (rounded to whole tokens), cost_per_answer_inr (the cost of the average, as a string), answers_per_week and cost_per_week_inr (that times answers_per_week, rounded to 2 places, as a string).
The Check prices usages of its own (317 in and 93 out is Rs 0.1378), then sends two answers and reads GET /costs as Anjali: a week of 3,000 answers.
- Task 2
batch.py: a class's answers at once
In app/grading/batch.py write grade_batch(ai, question, texts, *, workers=None, model=None): a ThreadPoolExecutor with max_workers = workers or settings.workers, one submit of grade_answer(ai, question, text, model=model) per text, then in the texts' order a BatchItem(index, graded, None) for each result, or BatchItem(index, None, error) with the error's first line when that one raised.
The Check grades nine texts with a slow stand-in for grade_answer (at most 3 at once, never one at a time), one text that fails among them, then six answers with lab-chat-limited (3 requests at a time) and POST /batches as Anjali.
- Task 3 · interview question
Interview: feedback.py, what the student reads
In app/feedback.py write point_message(marks, out_of, tip) ("Well done: 2 of 2.", "1 of 2: you are close. <tip>", "0 of 2: <tip>"), summary(score, out_of) (all the marks, 60% or more, more than 0, none: the four sentences of the stub), and feedback(question, points): {"score", "out_of", "summary", "points": [{"point", "marks", "out_of", "message"}]} in rubric order.
The Check writes feedback for grades of its own, then Aditi sends an answer with POST /answers and reads the feedback on it.
Karthik will ask: the model already writes a reason for every point. Why build the student's feedback from the rubric's tips in code, instead of asking the model to write it?
What you will learn
- Tokens and rupees: what a reply's usage says, what a provider charges, and why money is a Decimal.
- A week's estimate from the average answer.
- A thread pool: why a few requests waiting at once is faster when the waiting is the model's, and why the pool is no bigger than the provider allows.
- Futures: a result that arrives later, and an error that arrives with it.
- Feedback from the rubric: kind, specific, the same words for the same marks.
- The take-home: the whole grader on your own machine, with a real model.
More in Ready for the institute
Build "Cost and feedback" yourself
Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI answer grader project are free. This one comes with the all labs plan.