AI answer grader project (TypeScript) · Chapter 12 of 12
Cost and feedback
By the end of this chapter you can say in rupees what grading one answer and a week of answers costs, grade a whole class's answers a few at a time within the provider's limit, and give each student feedback that says, point by point, what they got and what to do next. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Ready for the institute
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of AI answer grader project (TypeScript) are free. This one comes with the all labs plan.
Why you need this
The grader is measured, tested and gated. Before Kavita switches it on for every batch, three questions are left.
What does it cost? About 3,000 answers a week. Kavita has a budget, and "a few tokens an answer" is not a number she can plan with. She wants rupees: per answer, and per week.
How long does Saturday night take? POST /answers grades one answer while the student waits. Anjali wants to grade a whole class's answers to one question at once. One after another, each one waits for the model's reply before the next is sent: chem-salt's 17 answers on lab-chat-limited take 8.72 seconds. A real model can take a few seconds a reply, and 3,000 answers one at a time would run for hours.
What does Aditi read? Today a final grade gives her a number: "score": 3, "outOf": 5. She cannot tell which point she missed or what to write next time, which was the reason Kavita started these answers in the first place. The grade has the model's reasons, but they were written for the code and for a teacher, not for a student.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
usage.ts: what grading costs
In src/cost/usage.ts write costInr(usage): (input x config.priceInput + output x config.priceOutput) / 1,000,000, rounded to 4 places. Then weekEstimate(usages, answersPerWeek): answersGraded, avgInputTokens and avgOutputTokens (rounded to whole tokens), costPerAnswerInr (the cost of the average), answersPerWeek and costPerWeekInr (that times answersPerWeek, rounded to 2 places).
The Check prices usages of its own (317 in and 93 out is Rs 0.1378), then stores two grades with SQL and reads GET /costs as Anjali: a week of 3,000 answers is Rs 413.4.
- Task 2
batch.ts: a class's answers at once
In src/grading/batch.ts write gradeBatch(ai, question, texts, { workers, model }): grade every text with gradeAnswer(ai, question, text, { model }), at most workers (config.workers) at a time, and return in the texts' order { index, graded, error: null } for each, or { index, graded: null, error: firstLine(e) } when that one threw.
The Check grades nine texts with a slow fake model that counts the requests out at once (at most 3, never one at a time), one text that fails among them, then six answers with lab-chat-limited (3 requests at a time) and POST /batches as Anjali.
- Task 3 · interview question
Interview: feedback.ts, what the student reads
In src/feedback.ts write pointMessage(marks, outOf, tip) ("Well done: 2 of 2.", "1 of 2: you are close. <tip>", "0 of 2: <tip>"), summary(score, outOf) (all the marks, 60% or more, more than 0, none: the four sentences of the stub), and feedback(question, points): { score, outOf, summary, points: [{ point, marks, outOf, message }] } in rubric order.
The Check writes feedback for grades of its own, then Aditi sends an answer with POST /answers and reads the feedback on it.
Karthik will ask: the model already writes a reason for every point. Why build the student's feedback from the rubric's tips in code, instead of asking the model to write it?
What you will learn
- Tokens and rupees: what a reply's usage says, what a provider charges, and why money is added up in whole numbers.
- A week's estimate from the average answer.
- Async workers: why a few requests waiting at once is faster when the waiting is the model's, and why there are no more of them than the provider allows.
- Order and failures: results in the order sent, and one error that stays one error.
- Feedback from the rubric: kind, specific, the same words for the same marks.
- The take-home: the whole grader on your own machine, with a real model.
More in Ready for the institute
Build "Cost and feedback" yourself
Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI answer grader project (TypeScript) are free. This one comes with the all labs plan.