AI answer grader project (TypeScript) · Chapter 3 of 12

A first grader

By the end of this chapter you can send a question, its rubric and a student's answer to a chat model in one request, ask for a grade in an exact shape (marks per rubric point, each with a quote from the answer), read the reply back into a typed object with its token count, and refuse a reply that is cut off, refused or not a grade. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: A first grader
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of AI answer grader project (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Chapter 1's keywords agreed with the teachers on 69 of 200 answers: a list of words got full marks (A008), "no gravity in space" got marks for "gravity" (A023), and "less dense" got nothing (A005). All three need someone to read the meaning of a sentence and compare it with each rubric point. Chapter 2 made the rubrics safe to trust.

A language model can read meaning. But its reply is text, and the grader needs numbers it can add, store and show to Aditi. So this chapter does two things: one request per answer that asks for a grade in an exact shape, and a hard "no" whenever the reply is not a whole grade.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    prompt.ts: what the model is sent

    In src/grading/prompt.ts write rubricLines(question): one line per point, "- p1 (2 marks): <text>" ("1 mark" for a point worth 1), joined with newlines. Then buildMessages(question, answer, version = PROMPT_VERSION): the system message PROMPTS[version] and a user message "Question: <text>", a blank line, "Rubric (<marks> marks):" and the rubric lines, a blank line, "Student answer:" and the answer on the next line.

    The Check builds the messages for phy-ice and a short answer, with and without version 1, and compares them with its own.

  2. Task 2

    schema.ts: the shape of a grade

    In src/grading/schema.ts give PointGrade the fields id (a string), awarded (a whole number), evidence (a string) and reason (a string), and ModelGrade the fields points (an array of PointGrade) and total (a whole number), keeping .strict(). Then responseFormat(): zodResponseFormat(ModelGrade, "grade").

    The Check reads your schema (every field required, no extra keys allowed), sends it to the lab model to see it accepted as strict, and reads grades back with readGrade (a key the grade does not have must be refused).

  3. Task 3 · interview question

    Interview: model.ts, one request, one grade

    In src/grading/model.ts write askGrader(ai, messages, { model, temperature, seed }): one await ai.chat.completions.create with model (or config.chatModel), messages, temperature (config.temperature when not given), response_format: responseFormat(), max_completion_tokens: config.maxOutputTokens, and seed only when one is given (0 is a seed). Throw GradingFailed when the first choice's finish_reason is "length" or its message has a refusal; read the content with readGrade, turning a GradeJsonError into a GradingFailed. Return { content, grade, usage: { input, output } } from the reply's usage. Then gradeOnce: askGrader with buildMessages(question, answer, version).

    The Check reads the lab model's request log for one answer, tries a seed, a 10-token limit, lab-chat-sloppy (JSON in a code fence) and a refusal, then POST /grade as Aditi.

    Karthik will ask: the app only needs one number, the score. Why ask the model for marks per rubric point, each with a quote from the answer?

What you will learn

  • A chat request: a list of messages in, one reply out; nothing is remembered between requests.
  • The parameters: model, messages, temperature, seed, response_format, max_completion_tokens.
  • A structured output: a JSON schema in the request, what zodResponseFormat writes, and what strict demands.
  • The reply: choices[0].message.content, finish_reason, refusal, and usage (tokens).
  • Why ?? and not || for a setting whose value can be 0.
  • When there is no grade: cut off, refused, or not JSON.

More in A first grader

Build "A first grader" yourself

Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI answer grader project (TypeScript) are free. This one comes with the all labs plan.