AI answer grader project · Chapter 3 of 12

A first grader

By the end of this chapter you can send a question, its rubric and a student's answer to a chat model in one request, ask for a grade in an exact shape (marks per rubric point, each with a quote from the answer), read the reply back into Python objects with its token count, and refuse a reply that is cut off, refused or not a grade. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: A first grader
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI answer grader project are free. This one comes with the all labs plan.

Why you need this

Chapter 1's keywords agreed with the teachers on 69 of 200 answers, and missed in three ways: a list of words got full marks (A008), a wrong sentence with the right word got marks (A023, "no gravity in space"), and right answers in other words got nothing (A005, "less dense"). All three need someone to read the meaning of a sentence and compare it with each rubric point. Chapter 2 made the rubrics safe to trust.

A language model can read meaning. But its reply is text, and the grader needs numbers it can add, store and show to Aditi. So this chapter does two things: one request per answer that asks for a grade in an exact shape, and a hard "no" whenever the reply is not a whole grade.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    prompt.py: what the model is sent

    In app/grading/prompt.py write rubric_lines(question): one line per point, "- p1 (2 marks): <text>" ("1 mark" for a point worth 1), joined with newlines. Then build_messages(question, answer, version=None): the system message PROMPTS[version] (PROMPT_VERSION when version is None) and a user message "Question: <text>", a blank line, "Rubric (<marks> marks):" and the rubric lines, a blank line, "Student answer:" and the answer on the next line.

    The Check builds the messages for phy-ice and a short answer, with and without version=1, and compares them with its own.

  2. Task 2

    schema.py: the shape of a grade

    In app/grading/schema.py give PointGrade the fields id (str), awarded (int), evidence (str) and reason (str), and ModelGrade the fields points (a list of PointGrade) and total (int), keeping extra="forbid". Then response_format(): {"type": "json_schema", "json_schema": {"name": "grade", "strict": True, "schema": ModelGrade.model_json_schema()}}.

    The Check reads your schema (every field required, no extra keys allowed), sends it to the lab model to see it accepted as strict, and reads a grade back with read_grade.

  3. Task 3 · interview question

    Interview: model.py, one request, one grade

    In app/grading/model.py write ask_grader(ai, messages, *, model=None, temperature=None, seed=None): one ai.chat.completions.create with model (or settings.chat_model), messages, temperature (settings.temperature when None), response_format=response_format(), max_completion_tokens=settings.max_output_tokens, and seed only when it is not None. Raise GradingFailed when the first choice's finish_reason is "length" or its message has a refusal; read the content with read_grade, turning GradeJsonError into GradingFailed. Return Reply(content, grade, Usage(prompt_tokens, completion_tokens)). Then grade_once: ask_grader with build_messages(question, answer, version).

    The Check reads the lab model's request log for one answer, tries a seed, a 10-token limit, lab-chat-sloppy (JSON in a code fence) and a refusal, then POST /grade as Aditi.

    Karthik will ask: the app only needs one number, the score. Why ask the model for marks per rubric point, each with a quote from the answer?

What you will learn

  • A chat request: a list of messages in, one reply out; nothing is remembered between requests.
  • The parameters: model, messages, temperature, seed, response_format, max_completion_tokens.
  • A structured output: a JSON schema in the request, and what strict demands of it.
  • The reply: choices[0].message.content, finish_reason, refusal, and usage (tokens).
  • When there is no grade: cut off, refused, or not JSON.

More in A first grader

Build "A first grader" yourself

Shikhar Classes in Kota marks thousands of written answers by hand every week. You build the grader that marks them per rubric point, and prove it agrees with its teachers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI answer grader project are free. This one comes with the all labs plan.