AI invoice reader project (TypeScript) · Chapter 11 of 12

Writing tests

By the end of this chapter you can write Vitest tests for the parts of the invoice reader that must never break (the amount rules, the confidence scores and the retry loop), test code that calls a model with a fake one, and tell a test that protects the code from one that only passes. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ready for accounts
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of AI invoice reader project (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Chapter 10 showed what one sentence in the instructions can do: prompt 2 took the silent errors from 2 to 0. That is also the danger. The reader will keep changing. Asha will book a new vendor with a new layout, Vaishali wants a rule for TDS, Nikhil wants to try a cheaper model. Every change can break something that works today, and three things decide whether a vendor is paid right:

  • the rules: a line printed 50 x 120 = 6,500 must be found, and a total that forgot the taxes must be found;
  • the confidence scores: an amount the model made up must not be "found" inside a longer number in the text;
  • the retry loop: a retry must tell the model what is wrong, and stop when the same problem comes back.

The eval cannot guard these. It needs the whole reader and a model, and when a number drops it does not say which line of code broke. The Checks cannot either: they are not there when someone changes checkTotals on a Friday evening in Diwali week. Tests in the repository are. They run in under a second, need no database and no model, and name the function that broke. In this chapter you write them, and the Check grades them the hard way: on our code they must pass, and on broken copies of our code they must fail.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    rules.test.ts: test the amount rules

    In tests/rules.test.ts write Vitest tests (it(...) blocks) for checkLines and checkTotals with makeInvoice and good from tests/helpers.ts: the good invoice gets [] from both; with line 2's amount 3900 (2 x 1,800 is 3,600), checkLines finds ["line_items[1].amount", "line_amount"] as [field, code] pairs; with a total of 10600 (no taxes), checkTotals finds ["total", "total"]. Run npm test in the terminal.

    The Check runs your file on our code, where every test must pass, and on two broken copies, where at least one test must fail: checkLines that lets wrong amounts through, checkLines that reports every line, and checkTotals that forgets the taxes.

  2. Task 2

    confidence.test.ts: test the confidence scores

    In tests/confidence.test.ts write a short invoice text in the file (a name, a GSTIN, lines with Indian amounts such as 7,000.00 and 3,600.00, a Total 10,600.00 and a Grand Total 12,508.00) and test: appearsIn finds 12508; it does not find 600, only part of 10,600.00 and 3,600.00; scoreFields(makeInvoice(), text, [a problem on "line_items[1].amount" with the message "Line 2 is wrong"]) gives line_items[1] a confidence below 0.8 with that message as a reason, and line_items[0] 1.

    The Check runs your file on our code, where every test must pass, and on two broken copies, where at least one test must fail: appearsIn that finds 600 inside 10,600.00, and scoreFields that ignores the problems.

  3. Task 3 · interview question

    Interview: retry.test.ts, a fake model

    In tests/retry.test.ts test extractChecked(ai, text, { today: TODAY }) with fakeAi from tests/helpers.ts. First, fakeAi({ ...good(), total: 10600 }, good()): attempts 2, no problems left, and ai.requests[1].messages ends with an assistant message, then a user message whose content contains "total". Second, the same wrong answer three times (good with line 2's amount 3900, subtotal 10900, total 12808): attempts 2 and the problems' codes ["line_amount"].

    The Check runs your file on our code, where every test must pass, and on two broken copies, where at least one test must fail: a retry that sends no feedback, and a loop that never stops early.

    Nikhil will ask: how do you test code that calls a model, without the model? And what can such a test not tell you?

What you will learn

  • Vitest: describe, it, expect and its matchers, and how to read a failure.
  • Test data: the given tests/helpers.ts, a good invoice with one field changed.
  • Testing a rule: one good case and one broken case, compared as [field, code] pairs.
  • A fake model: a stand-in client that answers from a script and keeps every request.
  • Mutation testing: breaking the code on purpose to see whether a test notices.
  • What a fake proves, and what only an eval can show.

More in Ready for accounts

Build "Writing tests" yourself

Parda's venues get invoices from sound, printing and canteen vendors, typed and messy. You build the pipeline that reads them, checks every rupee, and sends the doubtful ones to a person. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI invoice reader project (TypeScript) are free. This one comes with the all labs plan.