AI invoice reader project · Chapter 11 of 12

Writing tests

By the end of this chapter you can write pytest tests for the parts of the invoice reader that must never break (the amount rules, the confidence scores and the retry loop), test code that calls a model with a fake one, and tell a test that protects the code from one that only passes. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ready for accounts
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI invoice reader project are free. This one comes with the all labs plan.

Why you need this

Chapter 10 showed what one sentence in the instructions can do: prompt 2 took the silent errors from 8 to 0. That is also the danger. The reader will keep changing. Farah wants a new supplier's layout, Rafiq wants a rule for TDS, Kunal wants to try a cheaper model. Every change can break something that works today, and three things decide whether money is paid right:

  • the rules: a line printed 5 x 2650 = 13,520 must be found, and a total that forgot the taxes must be found;
  • the confidence scores: an amount the model made up must not be "found" inside a longer number in the text;
  • the retry loop: a retry must tell the model what is wrong, and stop when the same problem comes back.

The eval cannot guard these. It takes the whole reader and a model, and when a number drops it does not say which line of code broke. The Checks cannot either: they are not there when Kunal changes check_totals on a Friday evening. Tests in the repository are. They run in under a second, need no database and no model, and name the function that broke. In this chapter you write them, and the Check grades them the hard way: on our code they must pass, and on broken copies of our code they must fail.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    test_rules.py: test the amount rules

    In tests/test_rules.py, write tests (functions named test_...) for check_lines and check_totals with the invoice and good fixtures: the good invoice gets [] from both; with line 2's amount 13520 (5 x 2650 is 13,250), check_lines finds ("line_items[1].amount", "line_amount") as (field, code) pairs; with a total of 34850 (no taxes), check_totals finds ("total", "total"). Run pytest in the terminal.

    The Check runs your file on our code, where every test must pass, and on two broken copies, where at least one test must fail: check_lines that lets wrong amounts through, and check_totals that forgets the taxes.

  2. Task 2

    test_confidence.py: test the confidence scores

    In tests/test_confidence.py, write a short invoice text in the file (a name, a GSTIN, lines with Indian amounts such as 21,600.00 and 13,250.00, a Grand Total 36,592.50) and test: appears_in finds Decimal("36592.5"); it does not find Decimal("250"), only part of 13,250.00; score_fields(invoice(), text, [Problem("line_items[1].amount", "line_amount", "Line 2 is wrong")]) gives line_items[1] a confidence below 0.8 with that message as a reason, and line_items[0] 1.0.

    The Check runs your file on our code, where every test must pass, and on two broken copies, where at least one test must fail: appears_in that finds 250 inside 13,250.00, and score_fields that ignores the problems.

  3. Task 3 · interview question

    Interview: test_retry.py, a fake model

    In tests/test_retry.py, test extract_checked(ai, text, today=today) with the fake_ai fixture. First, fake_ai({**good, "total": 34850}, good): attempts 2, no problems left, and ai.requests[1]["messages"] ends with an assistant message, then a user message whose content contains "total". Second, the same wrong answer three times (good with line 2's amount 13520, subtotal 35120, total 36862.5): attempts 2 and the problems' codes ["line_amount"].

    The Check runs your file on our code, where every test must pass, and on two broken copies, where at least one test must fail: a retry that sends no feedback, and a loop that never stops early.

    Kunal will ask: how do you test code that calls a model, without the model? And what can such a test not tell you?

What you will learn

  • pytest: test functions, plain assert, and how to read a failure.
  • Fixtures: asking for a value by naming a parameter; tests/conftest.py's four.
  • Testing a rule: one good case and one broken case, compared as (field, code) pairs.
  • A fake model: a stand-in client that answers from a script and keeps every request.
  • Mutation testing: breaking the code on purpose to see whether a test notices.
  • What a fake proves and what only an eval can show.

More in Ready for accounts

Build "Writing tests" yourself

DabbaDrop's kitchens get piles of supplier invoices, typed and messy. You build the pipeline that reads them, checks every number, and sends the doubtful ones to a person. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI invoice reader project are free. This one comes with the all labs plan.