AI invoice reader project · Chapter 10 of 12
Evals
By the end of this chapter you can run the whole reader over every invoice Rafiq labelled by hand, count how often each field comes back right, and find the number that matters most to accounts: the wrong invoices the reader accepted without asking anyone. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Ready for accounts
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of AI invoice reader project are free. This one comes with the all labs plan.
Why you need this
Farah wants to switch the reader on for the October close. Her question is not "do the functions work?". The Check has answered that nine times. Her question is "how often is it right, on our invoices?" And Rafiq knows that mistakes are not all equal. A mistake somebody sees costs a minute to fix. A mistake nobody sees, like September's Patel Rice invoice paid twice, costs money until a supplier's statement shows it, if it ever does.
Everything you have built so far looks at one invoice at a time. A rule says whether this total adds up; a score says whether this value is in this text. None of it says how many of 40 invoices come out right, which fields are weak, or how many wrong invoices slipped through without a review task. Rafiq has typed what is printed on all 40 invoices of data/inbox into data/labels. In this chapter you compare the reader with his labels and count.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
accuracy.py: right or wrong, field by field
In app/evals/accuracy.py write same_value(a, b): None equals only None; when b is an int or float (not a bool), equal when Decimal(str(a)) == Decimal(str(b)), False when a is not a number (ArithmeticError); otherwise a == b. Then compare(pred, label): {field: bool} for every field of FIELDS; a header field with same_value(pred.get(f), label.get(f)); line_items True only with as many lines as the label and every LINE key of every line the same. pred None: every field False.
The Check tries nine pairs of values, then compares a right prediction (numbers as text, as the API writes them), one with a translated name and a date read month first, one with a line missing, one with cgst "0" where the label has none, and None.
- Task 2
accuracy.py: accuracy per field
Write field_accuracy(results), each result {"file", "status", "should_review", "fields": compare(...)}: return {"invoices": n, "fields": {field: share of results where it is True, rounded to 3 places (0.0 with no results)}, "all_right": how many results have every field True}.
The Check gives three results (one all right, one with line_items wrong, one with total and line_items wrong), then none.
- Task 3 · interview question
Interview: accuracy.py, what each mistake costs
Write outcomes(results): "silent_errors", the files accepted with a field wrong; "missed", the files accepted whose should_review is True; "unneeded_reviews", the files not accepted whose fields are all right and whose should_review is False; "review_rate", the share not accepted (3 places, 0.0 with none). Lists in the results' order.
The Check gives eight results of every kind, then calls GET /evals/report with prompt 1 and with prompt 2 as Rafiq (prompt 1 translates names on invoices it accepts: silent errors; prompt 2 has none), and as Meera (403).
Kunal will ask: with prompt 1, nine of the ten fields are over 90% right. Why is silent_errors the number Rafiq reads first?
What you will learn
- What an eval is: the whole reader over labelled examples, counted, and what it measures that a test does not.
- Labels: the right answer, typed by a person, and comparing "1280.00" with 1280 as an invoice means it.
- Accuracy per field: one number per field, so you can see where the reader is weak.
- The four outcomes: accepted or sent to review, right or wrong, and why a silent error is the worst of them.
- Prompt versions: changing the instructions, and measuring the change with the same eval.
More in Ready for accounts
Build "Evals" yourself
DabbaDrop's kitchens get piles of supplier invoices, typed and messy. You build the pipeline that reads them, checks every number, and sends the doubtful ones to a person. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI invoice reader project are free. This one comes with the all labs plan.