AI invoice reader project (TypeScript) · Chapter 10 of 12
Evals
By the end of this chapter you can run the whole reader over every invoice Vaishali labelled by hand, count how often each field comes back right, and find the number that matters most to accounts: the wrong invoices the reader accepted without asking anyone. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Ready for accounts
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of AI invoice reader project (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Today is the October month-end close, and Sayali wants to switch the reader on for it. Her question is not "do the functions work?". The Check has answered that nine times. Her question is "how often is it right, on our invoices?" Vaishali knows that mistakes are not equal. A wrong invoice that waits in her review queue costs her a minute. A wrong invoice that nobody sees, like September's ₹48,600 typed as ₹46,800, is paid wrong and found at the vendor's statement, if it is ever found.
Everything you have built looks at one invoice at a time. A rule says whether this total adds up; a score says whether this value is in this text. None of it says how many of 40 invoices come out right, which fields are weak, or how many wrong invoices slipped through without a review task. Vaishali has typed what is printed on all 40 invoices of data/inbox into data/labels. In this chapter you compare the reader with her labels, and count.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
accuracy.ts: right or wrong, field by field
In src/evals/accuracy.ts write sameValue(a, b): when either is null or undefined, true only when both are; when b is a number, true when a is the same number of paise (a number, or a string that is not empty and turns into a number with Number(): "1280.00" is 1280; anything else is not equal); otherwise a === b. Then compare(pred, label): { field: boolean } for every field of FIELDS; a header field with sameValue(pred[f], label[f]); line_items true only with as many lines as the label and every LINE key of every line the same. pred null: every field false.
The Check tries twelve pairs of values, then compares a right prediction (some numbers as text), one with a translated name and a date read month first, one with a line missing, one with cgst "0" where the label has none, and null.
- Task 2
accuracy.ts: accuracy per field
Write fieldAccuracy(results), each result { file, status, shouldReview, fields: compare(...) }: return { invoices: n, fields: { field: the share of results where it is true, rounded to 3 places (0 with no results) }, allRight: how many results have every field true }.
The Check gives three results (one all right, one with line_items wrong, one with total and line_items wrong), then none.
- Task 3 · interview question
Interview: accuracy.ts, what each mistake costs
Write outcomes(results): silentErrors, the files accepted (status 'accepted') with a field wrong; missed, the files accepted whose shouldReview is true; unneededReviews, the files not accepted whose fields are all right and whose shouldReview is false; reviewRate, the share not accepted (3 places, 0 with none). Lists in the results' order.
The Check gives eight results of every kind, then calls GET /evals/report with prompt 1 and with prompt 2 as Vaishali (prompt 1 translates names on invoices it accepts: silent errors; prompt 2 has none), and as Omkar (403).
Nikhil will ask: with prompt 1, nine of the ten fields are over 90% right. Why is silentErrors the number Vaishali reads first?
What you will learn
- What an eval is: the whole reader over labelled examples, counted, and what it measures that a test does not.
- Labels: the right answer, typed by a person, and comparing two values the way an accountant means "the same".
- Accuracy per field: one number per field, so you can see where the reader is weak.
- The four outcomes: accepted or sent to review, right or wrong, and why a silent error is the worst of them.
- Prompt versions: changing the instructions, and measuring the change with the same eval.
More in Ready for accounts
Build "Evals" yourself
Parda's venues get invoices from sound, printing and canteen vendors, typed and messy. You build the pipeline that reads them, checks every rupee, and sends the doubtful ones to a person. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI invoice reader project (TypeScript) are free. This one comes with the all labs plan.