AI invoice reader project · Chapter 7 of 12
Confidence and the review queue
By the end of this chapter you can give every field the reader fills in a confidence and the reasons for it, found as evidence in the invoice's own text; store the invoice, its fields and, when a person must look at it, a review task, all in one transaction; and catch an invoice that is already stored. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Checking the model
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of AI invoice reader project are free. This one comes with the all labs plan.
Why you need this
Chapter 6's reader reads an invoice, checks it and asks again when a rule fails. Then it forgets everything: the answer goes back in the response and nothing is kept. And it cannot tell Rafiq which answers to look at.
Take ganesh-sg-1052.pdf. Shree Ganesh prints Date: 09/10/2026, the 9th of October. The model reads it month first, as 10 September. Every rule passes: the date is not in the future and not 120 days old. The reader would hand back a wrong date with no problem at all, exactly the "date typed month first" that went wrong in September. Kisan Fresh Farms prints Marathi names, "Kanda" and "Batata", and the model writes "Onion" and "Potato". The amounts add up, so no rule says a word, but Rafiq can no longer match the bill line by line against what the farm sent.
And a rule cannot catch Patel Rice's invoice being paid twice, because each copy is right on its own. Only something that remembers the first one can.
So this chapter adds three things: evidence for every field, a database that keeps every invoice, and a review queue: a list of invoices Rafiq must look at, each with the reason, while the rest go through.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
confidence.py: is the value on the invoice?
In app/review/confidence.py write appears_in(text, value): None is True; a str is True when it is in the text ignoring case and runs of whitespace; a Decimal or a date is True when one of renderings(value) is in the text as a whole token (not right after a digit, dot or comma; not followed by a digit, or by a dot or comma and a digit).
The Check looks for fifteen values in an oil invoice's text: the name in other spacing and case, a made-up name, 36592.5 printed 36,592.50, 592.50 and 250 (only parts of longer numbers), 8 October printed 08/10/2026, 10 August (the date read month first), and 3 October in "Dt. 3.10.26".
- Task 2
confidence.py: a score and reasons for every field
Write score_fields(inv, text, problems): a FieldScore per field, HEADER's names in order, then f"line_items[{i}]" per line; value from inv.model_dump(mode="json"); confidence 1.0. A header field whose value (getattr(inv, name)) is not appears_in the text, or a line whose description, quantity, rate or amount is not: confidence NOT_FOUND and the reason "not found in the invoice text". Each problem whose field is the name or starts with the name and a dot: confidence at most PROBLEM, and its message as a reason.
The Check (with its own appears_in, so this task stands alone) scores an invoice read right, then one with a made-up name, a date read month first, an item named differently, a wrong line and a total that does not add up.
- Task 3 · interview question
Interview: pipeline.py, store it all at once
In app/pipeline.py write store(db, *, file, sha256, user, invoice, problems, scores, attempts, usage, notes=()). First, when the invoice has a supplier_gstin, look for a stored InvoiceRow with the same supplier_gstin and invoice_number in its data and a status that is not rejected; if there is one, add Problem("invoice_number", "duplicate", a sentence naming its id) to a copy of problems. Then reason = review_reason(problems, scores, list(notes)). Add an InvoiceRow (file, sha256, kitchen=user.kitchen, submitted_by=user.email, status "needs_review" with a reason or "accepted", data=invoice.model_dump(mode="json"), problems as dicts, attempts, tokens_in, tokens_out), flush, add a FieldRow per score and a ReviewTask when there is a reason, and commit ONCE at the end. Return the row.
The Check stores an invoice, then the same supplier's same number again (a duplicate with a review task), then one whose field row cannot be saved (nothing of it may stay), then POST /invoices for patel-prg-0462.pdf, whose printed total does not add up.
Kunal will ask: why save the invoice, its fields and its review task in one transaction?
What you will learn
- Confidence as evidence: a number from 0 to 1 your code works out from what it found, not a feeling of the model's.
- Looking for a value in the text: one number, many printed forms, and regex lookarounds ((?<!...), (?!...)).
- The tables: migration 0001, invoices (JSONB data), invoice_fields, review_tasks.
- flush(): the INSERT sent, the id known, nothing committed yet.
- One transaction: why an invoice and its review task are saved together or not at all.
- Duplicates: the same supplier's same invoice number, found with a JSONB query.
More in Checking the model
Build "Confidence and the review queue" yourself
DabbaDrop's kitchens get piles of supplier invoices, typed and messy. You build the pipeline that reads them, checks every number, and sends the doubtful ones to a person. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI invoice reader project are free. This one comes with the all labs plan.