AI invoice reader project · Chapter 6 of 12
Retries with feedback
By the end of this chapter the reader asks the model again when the rules find problems: it sends the whole conversation back with the model's own answer and a message that says what is wrong, keeps the corrected answer, stops as soon as an answer comes back with the same problems, and adds up what every request cost in tokens. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Checking the model
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of AI invoice reader project are free. This one comes with the all labs plan.
Why you need this
Chapter 5's rules, run on the 39 invoices of the inbox that have text, find problems in 18. Twelve of them are the model's own misreadings: dates read month first and plain Totals taken for the total, on Shree Ganesh, Gokul Dairy, Maharashtra Oil and QuickPay invoices. The paper is right every time; the model just read it the quick way. The other six are on the paper itself, like Shree Ganesh's 604 for 10 x 64 and Surat Steel's GSTIN.
Rafiq could fix the twelve by hand. But the rules already know exactly what is wrong with each: "The invoice date 2026-01-10 is more than 120 days before today", "₹38,060.00 plus ₹1,903.00 of tax is ₹39,963.00, but the total is ₹38,060.00". Tell the model that, and it can read the invoice again with those fields in mind. Rafiq should only see what is still wrong after that.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
retry.py: the feedback, and comparing problems
In app/extract/retry.py write feedback(problems): the lines "Your JSON does not match the invoice. These checks failed:", then f"- {p.field}: {p.message}" for each problem, then "Read the invoice text again and answer with the corrected JSON for the whole invoice.", joined with "\n". Then same(a, b): True when both lists hold the same set of (field, code) pairs.
The Check compares your feedback for two problems with its own, then tries same on four pairs of lists.
- Task 2
retry.py: ask again with the problems
Write extract_checked(ai, text, instructions=None, today=None): messages = build_messages(text, instructions); reply = ask_model(ai, messages); check_invoice(reply.invoice, today). While there are problems and fewer than settings.max_attempts requests were made, set messages to messages plus {"role": "assistant", "content": reply.content} and {"role": "user", "content": feedback(problems)}, ask again and check again. Add up every reply's usage; return Extraction(invoice, problems, attempts, usage, messages) with the last checked answer.
The Check reads the lab model's request log for oil-mod-2291.pdf (the first answer reads the date month first and the plain Total as the total; one retry fixes both), patel-prg-0417.pdf (right the first time) and ganesh-sg-1063.pdf (a line printed wrong: the problem stays).
- Task 3 · interview question
Interview: retry.py, stop when nothing changes
In extract_checked's loop, stop as soon as an answer's problems are the same (same()) as the problems of the answer before it.
The Check runs ganesh-sg-1063.pdf, whose line 4 is printed 10 x 64 = 604: the retry gives the same problem, so it wants exactly 2 requests. Then POST /invoices/read for gokul-gd-3381.pdf (one retry fixes it).
Kunal will ask: why does the same problem coming back mean the invoice is wrong, not the model? And what does each retry cost?
What you will learn
- The model remembers nothing: a retry sends the whole conversation again.
- The assistant role: putting the model's own answer back into the conversation.
- Feedback written for the model: the rules' own words, one line per problem.
- Comparing problems as sets of (field, code) pairs.
- When to stop: no problems, settings.max_attempts, or the same problems twice.
- What a retry costs, in tokens.
More in Checking the model
Build "Retries with feedback" yourself
DabbaDrop's kitchens get piles of supplier invoices, typed and messy. You build the pipeline that reads them, checks every number, and sends the doubtful ones to a person. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI invoice reader project are free. This one comes with the all labs plan.