AI invoice reader project (TypeScript) · Chapter 9 of 12
Batch jobs
By the end of this chapter you can queue a whole inbox in one request that answers at once; run workers, one or many, that read the invoices one job at a time; put a failed job back for later, waiting longer each time; and let two workers share one queue without ever taking the same job, with FOR UPDATE SKIP LOCKED. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: People and volume
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of AI invoice reader project (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Today is Saturday 31 October, the month-end close. The month's invoices are in the inbox, all 40, and POST /invoices reads one file per request while Omkar waits. In the lab the stand-in model answers at once (mudran-mp-0614.pdf, two pages, took 0.14 s); a real model takes seconds per invoice, and its API is sometimes busy for a minute. Forty requests in a row means forty waits, and three things go wrong:
- The model's API is busy for one file. That request answers 502 and the file is read by nobody, unless someone notices and sends it again.
- The server restarts halfway through. Whatever was in progress is simply gone; nothing says which files were done.
swarnim-ssl-2219.pdfandswarnim-ssl-2219-copy.pdfare the same PDF (it came on WhatsApp and by email). Read both and Parda pays the model twice for one invoice, then Vaishali gets a duplicate to reject.
What is missing is a written list of work: each file to read, its state, how many tries it has had, and when to try again. That list is a job queue, and here it is a table.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
queue.ts: a job per file
In src/jobs/queue.ts write enqueue(files, user): first each file's sha256 (await fileHash(inboxPath(name)): a name that is not in the inbox throws before anything is queued). Then in ONE db.transaction, for each name in order: the job that already has that sha256, or a new one inserted with .returning() (file, sha256, venue: user.venue, submittedBy: user.email). Return the jobs in the order of the files.
The Check queues swarnim-ssl-2219.pdf, neer-nas-0781.pdf, swarnim-ssl-2219.pdf again and swarnim-ssl-2219-copy.pdf (the same bytes under another name): 2 jobs, and the same job back for the same PDF. Then a list with a file that is not there: nothing queued.
- Task 2
queue.ts: done, or try again later
Write finish(job, invoiceId): update the job to status 'done', its invoiceId, lastError null, updatedAt sql
now(), and return it (.returning()). Then fail(job, error): lastError (its first 500 characters), lockedBy null, updatedAt sqlnow(); with job.attempts at config.maxJobAttempts or more, status 'failed'; otherwise status 'queued' and runAfter sqlnow() + make_interval(secs => ${wait}), wait being config.retryAfterSeconds * 2 ** (job.attempts - 1). Return the updated job.The Check finishes a job, then fails jobs on their first, second and third attempt: due in 30 s, in 60 s, then failed.
- Task 3 · interview question
Interview: queue.ts, claim with SKIP LOCKED
Write claim(worker): in one db.transaction, select the oldest job (asc(runAfter), then asc(id)) whose status is 'queued' and runAfter <= sql
now(), .limit(1).for('update', { skipLocked: true }). None: return null. Otherwise update it to status 'running', lockedBy the worker, attempts one more, updatedAt sqlnow(), and return it (.returning()).The Check queues three jobs (one due in 2099), holds the oldest locked in a transaction of its own and runs your claim: it must not wait, and must take the second. Then it claims the first once it is free, gets null, and queues two files through POST /jobs and runs them with POST /jobs/work.
Nikhil will ask: two workers ask for the next job at the same moment. What does FOR UPDATE alone do, and what does SKIP LOCKED add?
What you will learn
- A job queue in a table: one row per file, queued, running, done or failed.
- One job per file: the file's hash, not its name, and a unique constraint.
- Workers: processes that claim a job, run it and record what happened.
- Retries with backoff: try again later, wait longer each time, give up after three.
- ` sqlnow() `: the database's clock, the same all through one transaction.
- FOR UPDATE SKIP LOCKED: many workers, one queue, nobody waiting, no job taken twice.
More in People and volume
Build "Batch jobs" yourself
Parda's venues get invoices from sound, printing and canteen vendors, typed and messy. You build the pipeline that reads them, checks every rupee, and sends the doubtful ones to a person. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI invoice reader project (TypeScript) are free. This one comes with the all labs plan.