AI invoice reader project · Chapter 9 of 12

Batch jobs

By the end of this chapter you can queue a whole inbox in one request that answers at once; run workers, one or many, that read the invoices one job at a time; put a failed job back for later, waiting longer each time; and let two workers share one queue without ever taking the same job, with FOR UPDATE SKIP LOCKED. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: People and volume
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI invoice reader project are free. This one comes with the all labs plan.

Why you need this

Today is Saturday 31 October, the month-end close. Meera has the month's invoices in the inbox, all 40, and POST /invoices reads one file per request while she waits. In the lab the stand-in model answers at once (packright-prc-7765.pdf, two pages, took 0.25 s); a real model takes seconds per invoice, and its API is sometimes busy for a minute. Forty requests in a row means forty waits, and three things go wrong:

  • The model's API is busy for one file. That request answers 502 and the file is read by nobody, unless someone notices and sends it again.
  • The server restarts halfway through. Whatever was in progress is simply gone; nothing says which files were done.
  • oil-mod-2291.pdf and oil-mod-2291-copy.pdf are the same PDF (it came on WhatsApp and by email). Read both and you pay the model twice for one invoice, then Rafiq gets a duplicate to reject.

What is missing is a written list of work: each file to read, its state, how many tries it has had, and when to try again. That list is a job queue, and here it is a table.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    queue.py: a job per file

    In app/jobs/queue.py write enqueue(db, files, user): for each name in order, its sha256 (file_hash(inbox_path(name))); use the Job that already has that sha256, or add Job(file, sha256, kitchen=user.kitchen, submitted_by=user.email) and flush. Commit once after the loop and return the jobs in the order of the files.

    The Check queues oil-mod-2291.pdf, fuel-vga-1184.pdf, oil-mod-2291.pdf again and oil-mod-2291-copy.pdf (the same bytes under another name): 2 jobs, and the same job back for the same PDF.

  2. Task 2

    queue.py: done, or try again later

    Write finish(db, job, invoice_id): status "done", its invoice_id, last_error None, updated_at func.now(), commit. Then fail(db, job, error): last_error (at most 500 characters), locked_by None, updated_at func.now(); with job.attempts at settings.max_job_attempts or more, status "failed"; otherwise status "queued" and run_after func.now() + timedelta(seconds=settings.retry_after_seconds * 2 ** (job.attempts - 1)). Commit.

    The Check finishes a job, then fails jobs on their first, second and third attempt: due in 30 s, in 60 s, then failed.

  3. Task 3 · interview question

    Interview: queue.py, claim with SKIP LOCKED

    Write claim(db, worker): the oldest job (order by run_after, then id) with status "queued" and run_after <= func.now(), read with .limit(1).with_for_update(skip_locked=True). None when there is none; otherwise status "running", locked_by the worker, attempts one more, updated_at func.now(), commit, and return it.

    The Check queues three jobs (one due in 2099), holds the oldest locked in a transaction of its own and runs your claim in a thread: it must not wait, and must take the second. Then it claims the first once it is free, gets None, and queues two files through POST /jobs and runs them with POST /jobs/work.

    Kunal will ask: two workers ask for the next job at the same moment. What does FOR UPDATE alone do, and what does SKIP LOCKED add?

What you will learn

  • A job queue in a table: one row per file, queued, running, done or failed.
  • One job per file: the file's hash, not its name, and a unique constraint.
  • Workers: processes that claim a job, run it and record what happened.
  • Retries with backoff: try again later, wait longer each time, give up after three.
  • func.now(): the database's clock, the same all through one transaction.
  • FOR UPDATE SKIP LOCKED: many workers, one queue, nobody waiting, no job taken twice.

More in People and volume

Build "Batch jobs" yourself

DabbaDrop's kitchens get piles of supplier invoices, typed and messy. You build the pipeline that reads them, checks every number, and sends the doubtful ones to a person. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI invoice reader project are free. This one comes with the all labs plan.