RAG in Python · Chapter 18 of 37
One command to index Paisa Setu
By the end of this chapter you can run the whole ingestion pipeline (load, chunk, embed, store) as one command that is safe to run again and again, does only the work that changed, and says what it did. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Vectors in Postgres
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.
Why you need this
Every piece exists: chapters 9 to 14 load and chunk documents, chapter 15 stores vectors, chapter 16 indexes them. Kabir's team edits help articles every week, and today someone runs a handful of scripts in the right order, by hand. Last month a script died halfway, was started again, and inserted half the chunks twice. A retired article kept answering merchants for a week because nobody deleted its chunks.
Arjun wants one command that a scheduler runs every night: python3 index.py. Run it twice and the second run must change nothing, embed nothing, and say so. What you know so far does each step once; this chapter makes the steps safe to repeat.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
pipeline.py: embed_all, batches the server accepts
Write the async embed_all(texts, batch_size=256, concurrency=4, on_progress=None): embed texts with lab-embed, batch_size texts per request (the lab's server refuses more than 256), with at most concurrency requests in flight at once, and return one vector per text, in the order of texts. Make the client with new_client() inside the function. Each item of a response has an index, the position of its text in that request. After every batch, call on_progress (when it is given) with the number of texts embedded so far and the total.
600 texts with batch_size 256 are 3 requests of 256, 256 and 88 texts, and on_progress is called 3 times, the last with (600, 600). 250 texts with batch_size 100 are 3 requests. No texts: send no request and return []. The check also gives embed_all a slow stand-in client that answers in reverse order, and counts the requests in flight: more than one, never more than concurrency.
- Task 2
pipeline.py: upsert_chunks, insert new, update changed, skip the same
Write upsert_chunks(conn, rows, table, batch_size=100): write the rows into the table, batch_size rows per INSERT, every value a parameter, in the order of COLUMNS (content is the row's text, the embedding goes in as a numpy array). Add ON CONFLICT (id) DO UPDATE SET every other column = EXCLUDED.column, WHERE table.hash IS DISTINCT FROM EXCLUDED.hash, and RETURNING (xmax = 0).
Return Upserted(inserted, updated): a returned row with true was new, false was updated; a skipped row returns nothing. 866 new rows give Upserted(inserted=866, updated=0); the same rows again give Upserted(inserted=0, updated=0), and the check sees that no row was written again. 3 changed rows, 2 new and 3 the same give inserted 2, updated 3.
- Task 3 · interview question
Interview: pipeline.py, index_documents
Write index_documents(conn, docs, table, on_progress=None): chunk every document with chunk_document, read the id and hash of every row already in the table, and keep the chunks whose id is not stored or whose chunk_hash differs from the stored hash. Embed only those with embed_all (asyncio.run it, passing on_progress on), upsert them as Rows (hash is chunk_hash of the chunk), then delete every row whose id is not among this run's chunks. Return the Summary: documents and chunks counted, embedded (the chunks you embedded), inserted and updated (from upsert_chunks), deleted (rows deleted) and unchanged (the chunks you did not embed).
The first run over all documents is Summary(documents=145, chunks=866, embedded=866, inserted=866, updated=0, deleted=0, unchanged=0); the second sends no embeddings request at all and reports 866 unchanged. A removed document's chunks are deleted; a document whose updated date changed is embedded and updated again. After a crash that stored only the first 500 chunks, a rerun embeds and inserts the other 366.
Arjun will ask: the job dies at chunk 500. What happens when you run it again?
- Task 4
index.py: one command with progress and a summary
Write summary_line(summary, ms): 145 documents, 866 chunks: 866 embedded, 866 inserted, 0 updated, 0 deleted, 0 unchanged (650 ms), with ms rounded to a whole number. Then finish main(): connect to args.database, create_table, index_documents over load_documents() into args.table, printing a line like embedded 256/866 after each batch, then print summary_line with the milliseconds since the start.
The check runs python3 index.py --table <its table> twice: the first run prints 4 progress lines and the summary; the second prints no progress line, 0 embedded and 866 unchanged. With --database set to one that does not exist, it must print the error to stderr, print no summary, and exit with a code that is not 0.
What you will learn
- Batches the server accepts, a few in flight at once with asyncio.
- Upserts: INSERT ... ON CONFLICT DO UPDATE, skipping rows that did not change.
- Content hashes: deciding what changed without re-embedding everything, and deleting what is gone.
- A command that can fail: a summary when it works, an exit code when it does not.
More in Vectors in Postgres
Build "One command to index Paisa Setu" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.