RAG: search that answers · Chapter 18 of 37
One command to index Paisa Setu
By the end of this chapter you can run the whole ingestion pipeline (load, chunk, embed, store) as one command that is safe to run again and again, does only the work that changed, and says what it did. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Vectors in Postgres
- 4 tasks
- In the all labs plan
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.
Why you need this
Every piece exists: chapters 9 to 14 load and chunk documents, chapter 15 stores vectors, chapter 16 indexes them. Kabir's team edits help articles every week, and today someone runs a handful of scripts in the right order, by hand. Last month a script died halfway, was started again, and inserted half the chunks twice. A removed article kept answering merchants for a week because nobody deleted its chunks.
Arjun wants one command that a scheduler runs every night: node exercise.ts. Run it twice and the second run must change nothing, embed nothing, and say so. What you know so far does each step once; this chapter makes the steps safe to repeat.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
embedAll: embeddings in batches the server accepts
Embed texts with ai.embeddings.create and the model lab-embed, batchSize texts per request (the lab's server refuses more than 256), and return one vector per text, in the order of texts: each item of a response has an index, the position of its text in that request. After every batch, call onProgress (when it is given) with the number of texts embedded so far and the total.
600 texts with batchSize 256 are 3 requests of 256, 256 and 88 texts, and onProgress is called with (256, 600), (512, 600), (600, 600). 250 texts with batchSize 100 are 3 requests. No texts: send no request and return [].
- Task 2
upsertChunks: insert new, update changed, skip the same
Write rows into TABLE (pipeline_chunks) with batchSize rows per INSERT, every value a parameter, in the order of COLUMNS: content is the chunk's text and embedding goes through pgvector.toSql. Add ON CONFLICT (id) DO UPDATE SET every other column = EXCLUDED.column, WHERE TABLE.hash IS DISTINCT FROM EXCLUDED.hash, and RETURNING (xmax = 0) AS inserted.
Return { inserted, updated }: a returned row with inserted true was new, false was updated; a skipped row returns nothing. 866 new rows give { inserted: 866, updated: 0 } in 9 statements; the same rows again give { inserted: 0, updated: 0 } and no row is written again.
- Task 3 · interview question
Interview: indexDocuments, a second run does nothing
Chunk every document with chunkDocument, read the id and hash of every row already in TABLE, and keep the chunks whose id is not stored or whose chunkHash differs from the stored hash. Embed only those with embedAll (pass onProgress on), upsert them (hash is chunkHash of the chunk), then delete every row whose id is not among this run's chunks. Return the Summary: documents and chunks counted, embedded (the chunks you embedded), inserted and updated (from upsertChunks), deleted (rows deleted) and unchanged (the chunks you did not embed).
The first run over all documents is { documents: 145, chunks: 866, embedded: 866, inserted: 866, updated: 0, deleted: 0, unchanged: 0 }; the second embeds nothing (no request at all) and reports 866 unchanged. A removed document's chunks are deleted. After a crash that stored only the first 500 chunks, a rerun embeds and inserts the other 366.
Arjun will ask: the job dies at chunk 500. What happens when you run it again?
- Task 4
exercise.ts: one command with progress and a summary
summaryLine(summary, ms) returns, with ms rounded: 145 documents, 866 chunks: 866 embedded, 866 inserted, 0 updated, 0 deleted, 0 unchanged (650 ms).
Then replace the throw in the block at the end with the command: connect, createTable, indexDocuments over loadDocuments() printing a line like embedded 256/866 after each batch (console.log), then print summaryLine with the milliseconds since the start. The Check runs node exercise.ts twice: the first run prints 4 progress lines and the summary; the second prints 0 embedded and 866 unchanged. If anything fails (the Check uses a database that does not exist), print the error with console.error, set process.exitCode to 1, print no summary, and still close the connection.
What you will learn
- Batches the server accepts: embeddings in requests of at most 256 texts, with progress.
- Upserts: INSERT ... ON CONFLICT DO UPDATE, and skipping rows that did not change.
- Content hashes: deciding what changed without re-embedding everything.
- A command that can fail: a summary when it works, an exit code when it does not.
More in Vectors in Postgres
Build "One command to index Paisa Setu" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.