RAG in Python · Chapter 14 of 37

Re-index only what changed

By the end of this chapter you can keep a manifest of what is indexed, work out what changed between two versions of the documents, apply only those changes to Postgres in one transaction, and say what a deletion does to chunks and to answers. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ingestion
  • 5 tasks
  • Python
  • In the all labs plan

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.

Why you need this

Paisa Setu's index is rebuilt from scratch every night: every document chunked, every chunk embedded, every row rewritten, even when three files changed. Kabir has a sharper problem. Last week he deleted the migrated UPI guide (the copy from chapter 13), and the assistant still cites it. Someone changed the nightly job to "only upsert what is new" to save time, and an upsert never deletes anything.

Neha asks the other half: the answers that cited that guide are in the audit log. Are they still true?

Chunking and embedding you can already do. What is missing is memory: what was indexed last time, so the next run knows what to add, what to replace and what to remove.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    reindex.py: manifest_of, a manifest of what you have

    Write manifest_of(docs) in reindex.py: a dict from every document's path to the SHA-256 of its whole file (raw, encoded as UTF-8) as 64 hex characters. manifest_of(docs)["help/help-refund-timelines.md"] is the hash of that file.

    Hash the whole file, front matter included: a document whose audience changed from merchant to internal must get a different hash, or its chunks keep the old audience. The same documents always give the same manifest.

  2. Task 2

    reindex.py: plan_changes, what changed between two manifests

    Write plan_changes(before, after) in reindex.py for the manifest from last time (before) and the new one (after). Return a Plan: inserts are the paths only in after, updates the paths in both with a different hash, deletes the paths only in before, and unchanged is how many paths are in both with the same hash. Each list is sorted.

    before {"a": "1", "b": "2", "c": "3"} and after {"b": "2", "c": "9", "d": "4"} give Plan(inserts=["d"], updates=["c"], deletes=["a"], unchanged=1). From an empty manifest every document is an insert; the same manifest twice gives nothing to do.

  3. Task 3

    reindex.py: apply_plan, the plan in one transaction

    Write apply_plan(conn, plan, docs) in reindex.py. First chunk every inserted and updated document of the plan (from docs) with chunk_rows, and embed all their contents with embed. Then, inside with conn.transaction(): delete every chunk of the deleted and updated documents from CHUNKS by path, insert each new chunk row with its embedding, delete the deleted documents from MANIFEST, and insert or update the manifest row (path, hash) of each inserted or updated document. An error anywhere must undo all of it and still reach the caller: a plan that fails halfway changes nothing.

    Return Applied(removed, inserted): the ids of the chunks you deleted and of those you inserted, each sorted. After version 2, help-refund-timelines has 6 chunks, not 7: deleting by path first leaves no ghost. Chunks of unchanged documents are never touched. Use the constants CHUNKS and MANIFEST for the table names, and parameters for every value.

  4. Task 4

    reindex.py: reindex, a second run that does no work

    Write reindex(conn, docs) in reindex.py: read what was indexed from MANIFEST (path, hash) into a dict, make the manifest of docs, plan the changes, and apply them. Return a Report: the plan's four fields with applied's removed and inserted.

    On the 18 documents of version 1 the first run is 18 inserts. The same documents again return no inserts, updates or deletes, 18 unchanged, and empty removed and inserted lists: nothing is embedded and no chunk or manifest row is written (the check compares every row's xmin). Version 2 then embeds only the chunks of its three new or changed documents.

  5. Task 5 · interview question

    Interview: reindex.py, mark_stale, answers that cited a removed chunk

    ANSWERS keeps every answer with the ids of the chunks it cited (cited, a text array). Write mark_stale(conn, removed) in reindex.py: set stale to true on every answer that cites at least one of the removed chunk ids and is not stale yet, and return those answers' ids, sorted. Never delete an answer: Neha's audit needs every one.

    With the migrated guide's six chunks and help-refund-timelines#3 removed, an answer citing help-upi-payments-guide#0 and one citing help-refund-timelines#3 are marked; one already stale is not returned again, and the rest stay as they were. A second call marks nothing, and so does an empty list.

    Arjun will ask: a document is deleted, what happens to its chunks and to the answers that cited them?

What you will learn

  • A manifest: every document's path and a hash of its file.
  • A plan: inserts, updates and deletes, from two manifests.
  • Applying it safely: one transaction, old chunks deleted first, no ghosts, and a second run that does nothing.
  • Deletes downstream: what happens to the answers that cited a removed chunk.

More in Ingestion

Build "Re-index only what changed" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.