RAG in Python · Chapter 12 of 37

Every chunk knows where it came from

By the end of this chapter you can give every chunk an id that survives a rebuild and a hash of its text, carry its document's metadata on it, and refuse a document whose metadata is wrong. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ingestion
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.

Why you need this

Neha, head of compliance, opens last week's audit log. An answer to a merchant cited chunk-412, a paragraph about the Growth plan's ₹999 fee. Since then a new article was added and the index rebuilt, and chunk-412 is now a paragraph about the Starter plan. Nobody can prove what the merchant was shown.

Kabir has a second problem: someone edited the goodwill credits runbook and deleted its audience: internal line by mistake. The indexer filled in merchant, and the page that says support may give up to ₹500 without approval became searchable by merchants.

Your chunker from chapter 11 returns text and a heading. That is enough to search, but not to cite, to filter by audience, to prefer the newest policy, or to update one document without rebuilding everything. Each of those needs facts that travel with the chunk.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    chunks.py: chunk_keys, ids that survive a rebuild

    Neha needs a citation to name the same text tomorrow. Write chunk_keys(doc_id, texts) in chunks.py: one ChunkKey per text, in order. Its id is the document id, a # and the position from 0: chunk_keys("help-refund-timelines", texts) gives help-refund-timelines#0, help-refund-timelines#1 and so on. Its hash is the first 16 hex characters of the SHA-256 of the text, encoded as UTF-8, from hashlib.

    The same document chunked twice must get exactly the same keys, and changing one text changes only that chunk's hash, never an id. No uuid4 and no counter across the corpus: they break every citation on the next rebuild.

  2. Task 2

    chunks.py: DocMeta, metadata you can trust

    Kabir's draft of the Pydantic model DocMeta accepts anything. Make it strict. id, title and section are text and never empty. audience is "merchant" or "internal" and is required: the goodwill runbook with its audience line deleted must be rejected with an error on audience, never become a merchant document. type is one of help, developer, policy, runbook, community, changelog. updated is a date; effective_from and effective_to are dates or None ("20/04/2026" and "1 March 2026" are rejected). version is a whole number above 0 or None ("two" and "0" are rejected).

    In a model validator that raises ValueError: a policy must have a version and an effective_from, and effective_to must not be before effective_from. All 145 of Paisa Setu's documents must still pass, with version as an int and the dates as datetime.date (policy-settlement-policy has version 3).

  3. Task 3

    chunks.py: chunk_document, every chunk carries its metadata

    Write chunk_document(path, raw) in chunks.py. Read the front matter with read_front_matter and check it with DocMeta. When it is not valid, raise ValueError whose message has the path and the failing field: the goodwill runbook with no audience gives a message with runbooks/runbook-goodwill-credits.md and audience. When the front matter id is not the file's name (help/help-refund-time.md holding help-refund-timelines), raise ValueError with the path and the word id.

    Then cut the body with split_sections and return one IndexedChunk per section: id and hash from chunk_keys, doc_id, title, type, audience, section, updated, version, effective_from, effective_to, heading (the heading path joined with " > ", "" for the part before the first heading), text and tokens. Over all 145 documents this is the 866 chunks of the lab's reference index, field for field.

  4. Task 4 · interview question

    Interview: chunks.py, check_citation, does the citation still hold?

    An answer stored its citation as a ChunkKey (id and hash). Write check_citation(cited, chunks): chunks are today's ChunkRefs (id, doc_id, hash). Return a CitationStatus. ok: a chunk has the cited id and the same hash, now is that id. moved: a chunk of the same document (the part of the cited id before #) has the hash under another id, now is that id. changed: the id exists but its document has no chunk with the hash, now is None. gone: neither, now is None.

    After a section is added above it, help-refund-timelines#3 is moved with now help-refund-timelines#4. A copy of the cited text in another document does not count: that citation is changed (or gone, when its document was deleted).

    Arjun will ask why you do not give every chunk a random UUID.

What you will learn

  • Stable ids: why random ids and corpus-wide counters break citations, and what to build ids from instead.
  • Content hashes: a fingerprint of a chunk's text, to tell when it changed.
  • Metadata on every chunk: audience, type, version, effective dates, updated, and the heading path.
  • Validation with Pydantic: reject a document with a missing or wrong field instead of guessing.

More in Ingestion

Build "Every chunk knows where it came from" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.