RAG: search that answers · Chapter 12 of 37
Every chunk knows where it came from
By the end of this chapter you can give every chunk an id that survives a rebuild and a hash of its text, carry its document's metadata on it, and refuse a document whose metadata is wrong. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Ingestion
- 4 tasks
- In the all labs plan
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.
Why you need this
Neha, head of compliance, opens last week's audit log. An answer to a merchant cited chunk-412, a paragraph about the Growth plan's ₹999 fee. Since then a new article was added and the index rebuilt, and chunk-412 is now a paragraph about the Starter plan. Nobody can prove what the merchant was shown.
Kabir has a second problem: someone edited the goodwill credits runbook and deleted its audience: internal line by mistake. The chunker filled in merchant, and the page that says support may give up to ₹500 without approval is now searchable by merchants.
So far your chunker returns text and a heading. That is enough to search, but not enough to cite, to filter by audience, to prefer the newest policy, or to update one document without rebuilding everything. Every one of those needs facts that travel with the chunk.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
chunkKeys: ids that survive a rebuild
Return one ChunkKey per text of a document, in order. The id is the document id, a hash sign and the position from 0: chunkKeys('help-refund-timelines', texts) gives help-refund-timelines#0, help-refund-timelines#1, and so on. The hash is the first 16 hex characters of the SHA-256 of the text, from createHash in node:crypto.
The same document chunked twice must get exactly the same keys, and changing one text changes only that chunk's hash. No random UUIDs and no counter across the corpus: they break every citation on the next rebuild.
- Task 2
DocMeta: metadata you can trust
Make the zod schema DocMeta strict. id, title and section are non-empty text. audience is 'merchant' or 'internal' and is required: a runbook with no audience line must be rejected with an issue on audience, never turned into a merchant document. type is one of help, developer, policy, runbook, community, changelog. updated, effective_from and effective_to are ISO dates like 2026-04-14 ('20/04/2026' and '1 March 2026' are rejected).
version comes from the front matter as text: turn it into a whole number above 0 ('two' and '0' are rejected). A missing version, effective_from or effective_to becomes null. A policy must have a version and an effective_from, and effective_to must not be before effective_from.
All 145 of Paisa Setu's documents must still pass, with version as a number (policy-settlement-policy has version 3).
- Task 3
chunkDocument: every chunk carries its metadata
chunkDocument(path, raw) reads the front matter with readFrontMatter and checks it with DocMeta. When it is not valid, throw an Error whose message contains the path and the failing field: the goodwill runbook with no audience gives a message with runbooks/runbook-goodwill-credits.md and audience. When the front matter id is not the file's name (help/help-refund-time.md holding help-refund-timelines), throw an Error with the path and the word id.
Then cut the body with splitSections and return one IndexedChunk per section: id and hash from chunkKeys, doc_id, title, type, audience, section, updated, version, effective_from, effective_to, heading (the heading path joined with ' > ', '' for the intro), text and tokens.
Over all 145 documents this gives the 866 chunks of the lab's reference index, field for field.
- Task 4 · interview question
Interview: checkCitation, does the citation still hold?
An answer stored a citation as { id, hash }. Given today's chunks (id, doc_id, hash), return { status, now }. ok: a chunk has that id and the same hash, now is the id. moved: a chunk of the same document (the part of the id before #) has the hash under another id, now is that id. changed: the id exists but the document has no chunk with the hash, now is null. gone: neither, now is null.
After a section is added above it, help-refund-timelines#3 is moved with now help-refund-timelines#4. A copy of the cited text in another document does not count: that citation is changed (or gone, when its document was deleted).
Arjun will ask why you do not give every chunk a random UUID.
What you will learn
- Stable ids: why random ids and corpus-wide counters break citations, and what to build ids from instead.
- Content hashes: a fingerprint of a chunk's text, to tell when it changed.
- Metadata on every chunk: audience, type, section, version, effective dates, updated, and the heading path.
- Validation with zod: reject a document with a missing or wrong field instead of guessing.
More in Ingestion
Build "Every chunk knows where it came from" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.