RAG: search that answers · Chapter 10 of 37
Cutting into chunks
By the end of this chapter you can cut documents into chunks by tokens, with overlap and at sentence boundaries, and say what a chunk size costs and buys, with numbers. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Ingestion
- 4 tasks
- In the all labs plan
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.
Why you need this
Kabir asked the search, "What does a UPI payment above ₹2,000 cost on Growth?" With one vector per document, the answer is somewhere in a 575-token article (the median), and the five best documents together are 3,069 tokens: more than a third of the lab model's 8,192-token window, paid for on every question, to answer with one table row.
Until now the search in this lab used chunks that were made for you. To index new documents you have to cut them yourself, and the obvious cut (every so many characters) breaks words, numbers and table rows. You need pieces small enough that each one is about one thing, and whole enough that each one still makes sense on its own.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
tokenChunks: every size tokens, with overlap
Encode text with enc.encode, cut the list of token ids into windows of size tokens, and decode each window with enc.decode. The first window starts at token 0 and each next one starts size - overlap tokens later (overlap is 0 when it is not given). The window that reaches the end of the text is the last one, so there is never an extra window made only of overlap. An empty text has no chunks.
For SETTLE = 'Settlements reach your bank by 11 am on the next working day. A holiday moves them to the day after.' (24 tokens), tokenChunks(SETTLE, 8) is ['Settlements reach your bank by 11', ' am on the next working day. A', ' holiday moves them to the day after.'], tokenChunks(SETTLE, 8, 3) has 5 windows starting at tokens 0, 5, 10, 15 and 20, and tokenChunks(SETTLE, 12, 6) has 3 (at 0, 6 and 12).
With overlap equal to size the window would never move: throw a RangeError when size is under 1, or overlap is under 0 or not less than size.
- Task 2
sentenceChunks: whole sentences only
Split text into sentences with new Intl.Segmenter('en', { granularity: 'sentence' }). A sentence is a segment with the whitespace around it removed; skip segments that are only whitespace. A table row ends with a new line, so it is a sentence too.
A chunk is the original text from the first character of its first sentence to the last character of its last sentence (a slice of text, so line breaks inside stay), and its size is countTokens of that slice. Start a chunk with the next sentence, then add the following sentences one by one while the chunk stays at most size tokens. A sentence longer than size is a chunk on its own. Only whitespace gives no chunks.
sentenceChunks('UPI is free under ₹2,000. Cards cost 1.9%. Wallets cost 1.9%. Net banking costs ₹12. GST is 18%. RuPay debit is free.', 20) is ['UPI is free under ₹2,000. Cards cost 1.9%.', 'Wallets cost 1.9%. Net banking costs ₹12. GST is 18%.', 'RuPay debit is free.']. On real documents your chunks must be exactly the ones these rules give.
- Task 3
sentenceChunks: overlap in whole sentences
Give sentenceChunks its third parameter, overlap (0 when it is not given). When a chunk is closed and sentences are left, the next chunk starts with the last sentences of the closed one: go back one sentence at a time while the slice from that sentence to the end of the closed chunk is at most overlap tokens, but never back to the closed chunk's first sentence. If the carried sentences and the next new sentence together are over size, carry nothing.
With the same six sentences, sentenceChunks(text, 20, 12) is ['UPI is free under ₹2,000. Cards cost 1.9%.', 'Cards cost 1.9%. Wallets cost 1.9%.', 'Wallets cost 1.9%. Net banking costs ₹12. GST is 18%.', 'Net banking costs ₹12. GST is 18%. RuPay debit is free.'], and sentenceChunks(text, 12, 12) carries nothing until 'Net banking costs ₹12. GST is 18%.', then starts 'GST is 18%. RuPay debit is free.'.
- Task 4 · interview question
Interview: indexCost, what overlap costs
Cut every text with tokenChunks(text, size, overlap) and return { chunks, tokens, ratio }: chunks is how many chunks there are in all, tokens the sum of countTokens of every chunk (what you would embed and store), and ratio is tokens divided by the sum of countTokens of the texts themselves.
documents() gives the body of all 145 documents, 98,068 tokens. indexCost(documents(), 200, 0) has ratio 1, and indexCost(documents(), 200, 100) about 1.78: half a chunk of overlap stores almost twice the text. Arjun will ask: you set overlap to half the chunk to be safe; how much bigger is the index?
What you will learn
- Token windows: cutting every N tokens, the way most libraries do by default.
- Overlap: repeating the end of one chunk at the start of the next, and what it costs.
- Sentence boundaries: never cutting a sentence or a table row in half.
- Chunk size: what small and large chunks do to recall and to the bill, measured.
More in Ingestion
Build "Cutting into chunks" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.