RAG in Python · Chapter 10 of 37
Cutting into chunks
By the end of this chapter you can cut documents into chunks by tokens, with overlap and at sentence boundaries, and say what a chunk size costs and buys, with numbers. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Ingestion
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.
Why you need this
Kabir asked the search, "What does a UPI payment above ₹2,000 cost on Growth?" With one vector per document, the answer is somewhere in a 575-token article (the median), and the five best documents together are about 3,090 tokens: more than a third of the lab model's 8,192-token window, paid for on every question, to answer with one table row.
Until now the searches in this lab used chunks that were made for you. To index new documents you have to cut them yourself, and the obvious cut (every so many characters) breaks words, numbers and table rows. You need pieces small enough that each one is about one thing, and whole enough that each one still makes sense on its own.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
chunking.py: token_chunks, every size tokens, with overlap
Write token_chunks(text, size, overlap=0) in chunking.py: encode text with enc.encode, cut the list of token ids into windows of size tokens, and decode each window with enc.decode. The first window starts at token 0 and each next one starts size - overlap tokens later. The window that reaches the end of the text is the last one, so there is never an extra window made only of overlap. An empty text has no chunks.
For SETTLE = "Settlements reach your bank by 11 am on the next working day. A holiday moves them to the day after." (24 tokens), token_chunks(SETTLE, 8) is ["Settlements reach your bank by 11", " am on the next working day. A", " holiday moves them to the day after."], token_chunks(SETTLE, 8, 3) has 5 windows starting at tokens 0, 5, 10, 15 and 20, and token_chunks(SETTLE, 12, 6) has 3 (at 0, 6 and 12).
With overlap equal to size the window would never move: raise ValueError when size is under 1, or overlap is under 0 or not less than size.
- Task 2
chunking.py: sentence_chunks, whole sentences only
sentence_spans(text), given, returns (start, end) for every sentence, so text[start:end] is the sentence without the whitespace around it; a table row ends with a new line, so it is a sentence too. Write sentence_chunks(text, size): a chunk is the original text from the start of its first sentence to the end of its last (a slice of text, so line breaks inside stay), and its size is count_tokens of that slice. Start a chunk with the next sentence, then add the following sentences one by one while the chunk stays at most size tokens. A sentence longer than size is a chunk on its own. Only whitespace gives no chunks.
sentence_chunks("UPI is free under ₹2,000. Cards cost 1.9%. Wallets cost 1.9%. Net banking costs ₹12. GST is 18%. RuPay debit is free.", 20) is ["UPI is free under ₹2,000. Cards cost 1.9%.", "Wallets cost 1.9%. Net banking costs ₹12. GST is 18%.", "RuPay debit is free."]. On three real documents your chunks must be exactly the ones these rules give.
- Task 3
chunking.py: sentence_chunks, overlap in whole sentences
Make sentence_chunks's third parameter, overlap, work (0 means none). When a chunk is closed and sentences are left, the next chunk starts with the last sentences of the closed one: go back one sentence at a time while the slice from that sentence to the end of the closed chunk is at most overlap tokens, but never back to the closed chunk's first sentence. If the carried sentences and the next new sentence together are over size, carry nothing.
With the same six sentences, sentence_chunks(text, 20, 12) is ["UPI is free under ₹2,000. Cards cost 1.9%.", "Cards cost 1.9%. Wallets cost 1.9%.", "Wallets cost 1.9%. Net banking costs ₹12. GST is 18%.", "Net banking costs ₹12. GST is 18%. RuPay debit is free."], and sentence_chunks(text, 12, 12) carries nothing until "Net banking costs ₹12. GST is 18%.", then starts "GST is 18%. RuPay debit is free.".
- Task 4 · interview question
Interview: chunking.py, index_cost
Write index_cost(texts, size, overlap): cut every text with token_chunks(text, size, overlap) and return Cost(chunks=..., tokens=..., ratio=...): chunks is how many chunks there are in all, tokens the sum of count_tokens of every chunk (what you would embed and store), and ratio is tokens divided by the sum of count_tokens of the texts themselves.
document_bodies() gives the body of all 145 documents, 98,068 tokens. index_cost(document_bodies(), 200, 0) has ratio 1.0, and index_cost(document_bodies(), 200, 100) about 1.78: half a chunk of overlap stores almost twice the text. Arjun will ask: you set overlap to half the chunk to be safe; how much bigger is the index?
What you will learn
- Token windows: cutting every N tokens, the way most libraries do by default.
- Overlap: repeating the end of one chunk at the start of the next, and what it costs.
- Sentence boundaries: never cutting a sentence or a table row in half.
- Chunk size: what small and large chunks do to recall and to the bill, measured.
More in Ingestion
Build "Cutting into chunks" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.