AI support copilot project · Chapter 2 of 12

Chunking

By the end of this chapter you can cut a help page into chunks: a few sentences each, never more than a set number of tokens, never across a heading, overlapping a little with the chunk before, and each with an id that stays the same every time the page is cut. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The pages
  • 3 tasks
  • Python
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

The copilot will answer a member's question from a few pieces of the help pages, never from all of them (chapter 1 measured why: 9,180 tokens did not even fit). So the pieces have to exist. Advait asks: "my dabba came 20 minutes late, do I get anything?" The answer is one paragraph of late-delivery-refunds, the ₹50 credit. The page around it is 806 tokens and covers four things: how lateness is measured, the full refund, the credit, and the days the rules do not apply. Hand the model the whole page and it reads 806 tokens for two sentences; do that for five pages and you are back where chapter 1 started.

Searching needs the same small pieces. In chapter 3 every piece becomes a list of numbers that captures what it is about, and in chapter 4 the search compares a question with each piece. A piece about one thing matches a question about that thing well. A whole page about four things matches every question a little and none of them well.

Chapter 1's code reads whole pages. Nothing in the project can cut one.

The lesson

Tour of the codebase

What is new since chapter 1:

app/corpus/pages.py            given now: Page, parse_page, load_pages, corpus_stats
app/modules/pages/router.py    changed: GET /pages/{id}/chunks
app/corpus/chunking.py         yours: Chunk, split_sections, window, chunk_page

app/lib/text.py (given since chapter 1) has split_sentences(text), and app/config.py has the two settings the route uses: chunk_tokens = 120 and chunk_overlap = 1. There is still no database: chunks are made from the files every time you ask.

Start the app with python -m app. Until you write the tasks, "The late delivery page in chunks" answers 500 and the log ends with NotImplementedError: Task 3: chunk_page is not written yet. "A page that does not exist" answers 404 already: the route looks the page up first.

{"error":{"code":"not_found","message":"There is no help page dinner-menu"}}

What a chunk is

A chunk is a few whole sentences of one section of one page, with the page's title and the section's heading kept beside them. The given Chunk dataclass holds its id, the page's id and title, the heading, its position on the page, its text and its size in tokens. It also has a property, embedding_text, the text chapter 3 will turn into numbers: the title and heading, a new line, then the sentences. A chunk that says "You get ₹50 credit" means little alone; with "Late delivery refunds: ₹50 credit when your dabba is 15 to 30 minutes late" above it, it says what it is about.

Three rules shape a chunk, one per task:

  1. It never crosses a heading. A heading is the page author telling you the topic changes. A chunk that starts in "Full refund" and ends in "₹50 credit" is about two things.
  2. It is whole sentences, at most 120 tokens. Half a sentence can change its meaning ("You get the full order amount back" without "if your dabba arrives more than 30 minutes after your slot ends").
  3. It has an id made from its page and its number. The answer cites it, so it has to mean the same text tomorrow.

Sections

A page's body is markdown. A line that starts with ## is a heading, and the lines under it, up to the next heading, are its section. The lines before the first heading (most pages open with "This page explains ...") belong to no heading, so they go under the page's title. A heading followed straight by another heading has no text, and a section with no text makes no chunks: leave it out.

You know every tool already from chapter 1: splitlines(), startswith("## "), slicing off the first three characters, strip(), and "\n".join(lines) to put a section's lines back together. The shape is a walk down the lines that keeps a list of sections and adds each line to the last one opened.

Sentences

Cutting text into sentences is harder than splitting at full stops: "12:30 pm", "₹50" and a list of bullet points all get in the way. The project gives you split_sentences:

>>> from app.lib.text import split_sentences
>>> split_sentences("Chai is served at 3 pm. It costs ₹15! Ask the rider.\n\n- Masala chai\n- Adrak chai")
['Chai is served at 3 pm.', 'It costs ₹15!', 'Ask the rider.', 'Masala chai', 'Adrak chai']
>>> split_sentences("The thali has 4 rotis. 2 of them are bajra.")
['The thali has 4 rotis.', '2 of them are bajra.']

It cuts after ., ! or ? when a capital, a digit or ₹ follows, and it makes each bullet a sentence of its own.

Packing sentences into pieces

Now the sentences of one section go into pieces of at most max_tokens. Take sentences in order while the piece still fits, and close the piece when the next sentence would push it over. Two details matter.

Count the joined piece. A piece is its sentences joined with one space, and that text is what the model reads, so count_tokens of the joined text is what must stay under the limit.

A sentence longer than the limit is a piece alone. Leaving it out would lose a sentence of the help pages, and a member's answer might be in it. So a piece always takes its first sentence, whatever its size, and only then checks whether the next one fits.

The limit decides how many chunks DabbaDrop gets. Cutting every current page with three limits and an overlap of one sentence:

 40 tokens: 360 chunks, the biggest 47, on average 28
120 tokens: 162 chunks, the biggest 120, on average 52
400 tokens: 149 chunks, the biggest 219, on average 55

At 40 tokens a chunk is one sentence or two, too little to answer most questions, and one sentence of 47 tokens became a piece alone. At 400, the limit almost stops mattering: the sections are smaller than that, so the headings do the cutting. Kunal picked 120, a section's paragraph or so; chapter 11 measures whether search finds the right chunks with it.

The overlap

Cut a section into pieces and a sentence that needs the sentence before it can end up alone in the next piece. So each piece starts by repeating the last sentence (or the last overlap sentences) of the piece before. Look at two chunks of the late delivery page, from the finished code:

late-delivery-refunds-02 107 How we decide that a dabba is late
late-delivery-refunds-03 118 How we decide that a dabba is late

Chunk 02 ends with "The delivery time is the time the rider marks your dabba as delivered in the rider app." and chunk 03 starts with that same sentence, so 03's "You can see this time on your order" still says which time.

Here is the trap. The next piece starts overlap sentences before the end of the piece before. If that piece held only one sentence (a long one, or a tight limit), "one sentence before the end" is where it started, and the next piece starts there again, and again, forever. The rule that saves you: the next start is always at least one sentence after the last start. Slices help you think about it: with s = ["A.", "B.", "C.", "D.", "E."], s[1:3] is ['B.', 'C.'], the sentences from index 1 up to, not including, index 3.

Ids that do not change

A chunk's id is its page's id, a dash, and its number on the page with two digits: late-delivery-refunds-04. The numbers run on across the sections of a page, so a page's second section does not start again at

  1. Two digits keep ids in order when they are sorted as text:
>>> f"{7:02d}", f"{12:02d}"
('07', '12')
>>> sorted(["menu-9", "menu-10", "menu-1"])
['menu-1', 'menu-10', 'menu-9']
>>> sorted(["menu-09", "menu-10", "menu-01"])
['menu-01', 'menu-09', 'menu-10']

:02d pads a number with zeros to two digits. The longest page has 10 chunks, so two digits are enough.

The bigger reason is stability. An answer will say "[late-delivery-refunds-04]". Sana will check that citation, a test will expect it, and chapter 12's answer cache notices a change to any chunk by its id and its text. Cut the same page tomorrow and -04 must be the same text. A random id changes on every run; a row number from Postgres changes whenever the table is filled again.

Your turn

  • Task 1, split_sections(page): the page as (heading, text) pairs. The trap: the text before the first heading is a section too, under the title, and an empty section is left out.
  • Task 2, window(sentences, max_tokens, overlap): sentences packed into pieces. The trap: a piece of one sentence with an overlap of one loops forever unless the next start always moves on.
  • Task 3, chunk_page(page, max_tokens, overlap): every chunk of a page, numbered across its sections. Kunal will ask: an answer will cite "late-delivery-refunds-04". Why give chunks ids made from the page and a number, and not random ones or the row numbers Postgres gives?

Send "The late delivery page in chunks" and "The pausing page in chunks" and read the chunks: where each one starts and ends, which sentence it repeats, its tokens. Then press Check. It cuts made-up pages and DabbaDrop's own. A first message looks like this:

split_sections kept the section "Empty heading", which has no text: leave out a section whose text is empty

A window that does not always move on is stopped after 5 seconds:

window(["A.", "B.", "C."], max_tokens=2, overlap=1) did not finish in 5 seconds: each next piece must start at least one sentence later than the one before

And numbers that start again in each section:

the chunks' ids are ["tiffin-packing-01", "tiffin-packing-01", "tiffin-packing-02", "tiffin-packing-01"]: the numbers run on across sections, so no id repeats

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    chunking.py: cut a page at its headings

    Write split_sections(page) in app/corpus/chunking.py. Return the page's sections as a list of (heading, text) tuples, in order. A line that starts with "## " starts a section; its heading is the rest of the line, stripped. The lines before the first "## " heading are a section under the page's title. A section's text is its lines joined with new lines and stripped; leave out a section whose text is empty.

    The Check cuts a made-up page with an intro, a heading with spaces around it and no text, and two more sections, then a page whose body starts with a heading (no title section then).

  2. Task 2

    chunking.py: pack sentences into pieces, with an overlap

    Write window(sentences, max_tokens, overlap) in app/corpus/chunking.py. Pack the sentences, in order, into pieces: a piece is sentences joined with one space, and count_tokens of the joined piece stays at or under max_tokens. A piece takes the next sentence while it still fits, and always has at least one sentence (a sentence longer than max_tokens is a piece alone). The next piece starts overlap sentences before the end of the piece before (they share those sentences), but always at least one sentence later than the piece before started. Stop after the piece that holds the last sentence; no sentences, no pieces.

    window(["A.", "B.", "C."], max_tokens=4, overlap=1) is ["A. B.", "B. C."]. The Check runs seven cases, among them an overlap as big as a piece and a sentence over the limit, and stops a window that never moves on after 5 seconds.

  3. Task 3 · interview question

    Interview: chunking.py, chunks with stable ids

    Write chunk_page(page, max_tokens=120, overlap=1) in app/corpus/chunking.py. For each section of the page (split_sections), split its text into sentences (split_sentences, given) and window them; every piece becomes a Chunk: id is the page's id, a dash and the chunk's number on the page with two digits ("jain-food-03"); page_id and title are the page's; heading is the section's; position is the number (1, 2, 3 ... running on across every section of the page); text is the piece; tokens is count_tokens(text).

    The Check cuts a made-up page with max_tokens 8 (its own split_sections and window, so this task stands alone), then late-delivery-refunds with 120 and 1, then calls GET /pages/late-delivery-refunds/chunks.

    Kunal will ask: an answer will cite "late-delivery-refunds-04". Why give chunks ids made from the page and a number, and not random ones or the row numbers Postgres gives?

What you will learn

  • Why chunks: the right size for searching and for the model's window, and why a chunk never crosses a heading.
  • Sections: a page's ## headings, and the text before the first one.
  • Sentences: the given split_sentences, and why a chunk is whole sentences.
  • Windowing: packing sentences up to a token limit, the overlap between neighbours, and the loop that must always move on.
  • Stable ids: late-delivery-refunds-04, and why a citation needs an id that does not change.

More in The pages

Build "Chunking" yourself

DabbaDrop answers the same customer questions all day. You build the copilot that answers them from its own help pages, with sources, and says so when it does not know. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.