AI support copilot project (TypeScript) · Chapter 2 of 12
Chunking
By the end of this chapter you can cut a help page into chunks: a few whole sentences each, never more than a set number of tokens, never across a heading, sharing a sentence with the chunk before when a section is cut, and each with an id that stays the same every time the page is cut. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The pages
- 3 tasks
- TypeScript
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
The copilot will answer a member's question from a few pieces of the help pages, never from all of them (chapter 1 measured why: 8,301 tokens did not even fit). So the pieces have to exist. Pranav writes on WhatsApp: "my show was cancelled, do I have to ask for my money back?" The answer is two sentences of cancelled-shows: the refund is full, and it starts on its own. The page around them is 307 tokens and covers five things: what you get, where the money goes, moving to another showtime, why shows are cancelled, and rain. Hand the model the whole page and it reads 307 tokens for two sentences; do that for five pages and you are back where chapter 1 started.
Searching needs the same small pieces. In chapter 3 every piece becomes a list of numbers that captures what it is about, and in chapter 4 the search compares a question with each piece. A piece about one thing matches a question about that thing well. A whole page about five things matches every question a little and none of them well.
Chapter 1's code reads whole pages. Nothing in the project can cut one.
The lesson
Tour of the codebase
What is new since chapter 1:
src/corpus/pages.ts given now: Page, parsePage, loadPages, corpusStats
src/modules/pages/pages.routes.ts changed: GET /pages/:id/chunks
src/corpus/chunking.ts yours: Chunk, splitSections, slidingWindow, chunkPage
src/lib/text.ts (given since chapter 1) has splitSentences(text), and src/config.ts has the two settings the new route uses: chunkTokens: 120 and chunkOverlap: 1. There is still no database: the route cuts the page from its file every time you ask.
Start the app with npm run dev. Until you write the tasks, "The cancelled shows page in chunks" answers 500, and the log says why (this run used another folder and port):
Error: Task 3: chunkPage is not written yet
at chunkPage (/workspace/chapters/w-a/src/corpus/chunking.ts:92:9)
at <anonymous> (/workspace/chapters/w-a/src/modules/pages/pages.routes.ts:32:12)
...
"A page that does not exist" answers 404 already, because the route looks the page up before it calls your code:
{"error":{"code":"not_found","message":"There is no help page interval-menu"}}
What a chunk is
A chunk is a few whole sentences of one section of one page, with the page's title and the section's heading kept beside them. The given Chunk type holds its id, the page's id and title, the heading, its position on the page, its text and its size in tokens. The given embeddingText(chunk) is the text chapter 3 will turn into numbers: the title and the heading, a new line, then the sentences. A chunk that says "You do not need to ask for it" means little alone. With "Cancelled shows: What you get when a show is cancelled" above it, it says what it is about.
Three rules shape a chunk, one per task:
- It never crosses a heading. A heading is the page's author telling you the topic changes. A chunk that starts in "Where the money goes" and ends in "Why shows are cancelled" is about two things.
- It is whole sentences, at most 120 tokens. Half a sentence can change its meaning: "you get the same full refund" without "If a show is stopped for safety after it has started" promises too much.
- It has an id made from its page and its number. An answer cites it, so it has to mean the same text tomorrow.
Sections
A page's body is markdown. A line that starts with ## is a heading, and the lines under it, up to the next heading, are its section. The lines before the first heading (most of Parda's pages open with "This page explains ...") belong to no heading, so they go under the page's title. A heading followed straight by another heading has no text, and a section with no text makes no chunks: leave it out.
You know most of the tools from chapter 1. Three more checks in node show the details:
> "## Seats".startsWith("## ")
true
> "### Seats".startsWith("## ")
false
> "## Seats ".slice(3).trim()
'Seats'
> ["a", "", "b", ""].join("\n").trim()
'a\n\nb'
A smaller heading, ### , is not a new section: its fourth character is a #, not a space. slice(3) drops the two hashes and the space, and trim takes the extra spaces people type. Putting a section's lines back together with join("\n") and trimming keeps its blank lines inside but not at its ends. The shape of the task is a walk down the lines that keeps a list of sections and adds each line to the last one opened.
Sentences
Cutting text into sentences is harder than splitting at full stops: "6:30 pm", "₹40" and a list of bullet points all get in the way. The project gives you splitSentences:
> splitSentences("Doors open at 6:30 pm. Samosas are ₹40! Ask at the counter.\n\n- Masala chai\n- Cutting chai")
[
'Doors open at 6:30 pm.',
'Samosas are ₹40!',
'Ask at the counter.',
'Masala chai',
'Cutting chai'
]
> splitSentences("Screen 2 has 120 seats. 40 of them are recliners.")
[ 'Screen 2 has 120 seats.', '40 of them are recliners.' ]
It cuts after ., ! or ? when a space and then a capital, a digit or ₹ follow, and it makes each bullet a sentence of its own.
Packing sentences into pieces
Now the sentences of one section go into pieces of at most maxTokens. Take sentences in order while the piece still fits, and close the piece when the next sentence would push it over. Two details matter.
Count the joined piece. A piece is its sentences joined with one space, and that text is what the model reads, so countTokens of the joined text is what must stay at or under the limit. Checking each sentence on its own says nothing about the piece:
> const three = ["Doors open at 6:30 pm.", "Samosas are ₹40!", "Ask at the counter."]
undefined
> three.map((s) => countTokens(s))
[ 9, 7, 5 ]
> countTokens(three.join(" "))
21
With a limit of 16, each sentence fits on its own, and the three together do not.
A sentence longer than the limit is a piece alone. Leaving it out would lose a sentence of the help pages, and a member's answer might be in it. So a piece always takes its first sentence, whatever its size, and only then checks whether the next one fits.
Here is what the limit does to Parda's 37 current pages, cut with the finished code and an overlap of one sentence:
40 tokens: 288 chunks, the biggest 68, on average 28
120 tokens: 179 chunks, the biggest 120, on average 40
400 tokens: 179 chunks, the biggest 120, on average 40
At 40 tokens a chunk is one sentence or two, too little to answer most questions, and the longest sentence of the pages (68 tokens) became a piece alone. At 120 and at 400 the count is the same, 179, one chunk per section: Parda's pages are written in short sections, and the biggest, the examples of cancel-a-booking, is exactly 120 tokens. So with Nikhil's 120, today the headings do the cutting. The window is there for the day a section grows, and the pages Asha owns about venues will.
The overlap
When a section is cut into several pieces, a sentence that leans on the one before it can end up alone. So each piece starts by repeating the last sentence (or the last overlap sentences) of the piece before. To see it on Parda's pages, cut payment-failed with a limit of 60 tokens. Its last section, "Why it happens", becomes three chunks:
payment-failed-05 54 "A UPI payment can take up to 2 minutes to reach Parda when the bank's servers are busy, which happens most on Friday evenings and in Diwali week. If the payment arrives after your 10 minutes are over, your seats may already be gone."
payment-failed-06 41 "If the payment arrives after your 10 minutes are over, your seats may already be gone. In that case the booking is not made, and the money goes back on its own, as explained above."
payment-failed-07 52 "In that case the booking is not made, and the money goes back on its own, as explained above. Please do not pay a second time for the same seats while the first payment is still being processed: wait for the confirmation or for the failed message."
Chunk 06 starts with the last sentence of 05, so its "In that case" still says which case. Without the overlap it would start with "In that case" and nothing before it.
Here is the trap. The next piece starts overlap sentences before the end of the piece before. If that piece held only one sentence (a long one, or a tight limit), "one sentence before its end" is where it started, so the next piece starts there again, and again, forever. Your program does not crash; it hangs. The rule that saves you: the next start is always at least one sentence after the last start. Think in indexes, with slice:
> const s = ["A.", "B.", "C.", "D.", "E."]
undefined
> s.slice(1, 3)
[ 'B.', 'C.' ]
> s.slice(1, 3).join(" ")
'B. C.'
slice(1, 3) is the sentences from index 1 up to, not including, index 3. If a piece is the slice from start to end, the overlap moves the next start back from end, and the rule above moves it forward again when it did not get past the old start. Math.max picks the larger of two numbers.
Ids that do not change
A chunk's id is its page's id, a dash, and its number on the page with two digits: cancelled-shows-02. The numbers run on across the sections of a page, so a page's second section does not start again at 01. Two digits keep ids in order when they are sorted as text:
> String(7).padStart(2, "0")
'07'
> String(12).padStart(2, "0")
'12'
> ["snacks-9", "snacks-10", "snacks-1"].sort()
[ 'snacks-1', 'snacks-10', 'snacks-9' ]
> ["snacks-09", "snacks-10", "snacks-01"].sort()
[ 'snacks-01', 'snacks-09', 'snacks-10' ]
padStart(2, "0") adds zeros in front until the text is two characters long. The pages with the most chunks have 6, so two digits are plenty.
The bigger reason is stability. An answer will say "[cancelled-shows-02]". Datta will check that citation, a test will expect it, and chapter 12's answer cache notices a changed chunk by its id and its text. Cut the same page tomorrow and -02 must be the same text. A random id changes on every run; a row number from Postgres changes whenever the table is filled again.
Here is the whole page cut with the finished code, the first two chunks of "The cancelled shows page in chunks":
[{"id":"cancelled-shows-01","pageId":"cancelled-shows","title":"Cancelled shows","heading":"Cancelled shows","position":1,"text":"This page explains what happens when Parda or a venue cancels a show you booked.","tokens":18},{"id":"cancelled-shows-02","pageId":"cancelled-shows","title":"Cancelled shows","heading":"What you get when a show is cancelled","position":2,"text":"When a show is cancelled, you get a full refund: the ticket price and the booking fee. You do not need to ask for it. The refund starts automatically, and we tell you by SMS and email on the day the show is cancelled.","tokens":50},
...
Pranav's answer is all in chunk 02, 50 tokens instead of 307.
Your turn
- Task 1,
splitSections(page): the page as{ heading, text }sections. The trap: the text before the first heading is a section too, under the title, and an empty section is left out. - Task 2,
slidingWindow(sentences, maxTokens, overlap): sentences packed into pieces. The trap: a piece of one sentence with an overlap of one loops forever unless the next start always moves on. - Task 3,
chunkPage(page, maxTokens, overlap): every chunk of a page, numbered across its sections. Nikhil will ask: an answer will cite "cancelled-shows-02". Why give chunks ids made from the page and a number, and not random ones or the row numbers Postgres gives?
Send "The cancelled shows page in chunks" and "The children's tickets page in chunks" and read the chunks: where each one starts and ends, its heading, its tokens. Then press Check. It cuts made-up pages and Parda's own. A first message looks like this:
splitSections kept the section "Empty heading", which has no text: leave out a section whose text is empty
A window that does not always move on is stopped after 5 seconds:
slidingWindow(["A.","B.","C."], 2, 1) did not finish in 5 seconds: each next piece must start at least one sentence later than the one before
And numbers that start again in each section:
the chunks' ids are ["interval-snacks-01","interval-snacks-01","interval-snacks-02","interval-snacks-01"]: the numbers run on across sections, so no id repeatsWhat you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
chunking.ts: cut a page at its headings
Write splitSections(page) in src/corpus/chunking.ts. Return the page's sections as an array of { heading, text } objects, in order. A line of the body that starts with "## " starts a section; its heading is the rest of the line, trimmed. The lines before the first "## " heading are a section under the page's title. A section's text is its lines joined with "\n" and trimmed; leave out a section whose text is empty.
The Check cuts a made-up page with an intro, a heading with spaces around it and no text, and two more sections, then a page whose body starts with a heading (no title section then).
- Task 2
chunking.ts: pack sentences into pieces, with an overlap
Write slidingWindow(sentences, maxTokens, overlap) in src/corpus/chunking.ts. Pack the sentences, in order, into pieces: a piece is sentences joined with one space, and countTokens of the joined piece stays at or under maxTokens. A piece takes the next sentence while it still fits, and always has at least one sentence (a sentence longer than maxTokens is a piece alone). The next piece starts overlap sentences before the end of the piece before (they share those sentences), but always at least one sentence later than the piece before started. Stop after the piece that holds the last sentence; no sentences, no pieces ([]).
slidingWindow(["A.", "B.", "C."], 4, 1) is ["A. B.", "B. C."]. The Check runs seven cases, among them an overlap as big as a piece and a sentence over the limit, and stops a window that never moves on after 5 seconds.
- Task 3 · interview question
Interview: chunking.ts, chunks with stable ids
Write chunkPage(page, maxTokens = 120, overlap = 1) in src/corpus/chunking.ts. For each section of the page (splitSections), split its text into sentences (splitSentences, given) and window them (slidingWindow); every piece becomes a Chunk: id is the page's id, a dash and the chunk's number on the page with two digits ("kids-tickets-03"); pageId and title are the page's; heading is the section's; position is the number (1, 2, 3 ... running on across every section of the page); text is the piece; tokens is countTokens(text).
The Check cuts a made-up page with maxTokens 8, then cancelled-shows with 120 and 1, then calls GET /pages/cancelled-shows/chunks. It uses your Tasks 1 and 2, so write them first.
Nikhil will ask: an answer will cite "cancelled-shows-02". Why give chunks ids made from the page and a number, and not random ones or the row numbers Postgres gives?
What you will learn
- Why chunks: the right size for searching and for the model's window, and why a chunk never crosses a heading.
- Sections: a page's ## headings, and the text before the first one.
- Sentences: the given splitSentences, and why a chunk is whole sentences.
- The sliding window: packing sentences up to a token limit, the overlap between neighbours, and the loop that must always move on.
- Stable ids: cancelled-shows-02, and why a citation needs an id that does not change.
More in The pages
Build "Chunking" yourself
Parda answers the same audience questions all day. You build the copilot that answers them from its own help pages, with sources, and says so when it does not know. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.