Building with LLMs · Chapter 19 of 24
Long documents
By the end of this chapter you can work with text too long for one request: decide whether it needs to be sent at all, split it into chunks by its structure or by a token budget, and summarise it in pieces, then summarise the pieces (map-reduce). Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Building with LLMs
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.
Why you need this
At midnight, Zoya wants five lines on what customers wrote about today. Lata exports the day's WhatsApp messages, 240 of them, into messages.jsonl, and the obvious program sends the file to the model with "Summarise these customer messages in 5 bullets.". The API refuses it:
openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's
maximum context length is 8192 tokens. However, your messages resulted in
16231 tokens. Please reduce the length of the messages.", 'type':
'invalid_request_error', 'param': 'messages', 'code':
'context_length_exceeded'}}
Chapter 3 explained the window: a model reads at most so many tokens at once, the request and the reply together. One day already does not fit, and a month never will. The same wall comes up everywhere: a 60-page contract, a year of reviews, a long support thread. You need ways to feed a model text larger than its window, and to know what each way costs.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
sections.py: chunk by structure
Write split_sections(text) in sections.py. It splits a Markdown document into its sections: a section starts at a line that begins with two hashes and a space, and runs up to the next such line or the end. Return the sections as a list of strings, in order, each starting with its heading line, with blank lines removed from both ends. Drop whatever comes before the first such heading.
A heading with three hashes is inside a section, and two hashes in the middle of a line are not a heading. For policy.md it returns 9 sections.
- Task 2
pack.py: chunk by budget
Write pack(lines, max_tokens) in pack.py. It joins whole lines, in order, into chunks: a chunk is its lines joined with a newline, and holds at most max_tokens tokens, counted with tiktoken's o200k_base encoding on the chunk's text. Start a new chunk only when the next line would not fit.
A line is never split: if one line alone has more than max_tokens tokens, raise ValueError. An empty list of lines gives an empty list.
- Task 3
map_reduce.py: map, then reduce
Summarise /workspace/data/policy.md in two steps. Map: for each section (task 1's split_sections), ask lab-chat for a summary in 1 sentence, with the instruction as the system message and the section as the user message, one request per section, in order. Reduce: one more request asking for a summary in 4 bullets of all those sentences, joined by newlines.
Print only the reply to the last request.
- Task 4 · interview question
Interview: day_summary.py, the day in five bullets
day_summary.py sends Lata's export of the day's 240 messages to lab-chat as it is, and the API refuses it. Fix it so that it sends only the text of each message, one per line, each line starting with a dash and a space; packs those lines into chunks of at most 1,500 tokens (task 2's pack); asks lab-chat to summarise each chunk in 3 bullets, one request per chunk with the chunk as the user message; then asks once more for a summary in 5 bullets of those summaries joined by newlines, and prints only that.
Every message must reach the model exactly once.
What you will learn
- Send less first: most of a long input is not needed.
- A bigger window, and why it is not free.
- Chunking: by the document's structure, or by a token budget.
- Map-reduce: summarise each chunk, then summarise the summaries.
- What gets lost at each step.
More in Building with LLMs
Build "Long documents" yourself
SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.