RAG: search that answers · Chapter 9 of 37
Getting text out
By the end of this chapter you can turn Markdown with front matter, the help centre's web pages and policy PDFs into one clean shape, and report the files you could not read instead of losing them. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Ingestion
- 5 tasks
- In the all labs plan
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.
Why you need this
So far every search in this lab has read Paisa Setu's tidy Markdown. Real documents do not arrive that way. Kabir exports the help centre from the website, so every article comes wrapped in a cookie banner, a menu, breadcrumbs and a footer. Legal publishes the policies as PDFs, with a footer on every page and a line break every hundred characters.
Feed that to a search index as it is and the junk becomes part of every chunk. "Contact support" is on all 15 web pages, so a merchant who asks how to contact support gets 15 equally good matches. A clause cut by a page footer loses its meaning. Before you can cut documents into chunks (the next two chapters), you need clean text, and you need to know where it came from.
What you build: 5 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
readMarkdown: Markdown with front matter
raw is a whole .md file: a line ---, the front matter in YAML, another line ---, then the body. Parse the front matter with parseYaml (the yaml package) and return a Doc: id and title from the front matter, audience from the front matter, source 'markdown', and text, everything after the closing --- line, trimmed.
readMarkdown of help-refund-timelines.md is { id: 'help-refund-timelines', title: 'How long does a refund take to reach the customer?', audience: 'merchant', source: 'markdown', text: the body }. A quoted title with a colon, like dev-disputes-api.md's 'Disputes API: respond to chargebacks with evidence', stays whole, and a YAML comment after a value (audience: internal # staff only) is not part of it.
audience must be 'merchant' or 'internal': anything else, or no audience at all, throws (never guess merchant: it could be an internal runbook). A file with no front matter throws. Invalid YAML throws too: help-soundbox-rental.md has title: Paisa Setu soundbox: rental and setup, a bare colon YAML rejects, so let yaml's error through and do not repair it.
- Task 2
readHtml: the article, and nothing else
html is a whole page from data/raw/help-centre. Parse it with parseHtml and keep only its article element; a page with no article throws. Return a Doc: id is the article's data-id attribute, title the text of its first h1, audience 'merchant' (the website is public), source 'html'.
text is one block per element child of the article, in order, joined by a blank line. Skip every h1 (the page has the title twice) and the p with class meta. h2 to h6 become one # per level, a space and the text ('## Fees'). A ul is one line per li, '- ' and its text; an ol is one line per li, '1. ', '2. ' and so on. A table is one line per tr: its th and td cells joined by ' | ', with '| ' in front and ' |' behind ('| IMPS | Works |'). Anything else, like a p, is its text.
Every text has its runs of whitespace (spaces, new lines) turned into one space, and is trimmed; skip blocks that come out empty. On every real page the cookie banner, menu, breadcrumbs, related links, feedback widget, footer and scripts are gone, and every heading of the article is a line of its own.
- Task 3
cleanPdf: footers out, paragraphs back
pages is the text of each page of a PDF, as unpdf extracts it. Return paragraphs joined by a blank line ('\n\n'):
1. Split every page into lines, trim each line, and drop empty lines and the footer, which is on every page and looks like Page 2 of 4Paisa Setu Technologies Pvt. Ltd. · Refund Policy · Version 2 (no space before Paisa). 2. Take the lines of all pages as one list, so a sentence cut by a page break joins up. 3. A heading is a number, a full stop, a space, a capital letter, and then no full stop, colon or semicolon (3. Refund window): it becomes '## 3. Refund window', a paragraph of its own. 4. Any other line joins the line before it, with one space, when that line is at least 80 characters long, does not end with . ? ! : or ;, and is not a heading, and this line does not start with a clause number and a space (4.2 ). Otherwise it starts a new paragraph.
On the refund policy, clause 1.1 (wrapped over two lines) becomes one paragraph, and the long table row pending_balance The merchant's balance is too low; the refund is queued until funds arrive stays apart from clause 4.2.
- Task 4
readPdf: a policy PDF as a Doc
path is a PDF file. Read its bytes, open it with getDocumentProxy (it takes a Uint8Array), read its pages with extractText and mergePages: false, and its metadata with getMeta. Return a Doc: id is the file name without .pdf, title is the PDF's Title (info.Title, or '' when it has none), audience 'merchant' (the policies are public), source 'pdf', and text is cleanPdf of the pages.
For all 8 policy PDFs the title is the policy's title: (await readPdf('/workspace/data/raw/policies/policy-refund-policy.pdf')).title is 'Refund Policy', and the text has no page footer and has '## 1. Definitions' as a paragraph.
- Task 5 · interview question
Interview: loadAll, load everything, lose nothing
Load every .md, .html and .pdf file in dir and its sub-folders with the reader for its kind (readMarkdown and readHtml take the file's text, readPdf its path), ignore every other file, and return { docs, problems }.
One bad file must not stop the rest, and must not vanish either. When a reader throws, add a Problem { file, reason }: file is the file's path (dir joined with its path inside dir) and reason the error's message. When a document's text is empty after trimming (a scanned PDF has no text), add a Problem with reason 'no text' instead of the document.
(await loadAll('/workspace/data')).docs.length is 166 (143 Markdown documents, 15 web pages, 8 PDFs), and problems names help-soundbox-rental.md and help-static-vs-dynamic-qr.md. Arjun will ask: one file in the export is broken; do you stop the whole index, or skip it?
What you will learn
- Front matter: reading YAML with a real parser, and refusing what it cannot read.
- Web pages: keeping the article and dropping everything around it, with headings kept as ## lines.
- PDF text: removing repeated footers and joining hard-wrapped lines.
- Loading at scale: one bad file reported, not fatal and not silent.
More in Ingestion
Build "Getting text out" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.