RAG in Python · Chapter 9 of 37

Getting text out

By the end of this chapter you can turn Markdown with front matter, the help centre's web pages and policy PDFs into one clean Doc, and report the files you could not read instead of losing them. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ingestion
  • 5 tasks
  • Python
  • In the all labs plan

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.

Why you need this

So far every search in this lab has read Paisa Setu's tidy Markdown. Real documents do not arrive that way. Kabir exports the help centre from the website, so every article comes wrapped in a cookie banner, a menu, breadcrumbs and a footer. Legal publishes the policies as PDFs, with a footer on every page and a line break every hundred characters.

Feed that to a search index as it is and the junk becomes part of every chunk. "Contact support" is on all 15 web pages, so a merchant who asks how to contact support gets 15 equally good matches. A clause cut by a page footer loses its meaning. Before you can cut documents into chunks (the next two chapters), you need clean text, and you need to know where it came from.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    loaders.py: read_markdown, Markdown with front matter

    raw is a whole .md file: a --- line, the front matter in YAML, another --- line, then the body. Write read_markdown(raw) in loaders.py: parse the front matter with yaml.safe_load and return a Doc with id, title, audience and updated (a date) from the front matter, source "markdown", and text, everything after the closing --- line, stripped.

    read_markdown of help-refund-timelines.md has the title "How long does a refund take to reach the customer?". A quoted title with a colon, like dev-disputes-api.md's "Disputes API: respond to chargebacks with evidence", stays whole, and a YAML comment after a value (audience: internal # staff only) is not part of it.

    audience must be merchant or internal: anything else, or no audience at all, raises (never guess merchant: it could be an internal runbook). A file with no front matter raises ValueError. Invalid YAML raises too: help-soundbox-rental.md's title has a bare colon, so let yaml's error through and do not repair it.

  2. Task 2

    loaders.py: read_html, the article and nothing else

    html is a whole page from data/raw/help-centre. Write read_html(html): parse it with BeautifulSoup and keep only its article element; a page with no article raises ValueError. Return a Doc: id is the article's data-id attribute, title the text of its first h1, audience "merchant" (the website is public), source "html".

    text is one block per element child of the article, in order, joined by a blank line. Skip every h1 (the page has the title twice) and the p with class meta. h2 to h6 become one # per level, a space and the text ("## Fees"). A ul is one line per li, "- " and its text; an ol is one line per li, "1. ", "2. " and so on. A table is one line per tr: its th and td cells joined by " | ", with "| " in front and " |" behind ("| IMPS | Works |"). Anything else, like a p, is its text.

    Every text has its runs of whitespace (spaces, new lines) squashed to one space, and is stripped; skip blocks that come out empty. On every real page the cookie banner, menu, breadcrumbs, related links, feedback widget, footer and scripts are gone, and every heading of the article is a line of its own.

  3. Task 3

    loaders.py: clean_pdf, footers out, paragraphs back

    pages is the text of each page of a PDF, as pypdf extracts it. Write clean_pdf(pages): paragraphs joined by a blank line.

    1. Split every page into lines, normalise each line with unicodedata.normalize("NFKC", line) (pypdf gives the fi ligature as one character), strip it, and drop empty lines and the footer, which is on every page and looks like Page 2 of 4Paisa Setu Technologies Pvt. Ltd. · Refund Policy · Version 2 (no space before Paisa). 2. Take the lines of all pages as one list, so a sentence cut by a page break joins up. 3. A heading is a number, a full stop, a space, a capital letter, and then no full stop, colon or semicolon (3. Refund window): it becomes "## 3. Refund window", a paragraph of its own. 4. Any other line joins the line before it, with one space, when that line is at least 80 characters long, does not end with . ? ! : or ;, and is not a heading, and this line does not start with a clause number and a space (4.2 ). Otherwise it starts a new paragraph.

    On the refund policy, clause 1.1 (wrapped over two lines) becomes one paragraph, "## 1. Definitions" is a paragraph, and the long table row "pending_balance The merchant's balance is too low; the refund is queued until funds arrive" stays apart from clause 4.2.

  4. Task 4

    loaders.py: read_pdf, a policy PDF as a Doc

    Write read_pdf(path): open the file with PdfReader, take every page's extract_text(), and return a Doc: id is the file name without .pdf, title is the PDF's metadata title ("" when it has none), audience "merchant" (the policies are public), source "pdf", and text is clean_pdf of the pages.

    For all 8 policy PDFs the title is the policy's title: read_pdf("/workspace/data/raw/policies/policy-refund-policy.pdf").title is "Refund Policy", and the text has no page footer and has "## 1. Definitions" as a paragraph.

  5. Task 5 · interview question

    Interview: loaders.py, load_all

    Write load_all(folder): read every .md, .html and .pdf file in folder and its sub-folders with the reader for its kind (read_markdown and read_html take the file's text, read_pdf its path), ignore every other file, and return LoadReport(docs=..., problems=...).

    One bad file must not stop the rest, and must not vanish either. When a reader raises, add Problem(file=the file's path as a string, reason=the error's message). When a document's text is empty after stripping (a scanned PDF has no text), add Problem(reason="no text") instead of the document.

    load_all("/workspace/data").docs has 166 documents (143 Markdown files, 15 web pages, 8 PDFs), and its problems name help-soundbox-rental.md and help-static-vs-dynamic-qr.md. Arjun will ask: one file in the export is broken; do you stop the whole index, or skip it?

What you will learn

  • Front matter: YAML read with a real parser, and refused when broken.
  • Web pages: the article kept and everything around it dropped, with headings kept as ## lines.
  • PDF text: repeated footers removed, ligatures undone, hard-wrapped lines joined.
  • Loading at scale: one bad file reported, never fatal, never silent.

More in Ingestion

Build "Getting text out" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.