RAG in Python · Chapter 13 of 37
The same text twice
By the end of this chapter you can find exact and near-duplicate chunks from scratch, measure how many the index has, and decide which copy stays. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Ingestion
- 5 tasks
- Python
- In the all labs plan
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.
Why you need this
Kabir asks the vector search a merchant's question and reads the top five:
$ python3 top_five.py
What are the ways a customer can pay by UPI on my website?
0.672 help-accept-upi-payments#0
0.658 help-upi-payments-guide#0
0.619 help-accept-upi-payments#5
0.614 help-upi-payments-guide#5
0.609 help-accept-upi-payments#1
Four of the five slots hold two passages, each twice. The guide was migrated from the old help centre with its wording changed, and nobody deleted it. Five slots that could hold five different facts hold two, and the model reads every one of those tokens. If the old copy is ever edited and the new one is not, the two will disagree, and the model may cite either.
Removing copies sounds easy, until you notice that the two settlement policies (2024 and 2026) look like copies too, and one says T+2 by 6 pm while the other says T+1 by 11 am. Chapter 12 put the metadata on every chunk; now it decides.
What you build: 5 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
dedup.py: exact_duplicates, copies whatever the spacing
Write normalise(text) in dedup.py: the text in lower case, every run of whitespace (spaces, tabs, line breaks, carriage returns, non-breaking spaces) turned into one space, with nothing at either end. normalise(" Refund\r\n FAILED ") is "refund failed". Punctuation and numbers stay as they are.
Then exact_duplicates(chunks): group the chunks whose normalised texts have the same SHA-256 hash, and return only the groups of two or more, each a list of ids in the order of chunks. Paisa Setu's 866 reference chunks give []; with three articles exported again (Windows line endings, double spaces, an upper-case first word), every exported chunk is grouped with its original, original first.
- Task 2
dedup.py: shingles and jaccard, how much two texts share
Write shingles(text, k) in dedup.py: the set of every run of k words in a row, joined by single spaces. A word is a run of letters or digits in lower case, so "Refunds to UPI take 1-3 days." has the words refunds, to, upi, take, 1, 3, days. shingles("UPI refunds take 1 to 3 days", 3) is {"upi refunds take", "refunds take 1", "take 1 to", "1 to 3", "to 3 days"}. A text with fewer than k words gives one shingle of all its words, and a text with no words gives an empty set.
Then jaccard(a, b): the number of items in both sets divided by the number in either. {"a", "b", "c"} and {"b", "c", "d"} give 0.5, equal sets 1, nothing shared 0, and two empty sets 0.
- Task 3
dedup.py: near_duplicate_pairs, how many near-duplicates
Write near_duplicate_pairs(chunks, k, threshold) in dedup.py: compare every chunk with every chunk of another document (never two chunks of the same document) by the Jaccard of their k-word shingles, and return the pairs scoring threshold or more as Pairs: a is the chunk that comes first in chunks, score is the exact Jaccard. Sort by score, highest first.
On the reference chunks near_duplicate_pairs(chunks, 1, 0.6) is 15 pairs, starting with policy-settlement-policy-2024#11 and policy-settlement-policy#11 at about 0.871; the last scores exactly 0.6 and counts. near_duplicate_pairs(chunks, 5, 0.5) is []: no chunk was copied word for word. Make each chunk's shingles once, before comparing: the check counts how many times shingles is called.
- Task 4
dedup.py: signature and estimate, MinHash
Write signature(items, seeds) in dedup.py: one int per seed (128 with SEEDS), each the smallest mix(fnv1a(shingle), seed) over every shingle in the set. For a set with one shingle s it is mix(fnv1a(s), seed) for each seed. Use the seeds you are given, and the given fnv1a and mix, never Python's hash(), which changes in every process.
Then estimate(a, b): the share of positions where two signatures hold the same number. [1, 2, 3, 4] and [1, 9, 3, 9] give 0.5, and a signature with itself gives 1. Over many real pairs of chunks it is within a few hundredths of the exact Jaccard on average.
- Task 5 · interview question
Interview: dedup.py, winner, which copy stays
Write winner(a, b) in dedup.py: the chunk to keep, or None to keep both. Different audiences: None. Either chunk has a version: None (policy-settlement-policy-2024 and policy-settlement-policy are versions, not copies). Otherwise the type earlier in CANONICAL wins (help-instant-settlement beats a newer community post), then the newest updated (help-accept-upi-payments, 2026-04-14, beats help-upi-payments-guide, 2025-09-22), then the smaller id. The answer must not depend on the order of a and b.
Then duplicates_to_drop(pairs, chunks): every pair's loser, each once, sorted. For the 11 pairs closest in meaning, that is help-upi-payments-guide#0, #1 and #5, and no settlement policy chunk.
Arjun will ask which copy you keep, and what must never be merged.
What you will learn
- Exact duplicates: a hash of the normalised text.
- Shingles and Jaccard similarity: how much two texts share, word for word.
- MinHash: a short signature that estimates Jaccard, and finds similar pairs without comparing every pair.
- Which copy wins: rules from metadata, and what must never be merged.
More in Ingestion
Build The same text twice yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.