RAG: search that answers · Chapter 13 of 37

The same text twice

By the end of this chapter you can find exact and near-duplicate chunks from scratch, measure how many the index has, and decide which copy stays. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ingestion
  • 5 tasks
  • In the all labs plan

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.

Why you need this

Kabir asks the vector search a merchant's question, "what are the ways a customer can pay by UPI on my website?", and reads the top five:

$ node demo.ts
What are the ways a customer can pay by UPI on my website?
   0.663 help-accept-upi-payments#0
   0.651 help-upi-payments-guide#0
   0.622 help-accept-upi-payments#5
   0.614 help-upi-payments-guide#5
   0.607 help-accept-upi-payments#1

Four of the five slots hold two passages, each twice. The guide was migrated from the old help centre with its wording changed, and nobody deleted it. Five slots that could hold five different facts hold two, and the model reads every one of those tokens. If the old copy is ever edited and the new one is not, the two will disagree, and the model may cite either.

Removing a copy sounds easy, until you notice that the two settlement policies (2024 and 2026) look like copies too, and one says T+2 by 6 pm while the other says T+1 by 11 am.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    exactDuplicates: exact copies, whatever the spacing

    normalise(text) returns the text in lower case, with every run of whitespace (spaces, tabs, line breaks, carriage returns, non-breaking spaces) turned into one space, and trimmed: normalise(' Refund\r\n FAILED ') is 'refund failed'. Punctuation and numbers stay as they are.

    exactDuplicates(chunks) groups the chunks whose normalised texts have the same SHA-256 hash, and returns only the groups of two or more, each a list of ids in the order of chunks. Paisa Setu's 866 reference chunks give []; with three articles exported again (Windows line endings, double spaces, an upper-case first word), every exported chunk is grouped with its original, original first.

  2. Task 2

    shingles: shingles and Jaccard similarity

    shingles(text, k) returns the Set of every run of k words in a row, joined by single spaces. A word is a run of letters or digits in lower case, so 'Refunds to UPI take 1-3 days.' has the words refunds, to, upi, take, 1, 3, days. shingles('UPI refunds take 1 to 3 days', 3) is { 'upi refunds take', 'refunds take 1', 'take 1 to', '1 to 3', 'to 3 days' }. A text with fewer than k words gives one shingle of all its words, and a text with no words gives an empty set.

    jaccard(a, b) is the number of items in both sets divided by the number in either: { a, b, c } and { b, c, d } give 0.5, equal sets 1, nothing shared 0, and two empty sets 0.

  3. Task 3

    nearDuplicatePairs: how many near-duplicates?

    Compare every chunk with every chunk of another document (never two chunks of the same document) by the Jaccard of their k-word shingles, and return the pairs scoring threshold or more as { a, b, score }: a is the chunk that comes first in chunks, score is the exact Jaccard. Sort by score, highest first.

    On the reference chunks, nearDuplicatePairs(chunks, 1, 0.6) is 15 pairs, starting with policy-settlement-policy-2024#11 and policy-settlement-policy#11 at about 0.871; the last scores exactly 0.6 and counts. nearDuplicatePairs(chunks, 5, 0.5) is []: no chunk was copied word for word. Make each chunk's shingles once, before comparing: there are about 370,000 pairs.

  4. Task 4

    signature: MinHash, a short signature for a set

    signature(set, seeds) returns one number per seed (128 with SEEDS): for each seed, the smallest value of mix(fnv1a(shingle), seed) over every shingle in the set. For a set with one shingle s, it is mix(fnv1a(s), seed) for each seed. The order of the set does not matter, and it uses the seeds it is given.

    estimate(a, b) is the fraction of positions where two signatures hold the same number: [1, 2, 3, 4] and [1, 9, 3, 9] give 0.5, and a signature with itself gives 1. Over many real pairs of chunks it is within a few hundredths of the exact Jaccard on average. fnv1a, mix and SEEDS are given.

  5. Task 5 · interview question

    Interview: winner, which copy wins

    winner(a, b) returns the chunk to keep, or null to keep both. Different audiences: null. Either chunk has a version: null (policy-settlement-policy-2024 and policy-settlement-policy are versions, not copies). Otherwise the type earlier in CANONICAL wins (help-instant-settlement beats a newer community post), then the newest updated (help-accept-upi-payments, 2026-04-14, beats help-upi-payments-guide, 2025-09-22), then the smaller id. The answer must not depend on the order of a and b.

    duplicatesToDrop(pairs, chunks) returns every pair's loser, each once, sorted. For the 11 pairs closest in meaning, that is help-upi-payments-guide#0, #1 and #5, and no settlement policy chunk.

    Arjun will ask which copy you keep, and what must never be merged.

What you will learn

  • Exact duplicates: a hash of the normalised text.
  • Shingles and Jaccard similarity: how much two texts share, word for word.
  • MinHash: a short signature that estimates Jaccard, and finds similar pairs without comparing every pair.
  • Which copy wins: rules from metadata, and what must never be merged.

More in Ingestion

Build The same text twice yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.