RAG in Python · Chapter 19 of 37

Keywords and meaning together

By the end of this chapter you can search Paisa Setu by words in Postgres, fuse that with vector search by rank, and show with numbers, kind by kind, where the hybrid wins and where it does not. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Better retrieval
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.

Why you need this

Kabir has two complaints about the vector search from Part 4. A merchant types "SH-03" or "U66" and gets articles about holds and errors in general, not the one for that code. Another writes "the little speaker next to my QR has gone quiet" and a keyword search finds nothing, because no article says "quiet". Chapter 8 showed each search wins where the other loses. You do not have to choose: run both, and merge the two lists into one.

Merging is the hard part. A keyword score of 0.05 and a cosine similarity of 0.72 are not on the same scale, so you cannot add them. And the keyword search Kabir wrote returns nothing for most real questions.

Run lab index once in the terminal: it builds the chunks table this chapter searches (paisa.build_index, with the full-text column and its index too).

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    hybrid.py: keyword_search, a keyword search that finds something

    Kabir's keyword_search in the starter uses plainto_tsquery, which joins every word with AND, so a long question matches no chunk at all. Fix it: return the ids of the k chunks whose tsv column matches ANY of the query's words, ordered by ts_rank, highest first.

    keyword_search(conn, "Hi, my refund to the customer has failed twice since Monday, what should I do now?", 10) is 10 chunk ids (it was []), and keyword_search(conn, "what does UPI error U30 mean?", 5) starts with "help-upi-error-u30-z9#0". A merchant's text must never make it fail: to_tsquery raises a syntax error on text like 11:50 pm, a URL or an &, and the check tries those. On the keyword questions, a right document is in the top 5 for at least 80% of them.

  2. Task 2

    hybrid.py: rrf, reciprocal rank fusion

    Write rrf(lists, weights=None, k=60): fuse ranked lists of ids by rank. Each list gives an id weight / (k + rank), where rank is 1 for a list's first id, and an id's score is the sum over every list it appears in. Return every id once as Fused(id, score), highest score first. weights has one number per list (all 1 when it is None).

    rrf([["a", "b", "c"], ["c", "a"]]) is a with 1/61 + 1/62, then c with 1/63 + 1/61, then b with 1/62. With weights [2, 1], rrf([["x", "y"], ["y", "x"]]) puts x first with 2/61 + 1/62. rrf([["solo"]], [1], 0) scores solo 1. rrf([[], []]) is [].

  3. Task 3

    hybrid.py: hybrid_search, words and meaning, fused

    Write hybrid_search(conn, query, k=10, weights=Weights(), table): run keyword_search and vector_search for the query, CANDIDATES (50) chunks each, fuse the two lists with rrf (the keyword list first, with the weights' keyword and vector numbers in that order), and return the ids of the k best.

    hybrid_search(conn, question, 5) is 5 different ids. With Weights(keyword=0, vector=1) it is exactly vector_search(conn, question, 5). Over the 70 answerable labelled questions, searching 20 chunks, it puts a right document in the top 5 for more questions than vector search alone (65 of 70).

  4. Task 4 · interview question

    Interview: hybrid.py, evaluate by kind

    Write evaluate(questions, search): score a search on labelled questions, by document. Skip questions whose relevant list is empty. For each other question, call search with its question text, turn the chunk ids into document ids (the part before #), keep only the first time each document appears, and find the position of the first relevant one. Return a dictionary: for each kind, and for "all", a Score(n, hit5, mrr), where n is the number of questions, hit5 the share with a relevant document in the first 5 documents, and mrr the mean of 1 / rank (rank 1 for the first document; 0 when none is found).

    Chunks ["help-x#0", "help-x#1", "help-a#0"] for a question whose relevant is ["help-a"] are the documents help-x, help-a, so the rank is 2 and 1 / rank is 0.5. There is no row for unanswerable questions.

    Arjun will ask: hybrid is better on average; is it better for every kind of question, and which would you ship?

What you will learn

  • Full-text search in Postgres: the tsv column, plainto_tsquery, websearch_to_tsquery, and ts_rank.
  • Why an AND query fails on long questions, and how to match any word.
  • Reciprocal rank fusion (RRF): merging lists by rank, with weights.
  • Measuring by kind: hit@5 and MRR per kind of labelled question.

More in Better retrieval

Build "Keywords and meaning together" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.