AI support copilot project · Chapter 5 of 12

Keyword and hybrid search

By the end of this chapter you can search DabbaDrop's chunks by their words with Postgres full-text search (to_tsvector, @@, ts_rank_cd, a GIN index), fuse that ranking with chapter 4's vector search by reciprocal rank, and say why ranks fuse better than scores. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Search
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI support copilot project are free. This one comes with the all labs plan.

Why you need this

Vector search finds chunks that mean the same as the question. Members do not always write meanings, though. They write words: "OTP not coming", "GSTIN", "Paryushan", "steel dabba lost". A small embedding model has seen "OTP" rarely, and its arrow for that question points nowhere in particular. This is what chapter 4's search gives a member who cannot log in:

OTP not coming | vector: contact-support-04:0.236 office-orders-03:0.191 how-to-order-03:0.187 payment-methods-03:0.186 subscription-plans-05:0.183

The chunk that answers, account-and-login-02 ("we send the OTP to that number by SMS"), is eighth, with a score of 0.159. The five chunks the model would get are about support hours, office orders and payments. The word "OTP" is right there in the answer's text, and nothing in the code so far looks at words. Vector search has no way to say "this chunk contains the exact word the member typed".

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    keyword.py: full-text search in Postgres

    Write keyword_search(db, q, k=5) in app/search/keyword.py. Return the k chunks whose words match the question best, as Hit objects, best first. Use the given any_word(q) as the query. Keep only chunks whose search column matches it (HelpChunk.search.op("@@")(query)); rank with func.ts_rank_cd(HelpChunk.search, query); select the chunk's id, page_id, heading and text, the page's title (join HelpPage) and the rank; order by the rank, highest first, then the chunk's id; limit k. score is the rank, as a float.

    The Check compares your hits for three questions with its own SQL, and wants [] for a word no chunk has.

  2. Task 2

    hybrid.py: reciprocal rank fusion

    Write rrf(rankings, k=60) in app/search/hybrid.py. rankings is a list of rankings, each a list of chunk ids, best first. An id at rank r (from 1) of a ranking earns 1 / (k + r); an id in several rankings adds them up. Return every id with its total, as (id, score) tuples, highest score first, ties by id.

    rrf([["a", "b"], ["b", "c"]]) is [("b", 1/62 + 1/61), ("a", 1/61), ("c", 1/62)]. The Check runs four cases (one with an empty ranking, one with k=1) and the default k.

  3. Task 3 · interview question

    Interview: hybrid.py, vector and keyword search together

    Write hybrid_search(db, ai, q, k=5, candidates=None) in app/search/hybrid.py. n is candidates, or settings.candidates (20) when it is None. Embed q with embed_texts (one request), take vector_search's n nearest chunks and keyword_search's n best, fuse the two rankings of their chunk ids with rrf, and return the first k as Hit objects in the fused order, each with its fused score (dataclasses.replace). A chunk only one search found is still a hit.

    The Check (its own rrf and keyword_search, so this task stands alone) fuses 7 and 7 candidates for "It rained and my dabba was late, what do I get?" with k=4, checks the defaults, and calls GET /search?mode=hybrid.

    Kunal will ask: a cosine similarity of 0.7 and a ts_rank_cd of 1.3 came back for the same chunk. Why does rrf throw both numbers away and use only the ranks?

What you will learn

  • Full-text search in Postgres: to_tsvector, stemming, stop words, plainto_tsquery, & and |, and the @@ match.
  • Ranking by words: what ts_rank_cd rewards, and why bigger is better.
  • A GIN index: an index from every word to the rows that have it.
  • SQLAlchemy for operators and functions: .op("@@"), func.<name>, .desc().
  • Reciprocal rank fusion: adding 1 / (k + rank) from several rankings, and why it ignores the scores.

More in Search

Build "Keyword and hybrid search" yourself

DabbaDrop answers the same customer questions all day. You build the copilot that answers them from its own help pages, with sources, and says so when it does not know. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI support copilot project are free. This one comes with the all labs plan.