AI support copilot project (TypeScript) · Chapter 5 of 12

Keyword and hybrid search

By the end of this chapter you can search Parda's chunks by their words with Postgres full-text search (to_tsvector, @@, ts_rank_cd, a GIN index) from Drizzle's sql template, fuse that ranking with chapter 4's vector search by reciprocal rank, and say why ranks fuse better than scores. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Search
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of AI support copilot project (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Vector search finds chunks that mean the same as the question. Members do not always write meanings, though. They write the exact word they saw: "striped", "GSTIN", "Mandai", "DIWALI100". Neha is booking seats for her family and asks: "Why are some seats striped?" Chapter 4's search gives her:

vector: change-seats-03:0.3639 how-to-book-05:0.3381 change-seats-04:0.3372 change-seats-05:0.3283 how-to-book-02:0.3225

The chunk that answers, how-to-book-02 ("seats someone else is paying for are striped"), only just makes fifth place. The four above it are about changing seats and seat classes: the question is about seats, so the arrow points at seats in general. The scores sit within 0.04 of each other, so nothing says which one matters. The word "striped" is right there in the answer's text, and it is in no other chunk, but nothing in the code so far looks at words.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    keyword.ts: full-text search in Postgres

    Write anyWord(q) and keywordSearch(q, k = 5) in src/search/keyword.ts.

    anyWord(q) returns a Drizzle sql fragment: the question as a tsquery that matches a chunk with any of its words. Take plainto_tsquery('english', q) (q as a parameter), turn it into text, replace every & with |, and cast it back to tsquery. For "Why are some seats striped?" it is 'seat' | 'stripe'.

    keywordSearch(q, k) returns the k chunks whose words match the question best, as Hits ({ chunkId, pageId, title, heading, text, score }), best first. With db, select the chunk's id, pageId, heading and text, the page's title (an inner join of pages on its id) and the rank, ts_rank_cd(chunks.search, anyWord(q)); keep only the chunks whose search column matches it (@@); order by the rank, highest first, then the chunk's id; limit k. score is the rank, as a number.

    The Check runs anyWord in Postgres for two questions, compares your hits for three questions with its own SQL, wants [] for a word no chunk has, and calls GET /search?mode=keyword.

  2. Task 2

    hybrid.ts: reciprocal rank fusion

    Write rrf(rankings, k = 60) in src/search/hybrid.ts. rankings is an array of rankings, each an array of chunk ids, best first. An id at rank r (from 1) of a ranking earns 1 / (k + r); an id in several rankings adds them up. Return every id with its total, as [id, score] pairs, highest score first, ties by id.

    rrf([['a', 'b'], ['b', 'c']]) is [['b', 1/62 + 1/61], ['a', 1/61], ['c', 1/62]]. The Check runs five cases (one with an empty ranking, one with k = 1, one with a tie) and the default k.

  3. Task 3 · interview question

    Interview: hybrid.ts, vector and keyword search together

    Write hybridSearch(ai, q, k = 5, candidates = config.candidates) in src/search/hybrid.ts. Embed q with embedTexts (one request), take vectorSearch's candidates nearest chunks and keywordSearch's candidates best ones, fuse the two rankings of their chunk ids with rrf, and return the first k as Hits in the fused order, each with its fused score in place of the old one. A chunk only one search found is still a hit.

    The Check first makes sure your keywordSearch (Task 1) and rrf (Task 2) work, then fuses 7 and 7 candidates with k = 4 for "Does my two year old need a ticket?" and "How far is it from Mandai?", wants one embedding request, checks the defaults (k = 5, 20 candidates), and calls GET /search?mode=hybrid.

    Nikhil will ask: a cosine similarity of 0.58 and a ts_rank_cd of 1.6 came back for the same chunk. Why does rrf throw both numbers away and use only the ranks?

What you will learn

  • Full-text search in Postgres: to_tsvector, stemming, stop words, plainto_tsquery, & and |, and the @@ match.
  • Ranking by words: what ts_rank_cd rewards, and why bigger is better.
  • A GIN index: an index from every word to the rows that have it.
  • Drizzle's sql template: SQL that Drizzle has no function for, with columns and values inside.
  • Reciprocal rank fusion: adding 1 / (k + rank) from several rankings, and why it ignores the scores.

More in Search

Build "Keyword and hybrid search" yourself

Parda answers the same audience questions all day. You build the copilot that answers them from its own help pages, with sources, and says so when it does not know. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI support copilot project (TypeScript) are free. This one comes with the all labs plan.