AI support copilot project (TypeScript) · Chapter 6 of 12

Reranking

By the end of this chapter you can ask a cross-encoder to reread a search's best candidates beside the question and put them in its order, keep a few chunks a page so one page cannot fill the answer, and chain hybrid search, the reranker and that limit into the search the copilot will answer from. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Search
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of AI support copilot project (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Pranav has tickets for Raag Diwali at Sahyadri Lawns, the open-air venue, and asks: "Can I bring my dog to Sahyadri Lawns?" Chapter 5's hybrid search answers with these five:

hybrid5: outside-food-03:0.0325 pets-04:0.0323 rain-at-open-air-shows-02:0.0313 change-seats-04:0.0308 rain-at-open-air-shows-04:0.0301

First is outside-food-03: "Sahyadri Lawns is an open-air venue, and you may bring a picnic there." It shares the venue's name and the idea of bringing something, and a model reading it first could well answer "yes, bring what you like". The chunk that answers him, pets-04 ("Even at Sahyadri Lawns, the open-air venue, pets cannot come in"), is second, 0.0002 behind. And the fused scores, all between 0.0301 and 0.0325, say nothing about how good the best one is: they would look the same for a question the pages cannot answer at all.

Both searches compare things made apart: an arrow for the question and an arrow made for the chunk at ingest, or the question's words and the chunk's words. Neither reads the two together.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    rerank.ts: the reranker's order and scores

    Write rerank(ai, query, hits, topN) in src/search/rerank.ts. No hits: return [] without a request. No reranker configured (config.rerankModel is ''): return the first topN hits (all of them when topN is not given), unchanged. Otherwise send one request, ai.post<RerankResponse>('/rerank', { body }), with model config.rerankModel, query the query, documents one text per hit (the hit's title, ": ", its heading, a new line, its text) and top_n topN or hits.length. The answer's results, best first, are { index, relevance_score }: index is the hit's place in the list you sent. Return the hits in that order, each a copy with its relevance_score as its score.

    The Check reads the lab model's request log (one request, the model, top_n, the first document's text), compares your order and scores with the reranker's, then calls it with no topN, with no hits, and with config.rerankModel set to ''.

  2. Task 2

    rerank.ts: at most a few chunks a page

    Write limitPerPage(hits, perPage) in src/search/rerank.ts. Return the hits in their order, without every hit past the first perPage hits of the same pageId. Do not change the array you are given.

    The Check runs eight made-up hits from three pages with perPage 2, 1 and 5.

  3. Task 3 · interview question

    Interview: rerank.ts, the whole search

    Write retrieve(ai, q, k = config.k) in src/search/rerank.ts. Take config.candidates (20) chunks from hybridSearch(ai, q, config.candidates), rerank all of them, keep at most config.perPage (2) of a page with limitPerPage, and return the first k.

    The Check first makes sure your rerank (Task 1) and limitPerPage (Task 2) work, then compares your 5 chunks for "My show was cancelled. Do I get my money back?" with its own, wants one /rerank request of 20 documents, then k = 3, then GET /search.

    Nikhil will ask: the reranker is the best judge of the four searches. Why not rerank all 179 chunks for every question, and skip vector and keyword search?

What you will learn

  • A cross-encoder: what it reads, what its score means, and why it cannot be computed ahead like an embedding.
  • Calling an endpoint the openai package has no method for: ai.post(path, { body }).
  • Indexes into the list you sent: putting the reranker's answer back together with your hits.
  • Variety: at most a few chunks a page, and what it costs.
  • Retrieve, then rerank: why a cheap search finds 20 and the expensive judge reads only those.

More in Search

Build "Reranking" yourself

Parda answers the same audience questions all day. You build the copilot that answers them from its own help pages, with sources, and says so when it does not know. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI support copilot project (TypeScript) are free. This one comes with the all labs plan.