AI support copilot project · Chapter 6 of 12

Reranking

By the end of this chapter you can ask a cross-encoder to reread a search's best candidates beside the question and put them in its order, keep a few chunks a page so one page cannot fill the answer, and chain hybrid search, the reranker and that limit into the search the copilot will answer from. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Search
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI support copilot project are free. This one comes with the all labs plan.

Why you need this

Advait's dabba came 40 minutes late, and he asks: "My dabba came 40 minutes late. Do I get my money back?" Chapter 5's hybrid search answers with these five:

My dabba came 40 minutes late. Do I get my money back? | hybrid: late-delivery-refunds-09:0.0320 late-delivery-refunds-04:0.0320 late-delivery-refunds-05:0.0315 rain-delays-03:0.0306 late-delivery-refunds-07:0.0306

First is late-delivery-refunds-09, "When these late delivery rules do not apply" (a building that would not let the rider in). The chunk that answers him, late-delivery-refunds-04, "Full refund when your dabba is more than 30 minutes late", is second, tied on 0.0320. Hybrid search knows the words and the meaning are close; it does not know which chunk actually answers a question about 40 minutes. And its scores, all between 0.0306 and 0.0320, say nothing about how good the best one is.

Both searches compare things made apart: an arrow for the question and an arrow made for the chunk at ingest, or the question's words and the chunk's words. Neither reads the two together.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    rerank.py: the reranker's order and scores

    Write rerank(ai, query, hits, top_n=None) in app/search/rerank.py. No hits: return [] without a request. No reranker configured (settings.rerank_model is ""): return hits[:top_n or len(hits)] unchanged. Otherwise send one request, ai.post("/rerank", cast_to=object, body=...), with "model" settings.rerank_model, "query" the query, "documents" one text per hit (the hit's title, ": ", its heading, a new line, its text) and "top_n" top_n or len(hits). The answer is a dict whose "results" list, best first, holds {"index", "relevance_score"}: index is the hit's place in the list you sent. Return the hits in that order, each with its relevance_score as its score (dataclasses.replace).

    The Check reads the lab model's request log (one request, the model, the query, top_n, the first document's text), compares your order and scores with the reranker's, then calls it with no top_n and with no hits.

  2. Task 2

    rerank.py: at most a few chunks a page

    Write limit_per_page(hits, per_page) in app/search/rerank.py. Return the hits in their order, without every hit past the first per_page hits of the same page_id.

    The Check runs eight made-up hits from three pages with per_page 2, 1 and 5.

  3. Task 3 · interview question

    Interview: rerank.py, the whole search

    Write retrieve(db, ai, q, k=None) in app/search/rerank.py. k is k, or settings.k (5) when it is None. Take settings.candidates (20) chunks from hybrid_search(db, ai, q, k=settings.candidates), rerank them all, keep at most settings.per_page (2) of a page with limit_per_page, and return the first k.

    The Check (its own rerank and limit_per_page, so this task stands alone) compares your 5 chunks for "It rained and my lunch was late. What do I get?" with its own, wants one /rerank request of 20 documents, then k=3, then GET /search.

    Kunal will ask: the reranker is the best judge of the four searches. Why not rerank all 162 chunks for every question, and skip vector and keyword search?

What you will learn

  • A cross-encoder: what it reads, what its score means, and why it cannot be computed ahead like an embedding.
  • Calling an endpoint the openai package has no method for: ai.post(path, cast_to=object, body=...).
  • Indexes into the list you sent: putting the reranker's answer back together with your hits.
  • Variety: at most a few chunks a page, and what it costs.
  • Retrieve then rerank: why a cheap search finds 20 and the expensive judge reads only those.

More in Search

Build "Reranking" yourself

DabbaDrop answers the same customer questions all day. You build the copilot that answers them from its own help pages, with sources, and says so when it does not know. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI support copilot project are free. This one comes with the all labs plan.