RAG in Python · Chapter 20 of 37

A second opinion

By the end of this chapter you can rerank search results with a cross-encoder, in batches and with a deadline, and say from measured numbers when the extra second is worth it. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Better retrieval
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.

Why you need this

Hybrid search puts a right document in the top 5 for 96% of the labelled questions, but its MRR is stuck near 0.76: the right document is often second or fourth. Kabir shows one: "what should my code do when it gets a refund.failed webhook" comes back with the API errors page first, and the webhook events page is 16th. The model reads the first passages most carefully, so the order matters, not only the set.

A better embedding will not fix this. An embedding squeezes a whole chunk into 384 numbers before it has seen the question. You need a model that reads the question and the chunk together.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    rerank.py: rerank, ask the reranker

    Write the async rerank(query, documents): POST to /rerank with await new_client().post("/rerank", body=..., cast_to=object), the body {"model": RERANK_MODEL, "query": query, "documents": documents}. The reply is a dict whose "results" are {"index", "relevance_score"}, best first: return them as Ranked(index, score), in the same order.

    rerank("how long do card refunds take?", three sentences) puts the card refund sentence (index 2) first. An error must raise, never come back as an empty list: rerank("", documents) is a 400 from the reranker, and the check expects an exception with 400 in it.

  2. Task 2

    rerank.py: rerank_chunks, in batches, keep the best

    Write the async rerank_chunks(query, chunks, keep=5): rerank the chunks for the query, sending passage(c) for each (its title, a blank line, its text), and return the keep best Chunk objects, best first. The reranker takes at most MAX_DOCUMENTS (100) documents per request, so send longer lists in batches of 100, run together with asyncio.gather, with your rerank for each batch, and merge every batch's results by score.

    With 150 chunks, rerank_chunks sends two requests, of 100 and 50 documents, in flight at the same time. Each result's index is its position in its own batch: the best chunk at position 130 comes back as index 30 of the second batch.

  3. Task 3

    rerank.py: search_and_rerank, retrieve many, keep few

    Write the async search_and_rerank(conn, query, candidates=50, keep=5, table): take candidates chunk ids from hybrid_search for the query, turn them into chunks with chunk(id), rerank them with rerank_chunks, and return the ids of the keep best, best first.

    search_and_rerank(conn, "what does UPI error U30 mean?", 10, 5) sends the reranker 10 documents (the passages of the 10 hybrid results, in their order) and returns 5 of those ids, in the reranker's order.

  4. Task 4 · interview question

    Interview: rerank.py, rerank_within a deadline

    Write the async rerank_within(query, chunks, seconds, keep=5): rerank the chunks' passages in one request, with asyncio.wait_for and seconds as the deadline. In time, return Within(chunks=the keep best chunks in the reranker's order, reranked=True). When the deadline passes, wait_for raises TimeoutError: return Within(chunks=the first keep chunks as you were given them, reranked=False).

    rerank_within(query, chunks, 0.001) is Within(chunks=chunks[:5], reranked=False), at once and without raising. Only a timeout falls back: rerank_within("", chunks, 30) must still raise, because the empty query is a 400.

    Arjun will ask: the reranker adds a second or two to every question here. What does the merchant get when it is slow?

What you will learn

  • Bi-encoders and cross-encoders: why reading together ranks better, and why it costs more.
  • The rerank endpoint: a POST the openai package does not have a method for, and errors that do not raise.
  • Retrieve many, keep few: 50 candidates in, 5 out, in batches run together.
  • When it pays: hit@1, MRR and milliseconds, measured, and a deadline for when it is slow.

More in Better retrieval

Build "A second opinion" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.