RAG: search that answers · Chapter 20 of 37

A second opinion

By the end of this chapter you can rerank search results with a cross-encoder, in batches and with a deadline, and say from measured numbers when the extra second is worth it. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Better retrieval
  • 4 tasks
  • In the all labs plan

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.

Why you need this

Hybrid search puts a right document in the top 5 for 93% of the labelled questions, but its MRR is stuck near 0.75: the right document is often second or fourth. Kabir shows one: "what should my code do when it gets a refund.failed webhook" comes back with the API errors page first, and the webhook events page is not in the top 5 at all. The model reads the first passages most carefully, so the order matters, not only the set.

A better embedding will not fix this. An embedding squeezes a whole chunk into 384 numbers before it has seen the question. You need a model that reads the question and the chunk together.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    rerank: ask the reranker

    POST to RERANK_URL with fetch: the header content-type application/json, the header authorization with Bearer and the key in OPENAI_API_KEY (without it the lab answers 401), and a JSON body { model: 'lab-rerank', query, documents }. Pass signal on to fetch. The reply's results are { index, relevance_score }, best first: return them as Ranked objects { index, score }, in the same order.

    rerank('how long do card refunds take?', three sentences) puts the card refund sentence (index 2) first. fetch does not throw on an error status: when the reply is not ok, throw an Error whose message has the status. rerank('', docs) is a 400 from the reranker, so it must reject with '400' in the message.

  2. Task 2

    rerankChunks: rerank in batches, keep the best

    Rerank the chunks for the query and return the keep best Chunk objects, best first. Send passage(c) for each chunk (its title, a blank line, its text). The reranker takes at most MAX_DOCUMENTS (100) documents per request, so send longer lists in batches of 100 and merge the results of every batch by score.

    With 150 chunks, rerankChunks sends two requests, of 100 and 50 documents. Each result's index is its position in its own batch: the best chunk at position 130 comes back as index 30 of the second batch.

  3. Task 3

    searchAndRerank: retrieve many, keep few

    Take candidates chunk ids from hybridSearch for the query, turn them into chunks with chunk(id), rerank them with rerankChunks, and return the ids of the keep best, best first.

    searchAndRerank(db, 'what does UPI error U30 mean?', 10, 5) sends the reranker 10 documents (the 10 hybrid results) and returns 5 of those ids, in the reranker's order.

  4. Task 4 · interview question

    Interview: rerankWithin, a reranker with a deadline

    Rerank the chunks (their passages, in one request) with AbortSignal.timeout(ms) as the signal. In time, return { chunks: the keep best chunks in the reranker's order, reranked: true }. When the request times out, fetch rejects with an error named 'TimeoutError': return { chunks: the first keep chunks as you were given them, reranked: false }.

    rerankWithin(query, chunks, 1) is { chunks: chunks.slice(0, 5), reranked: false } and does not throw. Only a timeout falls back: rerankWithin('', chunks, 20000) must still reject, because the empty query is a 400.

    Arjun will ask: the reranker adds about two seconds to every question here. What does the merchant get when it is slow?

What you will learn

  • Bi-encoders and cross-encoders: why reading together ranks better, and why it costs more.
  • The rerank endpoint: a POST with fetch, and the errors fetch does not throw.
  • Retrieve many, keep few: 50 candidates in, 5 out, in batches.
  • When it pays: MRR, hit@1 and milliseconds, measured, and a deadline for when it is slow.

More in Better retrieval

Build "A second opinion" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.