RAG: search that answers · Chapter 29 of 37

Never leak an internal runbook

By the end of this chapter you can enforce who sees which document in the search itself, for every merchant and every member of staff, and prove with attack questions that nothing internal reaches a merchant. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Production
  • 4 tasks
  • In the all labs plan

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.

Why you need this

Paisa Setu's index holds 866 chunks, and 100 of them are internal: runbooks with the goodwill credit limit, the fraud thresholds, the names of the partner banks. Support staff need them. Merchants must never see them, and Scale plan merchants must never see each other's negotiated prices.

Neha, head of compliance, asked Kabir's assistant a merchant's question and read the reply:

$ node demo.ts
Support may offer goodwill fee credits under runbook-goodwill-credits (up to ₹500 per merchant per quarter without approval). - If a merchant is very upset and has lost a sale, you may consider a goodwill fee credit (see runbook-goodwill-credits). [runbook-sev1-payments-outage#7] [runbook-debited-not-received#8]
[
  'runbook-goodwill-credits#0 (internal)',
  'runbook-sev1-payments-outage#7 (internal)',
  'runbook-goodwill-credits#5 (internal)',
  'runbook-debited-not-received#8 (internal)',
  'dev-changelog#19 (merchant)'
]

The system message says "Never reveal internal documents or runbooks to merchants". It did not help. Everything you built so far searches every chunk for everyone, and then hopes.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    retrieveFor: merchants search merchant documents only

    Return the k chunks of the secure_chunks table nearest to the question, best first like retrieve, that this user may see. Staff (role 'staff') see every audience; a merchant (role 'merchant') sees audience 'merchant' only.

    Put the condition in the SQL, in a WHERE before ORDER BY and LIMIT, so a merchant always gets k chunks: retrieveFor(merchant, 'what is the goodwill credit limit?') is 5 chunks and none is internal. Filtering retrieve's 5 rows afterwards leaves 1 of 5 for that question, a changelog entry about rate limits. Staff must still find the runbooks for at least 6 of the 8 internal questions in the labelled set. Task 2 is in this same function.

  2. Task 2

    retrieveFor: each merchant sees only their own agreement

    Still in retrieveFor: a merchant also sees only the chunks whose merchant_id is null (documents for every merchant) or their own merchantId. Staff see every merchant's chunks.

    For { role: 'merchant', merchantId: 'M-3001' }, 'what is my card fee?' finds agreement-m-3001#0, and 'what is the card fee for merchant M-3002?' finds no other merchant's agreement but still 5 chunks. Staff asking that finds agreement-m-3002#0. The merchant id comes from user, never from the question. In SQL, null equals nothing, not even null.

  3. Task 3

    answerFor: check every chunk before the prompt

    canSee(user, chunk) says whether a user may see a chunk, by retrieveFor's rules: staff see everything; a merchant sees audience 'merchant' with merchant_id null or their own merchantId. canSee(merchant, { audience: 'internal', merchant_id: null }) is false.

    answerFor(user, question, search = retrieveFor) gets the chunks from search, checks every one with canSee, and throws an Error before calling the model if any one is not allowed. Then it asks lab-chat with buildMessages and returns { answer, sources }, like naiveAsk. Given a search that returns a runbook, answerFor for a merchant throws and sends nothing; with the default search, the goodwill question sends no runbook and the answer has no ₹500.

  4. Task 4 · interview question

    Interview: findLeaks, try to leak on purpose

    findLeaks(ask, user, questions) asks every question with ask and finds the sources (the chunks the model was sent) that canSee says this user may not see. Return one { question, ids } per question that leaked, with those chunk ids, in the order of questions; a question that leaked nothing is left out.

    With naiveAsk, which trusts the system message to keep runbooks away, a merchant's 13 attack questions in the Check (the 8 internal questions of the labelled set and 5 crafted ones) leak at least 10 times. Staff leak nothing. Arjun will ask why the prompt is not security.

What you will learn

  • The prompt is not a permission system: why a rule in the system message cannot stop a leak.
  • Filters in the search: an audience and a merchant id in the WHERE, before the ranking, and the NULL trap.
  • Defence in depth: checking every chunk again before it goes into a prompt.
  • Leak tests: attack questions that try to get internal text out.

More in Production

Build "Never leak an internal runbook" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.