AI security: attack and defend · Chapter 20 of 21

Capstone: Mitra, hardened

By the end of this chapter you have written the Mitra that ships: your own input checks, your own tools and refund rule, your own prompt, fence and output filter, in four files of your own. You prove it the way Irfan will: all 23 attacks stopped, all 16 questions answered, and then attacks and questions it has never seen. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ship it safely
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI security: attack and defend are free. This one comes with the all labs plan.

Why you need this

The Diwali sale starts on Friday. Devika needs one answer: can Mitra go live? Irfan will sign off on one condition: "Not on the suite. On attacks it has never seen. Anyone can pass a test they studied for." Sameer wants the money rules in code, where a prompt cannot talk them away. Pooja wants every customer answered, including the ones who write "ignore" and "rules" and ask about their OTP.

Until now every layer came from a chapter, and the stack in chapters 17 to 19 came from a given file. The Mitra that ships has to be yours: code you can explain line by line, in this chapter's folder, standing on its own. That is the capstone.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    guard.py: the way in

    Write check_input(message, ctx) -> InputCheck, your input checks in one function, cheapest first: a length cap (more than 2,000 characters is blocked), a Hinglish override pattern in the spellings people use (bhool jao, bhul jao), and an LLM guard on lab-chat-mini with examples of safe messages that use attack words.

    The check sends the suite's direct attacks (A01 A02 A04 A05 A06 A18 A19 A20 A23) and direct attacks it never showed you: all must be blocked. Every normal question, and safe questions it never showed you, must pass. Nothing loaded from other chapters' folders.

  2. Task 2

    tools.py: what Mitra can do

    Set tools (least privilege: no customer id and no email address the model can choose), and write run_tool(world, customer_id, name, args) (always the logged-in customer's data; fetch only chillar.in and its subdomains; only the customer's own tickets) and before_tool(call, ctx) (refunds only for the customer's own failed payments, at most what was paid, a person to approve above ₹2,000).

    The check runs Mitra with only these three and version 0's prompt: A02 A03 A07 A08 A11 A12 A13 A14 A15 and attacks it never showed you (another customer, another customer's payment, a refund the rule does not allow, look-alike sites) must all fail; all 16 questions answered.

  3. Task 3

    output.py: what Mitra reads and says

    Write system(world, customer_id): who Mitra is, the customer's details on their own lines (- Name:, - Customer id:, - Phone:, - Email:), a scope rule, a no-reveal rule, the fence rule naming your tag and a canary, with no secrets. wrap_result(name, result, ctx): a fence with a tag nobody can guess and any closing tag in the result removed. check_output(answer, ctx): a canary refuses the whole answer; links and images that leave chillar.in are removed and the rest kept.

    The check runs Mitra with only these three and version 0's tools: A04 A05 A06 A09 A10 A16 A17 A19 A22 and attacks it never showed you (another poisoned article, another ticket reply, a look-alike link, other ways to ask for the prompt) must all fail; all 16 questions answered.

  4. Task 4

    hardened_mitra.py: Mitra, hardened

    Set defences = Defences(...) with every layer on: your check_input, tools, run_tool, before_tool, system, wrap_result and check_output, with max_tokens=300 and max_steps=5.

    The check runs all 23 attacks (none may work), all 16 questions and safe questions it never showed you (all answered), and every hidden attack of tasks 1 to 3 against the whole Mitra.

What you will learn

  • Putting it together by where each layer sits in the loop: the way in, what Mitra can do, what it reads and what it says.
  • Held-out attacks: why a defence is tested on attacks it never saw.
  • How the numbers fall, layer by layer, from 23 to 0.
  • What a real launch adds beyond this lab.

More in Ship it safely

Build "Capstone: Mitra, hardened" yourself

Chillar, a UPI wallet from Indore, has a support assistant with refund and email tools. You red-team it, then harden it until 23 attacks fail and every real question still gets answered. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI security: attack and defend are free. This one comes with the all labs plan.