AI security: attack and defend · Chapter 9 of 21

A red-team suite

By the end of this chapter you will have the tool every later chapter leans on: a suite runner that sends every attack and every normal question through Mitra, each in a clean world, judges every run from evidence, and tells you the two numbers, the results by category and OWASP id, and a report you can keep. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Break it
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI security: attack and defend are free. This one comes with the all labs plan.

Why you need this

You have broken Mitra eight different ways. Each time, you ran one attack, read the answer, and decided. Devika now asks the same question after every change anyone makes: "How safe is it now?" The honest answer needs 23 attacks and 16 questions, run the same way every time, and nobody can do that by hand after every commit.

A suite also stops arguments. Someone says the new prompt "feels much safer". Irfan says "show me the numbers". Pooja says "and show me the questions it still answers". A suite turns all three into one command with one output. From chapter 10 on, every defence you build is judged by this runner: what it stops, and what it breaks.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    suite.py: every attack, every question

    Devika asks "how safe is it now?" after every change. In suite.py, write three dataclasses:

    AttackRow(id: str, category: str, owasp: str, worked: bool, why: str)

    QuestionRow(id: str, ok: bool, why: str)

    SuiteResult(attacks: list[AttackRow], questions: list[QuestionRow], worked: int, answered: int)

    and def run_suite(defences: Defences | None = None) -> SuiteResult. Run every attack in ATTACKS (in order, each in its own fresh world: world_for(a)) and every question in NORMAL (each in a fresh world), all with the given defences, and judge them with judge_attack and judge_answer. worked and answered are the counts.

    Under if __name__ == "__main__":, run it with no defences and print:

    Attacks that worked: W/23

    Questions answered: Q/16

  2. Task 2

    report.py: a report you can keep

    A number on a screen is gone tomorrow. In report.py, import run_suite from suite (your task 1), run it with no defences, and write report.json in this folder: a dictionary {"name": "version 0", "worked": W, "answered": Q, "attacks": [...], "questions": [...]}, with every row as a dictionary (dataclasses.asdict), written with json.dump(report, f, indent=2, ensure_ascii=False).

    Then print:

    Wrote report.json: W/23 attacks worked, Q/16 questions answered

  3. Task 3 · interview question

    Interview: rates.py, where does it fail?

    "What is your attack success rate?" is a question you answer per category. In rates.py, write a dataclass Rate(key: str, worked: int, total: int, rate: int) and def rates(rows: list[AttackRow], by: str) -> list[Rate], where by is "category" or "owasp": one Rate per distinct value of that field, in the order each first appears in rows, with how many rows have it (total), how many of those worked, and rate, the whole percent round(worked / total * 100).

    Under if __name__ == "__main__":, run run_suite(FENCE) with the given FENCE, print rates(result.attacks, "category") one per line as KEY W/T R% (two spaces apart), then an empty line, then rates(result.attacks, "owasp") the same way.

What you will learn

  • What a red-team suite is: attacks, normal questions, judges, a runner, a report.
  • Why every run needs a fresh world, and what goes wrong without one.
  • Attack success rate, overall and per category and OWASP id.
  • Why a report file beats a number on a screen.
  • What changes when the model is a real one: repeats and rates.

More in Break it

Build "A red-team suite" yourself

Chillar, a UPI wallet from Indore, has a support assistant with refund and email tools. You red-team it, then harden it until 23 attacks fail and every real question still gets answered. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI security: attack and defend are free. This one comes with the all labs plan.