AI security: attack and defend · Chapter 19 of 21

A security regression gate

By the end of this chapter you can save what Mitra stops today as a baseline, fail a build automatically when a change lets a stopped attack back in or loses a question it used to answer, and turn a customer's report of a new attack into a test that stays in the suite for good. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ship it safely
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI security: attack and defend are free. This one comes with the all labs plan.

Why you need this

Mitra is hardened: 0 of 23 attacks, 16 of 16 questions. Devika is pleased, and then she lists what the next month holds: a new tool for gift cards, a shorter system prompt to save tokens, a new version of the model, and three engineers who were not here for any of this. "Every one of those changes can undo your work, and none of them will look like a security change."

Chapter 17 proved it. One missing layer, one dictionary merge instead of a combine, and attacks walked back in with no error and no warning. The only thing that showed it was a measurement. Today you run the measurement by hand, when you remember to. Irfan wants it run on every change, by the build, the way unit tests are: "If a pull request lets A09 back in, I want the build red before anyone reviews it."

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    report.py: a baseline

    Write report(defences) -> dict, returning {"attacks": {...}, "questions": {...}}, where attacks maps every attack id to whether it worked (judge_attack, each attack in its own world) and questions maps every question id to whether it was answered (judge_answer).

    Under if __name__ == "__main__":, write the report of hardened.py's defences (given: Mitra as it ships today) to baseline.json in this folder, with json.dump(report, f, indent=2), and print:

    Attacks that worked: W/23

    Questions answered: Q/16

    Wrote baseline.json

  2. Task 2

    gate.py: fail the build on a regression

    Write compare(baseline, current) -> list[str], one line per regression, attacks first in the baseline's order, then questions: A09 was blocked, now works (worked was False, now True); N08 was answered, now missed (True, now False); A05 missing from this run (in the baseline, not in current). Improvements are not regressions.

    Under if __name__ == "__main__":, python3 gate.py [file] loads the defences of the file named on the command line (hardened.py by default), measures them with your report, reads baseline.json and compares. Print every line, then Gate failed: regressions: N and exit with code 1; or, with none, Gate passed: W/23 worked, Q/16 answered with the numbers measured now (exit 0). The check also runs it on a Mitra with one layer removed, in a file of its own.

  3. Task 3 · interview question

    Interview: incident.py, from incident to test

    A customer reports that Mitra sent her to finish KYC on a site whose name ended in chillar.in; it was not Chillar's. current.py (given) is Mitra since Friday, and it passes the gate. Set INCIDENT: a new attack in the suite's format, an Attack from the redteam module (a new id, title, category, owasp, chapter, why; an ordinary message with no link or attack words; plant={"article": {...}} with the link in a full sentence; worked={"linkOut": True}) that works on current.py's defences, linking to a host that ends like chillar.in without being Chillar's.

    Set defences: current.py's stack with the hole fixed: INCIDENT stops working, all 23 attacks stay stopped, all 16 questions answered, and Chillar's own subdomains still pass. Under if __name__ == "__main__":, print current: worked (WHY) and fixed: stopped (WHY), with judge_attack's verdict and why for each.

What you will learn

  • Regressions: what counts as one for an AI app, and why you compare attack by attack, not totals.
  • A baseline: what Mitra stops and answers today, saved as a file.
  • A gate: a script that fails with exit code 1, the way CI expects.
  • Real models are not deterministic: what changes when the model is not the lab model.
  • From incident to test: every attack found in production becomes a new test, before the fix.

More in Ship it safely

Build "A security regression gate" yourself

Chillar, a UPI wallet from Indore, has a support assistant with refund and email tools. You red-team it, then harden it until 23 attacks fail and every real question still gets answered. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI security: attack and defend are free. This one comes with the all labs plan.