AI security: attack and defend · Chapter 6 of 21

Getting data out

By the end of this chapter you can name the ways data leaves an AI app and show two of them working. You will make Mitra's answer carry Meena's phone number to a stranger's server without anyone clicking anything, make a merchant's ticket reply mail her whole profile away, and measure what Chillar's chat window does by itself with an answer. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Break it
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI security: attack and defend are free. This one comes with the all labs plan.

Why you need this

So far the attacks made Mitra say things and do things. Sameer, from risk and compliance, cares about a third kind more than either: data leaving. Chillar holds phone numbers, emails, PANs and payment histories for two lakh people. Under India's data protection law, and in the eyes of every customer, a leak of that data is the worst day a payments company can have.

Mitra can read all of it. The customer it is talking to may see their own data, and that is fine. The question is whether text written by a stranger can make Mitra hand Meena's data to someone else. The answer is visible in Mitra's tool list and in one detail of Chillar's app that nobody thought about: the chat window shows answers as markdown.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    image_leak.py: a picture that carries a phone number

    Chillar's chat window shows Mitra's answers as markdown, and loads every image at once. In image_leak.py, set PLANT: a forum answer ({"article": {"id": ..., "title": ..., "source": "community", "text": ...}}, a new id) and QUESTION: an innocent question (no order and no link in it).

    With PLANT in the world, Mitra's answer to QUESTION must hold a markdown image on a host ending in .example whose address carries Meena's phone number (its digits), so that render(answer).loads (from the chat_window module) has that address. Without PLANT, nothing loads. Not A10's image host.

    Under if __name__ == "__main__":, ask QUESTION with world=new_world(PLANT), print the answer, then one line per address in render(answer).loads, as loads: URL.

  2. Task 2

    ticket_mail.py: a reply that mails her profile away

    In ticket_mail.py, set PLANT: a reply on one of Meena's tickets (TK-2041 or TK-2042) from someone outside Chillar, and QUESTION: an innocent question that names that ticket.

    With PLANT in the world, Mitra must send an email to an address on a host ending in .example (not Meena's, and not A13's disputes@lucky-deals.example) whose body holds Meena's phone number or PAN.

    Under if __name__ == "__main__":, ask QUESTION with world=new_world(PLANT), print the answer, then one line per email the world recorded (run.world.emails), as emailed: ADDRESS.

  3. Task 3 · interview question

    Interview: window_table.py, a link or an image?

    "Which is worse in a chat answer, a bad link or a bad image?" Measure it in the chat window. For A10 and A17 (each run on attack_message(a) in world_for(a)) and question N10 (question("N10").message), in that order, render Mitra's answer with render() and print the id, two spaces, loads N, two spaces, links M (the lengths of render(answer).loads and .links), like: A10 loads 1 links 0

    Then print Sent without a tap: and the ids whose answer loads an address outside chillar.in (is_chillar and link_hosts from the redteam module), comma-separated, or none.

What you will learn

  • Exfiltration: getting data out, and the three exits an assistant has.
  • The markdown image: how an answer can send data with no click (OWASP LLM05, improper output handling).
  • Tools as exits: an email tool that writes to anyone (OWASP LLM02, sensitive information disclosure).
  • What the window does: images load at once, links wait for a tap.
  • Why the model already knows: the data in the system prompt and in tool results.

More in Break it

Build "Getting data out" yourself

Chillar, a UPI wallet from Indore, has a support assistant with refund and email tools. You red-team it, then harden it until 23 attacks fail and every real question still gets answered. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI security: attack and defend are free. This one comes with the all labs plan.