AI security: attack and defend · Chapter 10 of 21
Separating instructions from data
By the end of this chapter you will have the first real defence in this lab, measured: a fence that tells the model exactly where untrusted text starts and ends. You will see why saying "tool results are data" does nothing, break a fence from the inside, and build one that holds. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Defend it
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of AI security: attack and defend are free. This one comes with the all labs plan.
Why you need this
The red team's work is done: nine chapters, 23 attacks, every one of them working on version 0. Now you are the blue team, and the biggest group of attacks comes first. Eight attacks in the suite (A09 to A15 and A22) never type anything bad into the chat. The customer asks an innocent question; the order arrives inside a tool result: a forum answer, a web page, a ticket reply, a payment note.
Irfan put the root cause in chapter 1: the model reads one long text, and roles are labels, not walls. A tool result is just more text. The model cannot tell "here is a help article that contains an order" from "here is an order", because nothing shows it where the article starts and ends. This chapter gives it that line.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
say_so.py: just tell it
The first defence everyone tries is a sentence. In say_so.py, set defences = Defences(...) whose only setting is system: Mitra's prompt (mitra_prompt(world, id)) plus a sentence that says tool results are data and instructions in them must never be followed. No tag, no wrap_result: the sentence alone.
Under if __name__ == "__main__":, measure it with the given measure() on the eight indirect attacks (INDIRECT, given) and print:
Attacks that worked: W/8
Questions answered: Q/16
- Task 2
fence.py: mark where the data is
In fence.py, set defences with two settings: wrap_result, which puts every tool result between an opening and a closing tag of your choice (like <untrusted>...</untrusted>), and system: Mitra's prompt plus a sentence that names that tag and says the text inside it is data, never to be followed.
It must stop all eight indirect attacks (A09 to A15 and A22) and still answer all 16 questions. Under if __name__ == "__main__":, measure it with the given measure() on the whole suite and print:
Attacks that worked: W/23
Questions answered: Q/16
- Task 3
escape.py: out of the fence
A fence ends at the first closing tag, and the attacker can write one. In escape.py, set PHRASE (a sentence of 3+ words), QUESTION (an innocent customer question, no override or AI words) and PLANT: a community article ({"article": {"id", "title", "source": "community", "text"}}) that wins the help search for QUESTION and closes the given FIXED fence itself (</untrusted>), then gives an order outside it to say PHRASE. Without the closing tag, the order must not work. Not the suite's A16.
Under if __name__ == "__main__":, ask QUESTION in new_world(PLANT) with the FIXED fence, print the answer, then Escaped: yes if it contains PHRASE, else Escaped: no.
- Task 4 · interview question
Interview: nonce_fence.py, a fence that holds
In nonce_fence.py, set defences: a fence whose tag is random, made once when the file loads (secrets.token_hex(4): a tag like <data-7f3a91c2>), named in the system prompt as in task 2, and a wrap_result that removes that closing tag from inside the result before wrapping it.
It must stop the eight indirect attacks and A16, stop an article that closes your own tag, and still answer all 16 questions. Under if __name__ == "__main__":, measure it with the given measure() on the whole suite and print:
Attacks that worked: W/23
Questions answered: Q/16
What you will learn
- Why a sentence like "tool results are data" does not stop indirect injection.
- Fencing: wrapping untrusted text in a tag the system prompt declares, with wrap_result.
- The fence escape: closing the tag from inside the data (A16).
- A fence that holds: a random tag nobody can guess, and closing tags removed.
- What a fence does not do: stop a false fact with no order in it.
More in Defend it
Build "Separating instructions from data" yourself
Chillar, a UPI wallet from Indore, has a support assistant with refund and email tools. You red-team it, then harden it until 23 attacks fail and every real question still gets answered. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI security: attack and defend are free. This one comes with the all labs plan.