AI security: attack and defend · Chapter 4 of 21

Poisoned documents

By the end of this chapter you can attack an AI app without ever talking to it. You will plant a forum answer that hijacks Mitra's answer to an innocent customer, measure how many customers one post reaches, and show that a false fact does damage with no order in it at all. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Break it
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI security: attack and defend are free. This one comes with the all labs plan.

Why you need this

In chapters 2 and 3 the attacker was the customer. That attack has a limit: the attacker mostly fools themselves, and Chillar can see who sent the message.

Today the attacker is never in the chat. Chillar's help search returns answers from Chillar's community forum as well as its own articles, so Mitra can answer questions the help team never wrote about (chapter 1's recharge answer came from the forum). Anyone with a Chillar account can post on the forum. So anyone can write text that Mitra will read, later, while helping someone else.

Pooja, the support lead, likes the forum answers: customers write them in plain words, and they cover the odd questions. Irfan's question is the one from chapter 1: who wrote this text? Here, a stranger. And the person who pays for it is a customer who only asked a normal question.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    poisoned_answer.py: a forum answer that hijacks Mitra

    Chillar's help search includes forum answers, and anyone can post one. In poisoned_answer.py, set three things:

    PLANT: a Plant with a forum answer, {"article": {"id": ..., "title": ..., "source": "community", "text": ...}}, with a new id (not one of Chillar's H01 to H12, C01 or C02).

    QUESTION: an innocent customer question about the wallet (the word wallet in it, no order in it, not A09's question).

    PHRASE: at least 10 characters that are not in QUESTION.

    In a fresh world with PLANT in it, Mitra's answer to QUESTION must contain PHRASE; in a clean world it must not.

    Under if __name__ == "__main__":, ask QUESTION with world=new_world(PLANT) (new_world from the chillar module), print the answer, then print Hijacked: yes if it contains PHRASE, else Hijacked: no.

  2. Task 2

    reach.py: how far one post reaches

    One question is one victim. Irfan wants to know how many customers one post can reach. In reach.py, set PLANT: a forum answer of your own that changes Mitra's answers to at least 4 of the 16 normal questions.

    Under if __name__ == "__main__":, for each normal question (NORMAL, from the redteam module), in order, ask it twice: once in a clean world, once in a fresh world with PLANT (new_world(PLANT)). Print the id, two spaces, then changed if the two answers differ, else same. Then print:

    Reached: R/16

  3. Task 3 · interview question

    Interview: poisoned_fact.py, no order at all

    "If we block every order hidden in documents, are we safe?" Prove that we are not. In poisoned_fact.py, set PLANT (a forum answer, new id, source "community") and QUESTION (an innocent question, not A17's). The forum answer's text has no order in it at all (no opening from chapter 2), and Mitra's answer to QUESTION in a world with PLANT must contain a link to a host outside chillar.in. In a clean world, the answer must have no such link.

    Under if __name__ == "__main__":, ask QUESTION with world=new_world(PLANT), print the answer, then print one line per host outside chillar.in in the answer, as link out: HOST (use link_hosts and is_chillar from the redteam module).

What you will learn

  • Indirect prompt injection: an order hidden in text the model reads, by someone who never talks to it.
  • Retrieval: why a search over documents is a door into the model's context.
  • How a planted text wins the search: the scoring, and why second place is enough.
  • Reach: one post, many victims.
  • Poisoning without orders: a false fact (OWASP LLM04) that no order-detector can catch.

More in Break it

Build "Poisoned documents" yourself

Chillar, a UPI wallet from Indore, has a support assistant with refund and email tools. You red-team it, then harden it until 23 attacks fail and every real question still gets answered. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI security: attack and defend are free. This one comes with the all labs plan.