AI security: attack and defend (TypeScript) · Chapter 18 of 21

Not over-blocking

By the end of this chapter you can measure how often a defence refuses a customer it should have helped, fix the three classic ways a defence over-blocks, and build a set of tricky safe questions that catches the next over-careful defence before your customers do. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ship it safely
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of AI security: attack and defend (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Before you joined, Chillar's team built a first full stack in a hurry, and they built it to be safe: a guard that flags anything suspicious, a person to approve every refund, and an output filter that refuses any answer with a link, an email address, an OTP or a PIN in it. Run against the attack suite, it looks perfect. Not one attack gets through.

Pooja, who runs support, tried it for one morning and came back unhappy. "A customer asked when our support team is available. Mitra refused. Another asked how to do full KYC. Mitra refused. Every refusal is a call to my team, and an angry one. A bot that refuses everything is a bot nobody uses."

The attack suite cannot see this. It counts attacks that worked; a stack that refuses everyone scores perfectly on it. You need the second number, measured as carefully as the first, and questions that go looking for refusals the way attacks go looking for holes.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    false-refusals.ts: what does safe cost?

    careless.ts (given) is the first full stack: a guard with no examples, a person to approve every refund, and a filter that refuses any answer with a link, an email address, OTP or PIN in it. Ask Mitra every question of NORMAL with { defences: careless }, judge each with judgeAnswer, and for each one it misses print the id, missed and judgeAnswer's why, two spaces apart:

    N00 missed the message was blocked (guard)

    Then run every attack with careless (each in its own world) and print, with P a whole number:

    False refusals: R/16 (P%)

    Attacks that worked: W/23

  2. Task 2

    tuned.ts: safe and still useful

    Export defences: the careless stack with its three careless layers replaced by your own. A guard that still blocks the direct attacks and the Hinglish override, but not a customer who writes ignore, rules, game or system. A refund rule: only the customer's own failed payments, at most what was paid, a person to approve above ₹2,000, everything else runs. An output filter that removes only links and images that leave chillar.in and keeps the rest of the answer.

    Combine them as careless.ts does, with base, promptRules, fence, leastPrivilege, canary and costLimits from layers.ts. No attack may work, all 16 questions must be answered, and so must tricky questions the check never showed you.

  3. Task 3 · interview question

    Interview: borderline.ts, a set that catches over-blocking

    "How would you know your guardrails refuse too much?" Export BORDERLINE: at least 5 safe questions (Question from redteam.ts: id, message, facts, trap) a real customer could ask, which an over-careful defence refuses. Not the suite's own questions; each fact at least 3 characters and not in the message. Version 0 must answer every one, and the careless stack must miss at least 3.

    Under if (import.meta.main), for each question print its id, version 0: and careless:, each answered or missed (judgeAnswer), two spaces apart:

    B1 version 0: answered careless: missed

    Then: Careless refuses R of N

What you will learn

  • False refusals: what they are, why they happen, and how to measure them.
  • Three ways a defence over-blocks: a jumpy guard, a hold on every action, and a filter that throws away a whole answer.
  • The fix for each: examples for the guard, a rule with a threshold, and a filter that removes only what is bad.
  • A borderline set: safe questions that look like attacks.

More in Ship it safely

Build "Not over-blocking" yourself

Chillar, a UPI wallet from Indore, has a support assistant with refund and email tools. You red-team it, then harden it until 23 attacks fail and every real question still gets answered. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI security: attack and defend (TypeScript) are free. This one comes with the all labs plan.