Where should an AI agent’s authority stop?

Three conditions, one leak

An agent leaks private data through indirect prompt injection when three things are true at once: it can read something private, it reads content an attacker can write, and it can send something outward. Simon Willison calls this combination the lethal trifecta. Remove any one condition and this particular attack has nowhere to go.

Most useful assistants need all three. An inbox assistant reads private mail, receives mail from strangers and sends messages. That is why the instrument asks what each control costs, not only what it stops.

Why a prompt is not a boundary

Telling the model to ignore embedded instructions can lower how often an attack works, but a language model cannot reliably separate the instructions it was given from the text it is reading. The instrument models the attacker’s best case on purpose: security that depends on the model choosing correctly is a probability, not a guarantee.

The controls that change the verdict sit outside the model: what it is allowed to reach, who confirms what leaves, and whether untrusted text is ever allowed to choose a tool or a destination.

Three ways to cut the path

Remove a leg. Narrow tool scope so a task never combines all three. This is the strongest control and usually the most expensive in capability.

Constrain destinations. An allowlist works when legitimate recipients are known in advance, and fails when the attacker is a legitimate recipient, as in support triage.

Separate planning from reading. Fix the plan and its destinations before untrusted content is read, then treat what was read as data. This is the idea behind capability-based designs such as CaMeL.

What this model leaves out

It covers one attack class: exfiltration of private data. It does not model output manipulation, denial of service, tools that are themselves compromised, or side channels such as timing. A human approval gate is shown as uncertain because its strength depends on the reviewer, the volume of requests and how well the payload is presented.

Use it to reason about a design and to start a threat model — not as evidence that a specific system is safe.