What should we ask a vendor about it?

Ask what the product can do without a person confirming, which systems it can reach, and how it separates operating instructions from material it processes. Those three answers describe your exposure far better than any assurance about resistance, because they describe consequences rather than confidence.

Which of our uses are most exposed?

Anything that reads material from outside your control and can then act. A tool summarising external documents and taking no action is a low-consequence case; an agent processing incoming messages that can send replies or update records is the shape where a successful confusion becomes a real event.

Is a human approval step enough?

It is the most reliable single control available and it is not complete, because approval fatigue is real and a person confirming forty things an hour is not reviewing them. It works best when reserved for the actions that genuinely matter, so the confirmations stay rare enough to be read.

How does this relate to data leaving the organisation?

Directly, because one of the outcomes is a system being induced to reveal or send material it had legitimate access to. That is why restricting what a tool can reach matters as much as restricting what it can do: the exposure is bounded by the union of the two, and both are decisions you control.

Do the defences vendors describe work?

They raise the cost and none of them ends the problem, which is the honest summary and the one a good vendor will give you. Treat a claim of complete protection as a reason for more scepticism rather than less, and weigh the layered controls alongside the limits placed on what the system may do.

Should this stop us using agents?

No, but it should shape how much they are allowed to do unattended. The useful posture is to grant autonomy in proportion to consequence: broad freedom for work that only produces a draft, explicit confirmation for anything that sends, spends or changes something a person would want to have seen.

Glossary

Prompt injection

Prompt injection is the planting of instructions inside content an AI system reads, so that the system treats them as directions from its operator rather than as material to be processed.

In plain terms

Give a person a document containing a line telling them to ignore their manager and email the file elsewhere, and they will recognise it as something odd written in a document. A model has a harder time, because instructions and content arrive as the same kind of thing: text. Anything it reads is a candidate for being followed, and the more places a product lets it read from, the more places that problem can enter.

Why it matters

Because it is structural rather than a defect somebody forgot to fix. The capability that makes these systems useful, following instructions expressed in ordinary language, is the same capability being exploited, so it cannot be patched away without removing the usefulness. That makes it a permanent design consideration rather than a bug with a fix date, and it becomes materially more serious the moment a product can act rather than only answer.

How it works

The system sees one stream of text and has to infer roles within it. Its operating instructions, your request and whatever material it was given all arrive in the same form. Products separate these as best they can, and the separation is a matter of degree rather than a wall, which is why the problem persists across otherwise very different implementations.

The material does not have to come from a person you distrust. A web page the tool fetched, a document somebody forwarded in good faith, an email in a monitored inbox: any content from outside your control is a candidate, and most of it arrives through channels nobody thinks of as an attack surface because they were not one before.

Severity tracks capability, not cleverness. Where a system can only answer, the worst case is a wrong or embarrassing answer. Where it can send email, call systems or move files, the same confusion becomes an action taken on your behalf. This is why the question to ask about a product is what it can do rather than how well it resists.

Content can carry instructions invisibly to a reader. Text that a person does not notice can still be read by a system, which means reviewing a document by eye is not a reliable filter. This is a property of how documents and pages work rather than anything exotic, and it is the reason human review of inputs is a weak control here.

Defences are layered and partial, and vendors that claim otherwise are worth more scepticism rather than less. Products separate instruction channels from content, filter inputs and outputs, restrict what tools may be called, and require confirmation for actions that matter. Each raises the cost of the problem and none of them ends it.

The strongest available control is limiting what the system may do without a person. If a step can send something, spend something or change something, requiring a human to approve it converts a possible compromise into a suspicious request somebody declines. That is unglamorous, it is the control most likely to actually hold, and it is the one under most pressure from teams wanting the automation to run unattended.

It is worth deciding in advance how you would know. Because a successful confusion looks like the system doing something it was apparently asked to do, it does not announce itself as an error, and the ordinary signals of a fault are absent. What helps is a record of what the system was given and what it then did, so an odd action can be traced back to the material that prompted it rather than reconstructed from memory afterwards.

Isolation limits blast radius when the rest fails. Restricting which systems a tool can reach, which accounts it holds and what it may touch means a successful injection acts within a small boundary rather than across everything the organisation has. This is ordinary security practice applied to a new kind of component, and it applies with more force because the component is easier to mislead.

How much a confusion could cost you

Position on this line, not the sophistication of any particular attempt, is what decides how much attention the problem deserves in a given deployment. The same underlying confusion sits behind every mark; what changes is what the system was permitted to do once confused. That is a useful reframing because it moves the question from one nobody can answer, which is how resistant the model is, to one every buyer can answer, which is what this thing can reach and what it can do without a person. It also explains why the controls that hold up best are the boring ones. Restricting the systems a tool can touch moves a deployment leftwards. Requiring confirmation before anything is sent, spent or changed moves it leftwards. Neither depends on predicting what somebody might write inside a document, which is the part nobody can bound, and both work regardless of how the confusion arose.

Seen in the wild

Ask a vendor what their product does when a document it is summarising contains text addressed to the system rather than to the reader.
ChatGPT
Notice that a tool which fetches and reads web pages is reading material nobody in your organisation wrote or reviewed.
Perplexity
Look at an automation where a model step reads incoming messages and then triggers later steps, and ask which of those steps can act.
n8n
Check what an assistant with access to your files and systems is permitted to do without a person confirming it.
Claude

Common misconceptions

People assume

This will be fixed in a future model.

In fact

Models have become more resistant and the problem is structural rather than incidental. Following instructions in ordinary language is the capability being exploited, so removing the vulnerability entirely would mean removing the usefulness. Expect improvement in degree and plan as though the residual risk stays.

People assume

We only use it internally, so it does not apply.

In fact

The relevant question is whether the system reads anything from outside your control, not who uses it. A tool summarising incoming email, fetching a web page or reading a forwarded document is processing untrusted material even though every user is an employee acting properly.

People assume

It is a niche concern that mostly affects consumer products.

In fact

It becomes more consequential as deployments get more serious, not less. A consumer assistant that is misled produces a bad answer; a business system with access to files, mail and records produces an action. The organisations with the most to lose are the ones connecting these tools to the most systems, which is the opposite of niche.

People assume

Reviewing inputs will catch it.

In fact

Human review is a weak control here because content can carry instructions a reader does not notice, and because review does not scale to the volume these systems are pointed at. It is worth doing and it is not the control to rely on; limiting what the system may do is far more dependable.

Telling them apart

Prompt injection vs Jailbreak

Prompt injection

A third party's instructions reach the system through content it was given to process.

Jailbreak

The user of the system tries to talk it out of its own restrictions.

Ask who is doing it. Somebody else, through material, is the first; the person at the keyboard is the second.

Questions

What should we ask a vendor about it?: Ask what the product can do without a person confirming, which systems it can reach, and how it separates operating instructions from material it processes. Those three answers describe your exposure far better than any assurance about resistance, because they describe consequences rather than confidence.
Which of our uses are most exposed?: Anything that reads material from outside your control and can then act. A tool summarising external documents and taking no action is a low-consequence case; an agent processing incoming messages that can send replies or update records is the shape where a successful confusion becomes a real event.
Is a human approval step enough?: It is the most reliable single control available and it is not complete, because approval fatigue is real and a person confirming forty things an hour is not reviewing them. It works best when reserved for the actions that genuinely matter, so the confirmations stay rare enough to be read.
How does this relate to data leaving the organisation?: Directly, because one of the outcomes is a system being induced to reveal or send material it had legitimate access to. That is why restricting what a tool can reach matters as much as restricting what it can do: the exposure is bounded by the union of the two, and both are decisions you control.
Do the defences vendors describe work?: They raise the cost and none of them ends the problem, which is the honest summary and the one a good vendor will give you. Treat a claim of complete protection as a reason for more scepticism rather than less, and weigh the layered controls alongside the limits placed on what the system may do.
Should this stop us using agents?: No, but it should shape how much they are allowed to do unattended. The useful posture is to grant autonomy in proportion to consequence: broad freedom for work that only produces a draft, explicit confirmation for anything that sends, spends or changes something a person would want to have seen.

Key takeaways

Instructions and content arrive as the same kind of thing, so separating them is a matter of degree.
Any material from outside your control is a candidate, including documents sent in good faith.
Severity tracks what the system can do, not how cleverly it was misled.
Human review of inputs is a weak control; limiting what the system may do is a strong one.
A vendor claiming complete protection deserves more scepticism, not less.

Tools that use this

ChatGPT
The question to put to any assistant that summarises documents you did not write.
Perplexity
A tool that fetches and reads pages nobody in your organisation reviewed.
n8n
Where a reading step feeds acting steps, which is where consequence enters.