What should we ask a vendor about theirs?

What categories they test, how often, whether testing is repeated after a model change, and what they do with findings. Then ask the question that actually distinguishes vendors: what would they tell you if they found something serious in a product you were already running, and how quickly.

Can we do this ourselves without security specialists?

The most valuable half, yes. Listing what would be damaging in your specific business, and checking whether it can be made to happen, is domain knowledge rather than technical skill. Bring specialists in for the parts touching authentication, permissions and anything a regulator will examine.

What does a good finding actually look like?

A specific input, what the system did with it, why that matters in your context, and how severe it would be. Vague findings cannot be fixed or retested. The test of whether the exercise was worth commissioning is whether each finding could be turned into a case that runs automatically afterwards.

Glossary

Red-teaming

Red-teaming is deliberately testing an AI system the way somebody hostile would, before its users get the chance, in order to find out how it fails rather than confirming that it works.

In plain terms

Ordinary testing asks whether the thing does its job. This asks what happens when somebody is not trying to help it. The exercise borrows its name and its logic from security work, where the assumption has long been that finding your own weaknesses is considerably cheaper than having them found for you.

Why it matters

The failures that reach a board meeting are almost never the ones ordinary testing was looking for. They are an assistant disclosing something to the wrong person, an answer nobody would have approved appearing under your name, or a system doing something on the basis of text that arrived in a document from outside. None of those show up in testing that checks whether the tool works, because in all three cases it worked exactly as built.

How it works

The exercise is framed around consequences rather than techniques. What would embarrass us, what would breach an agreement, what would cost money, what would be difficult to explain: those questions produce the list of things worth attempting, and the list is specific to your deployment rather than borrowed from anybody else's.

The categories that recur are worth knowing even without the methods. A system can be made to disclose material the person asking should not see, or to produce content the organisation would not stand behind. It can act on instructions that arrived inside material it was asked to process. And it can be steered gradually across a long conversation rather than in a single request.

It is done by people, increasingly with automated help, and the two find different things. Automated probing covers volume and finds the known shapes quickly. People find the ones specific to your business, because knowing what would actually be damaging here is domain knowledge rather than security knowledge.

The output is a list of findings with severity attached, and its value depends entirely on what happens next. Findings that become fixed checks, added to a set that runs on every change, protect you. Findings that become a report protect nobody, and the difference is visible a year later when somebody asks whether the same thing could still happen.

Two kinds of testing, and which one an incident comes from

Both columns are testing and they answer different questions. The left column establishes that the thing does its job, which is essential and is where nearly all effort goes. The right column establishes what happens at the edges: a question outside the intended scope, a document that came from somebody else, a person who is not trying to be helped. The reason the second matters disproportionately is that the failures organisations end up explaining are almost never malfunctions. The system did what it was built to do, and what it was built to do included something nobody had thought about. That is a design gap rather than a bug, and design gaps are only found by somebody looking for them deliberately.

Seen in the wild

Run your own list of difficult and out-of-scope requests through a deployment before it goes live, and record what came back rather than trusting an impression.
Claude
Build a test flow that pushes a fixed list of awkward inputs through your system on every change, so a finding becomes a permanent check.
n8n
Give a document tool material that genuinely does not answer a question and confirm it says so, which is a property you can test rather than an assurance you accept.
NotebookLM

Common misconceptions

People assume

The provider has done this, so we do not need to.

In fact

Theirs covers the model and stops at their boundary. Yours has to cover what your system can reach, what it may do without approval, which documents it will read, and what your particular material would be damaging to disclose. The provider cannot test any of that because they do not know what you connected.

People assume

It is a one-off exercise before launch.

In fact

The system changes, the model underneath changes, new connections get added, and each of those can reopen something previously closed. The version that works is a set of cases that runs on every change, seeded by findings, rather than an exercise somebody remembers doing.

Telling them apart

Red-teaming vs Jailbreak

Red-teaming

The practice of looking for weaknesses deliberately and in advance, on your own system, in order to close them.

Jailbreak

One category of what such an exercise looks for, in which a request is crafted to make a model set aside its own rules.

One is an activity you commission and the other is a phenomenon it goes hunting for. A red-team exercise covers far more than that single category, including permissions, disclosure and what a system will act on.

Questions

What should we ask a vendor about theirs?: What categories they test, how often, whether testing is repeated after a model change, and what they do with findings. Then ask the question that actually distinguishes vendors: what would they tell you if they found something serious in a product you were already running, and how quickly.
Can we do this ourselves without security specialists?: The most valuable half, yes. Listing what would be damaging in your specific business, and checking whether it can be made to happen, is domain knowledge rather than technical skill. Bring specialists in for the parts touching authentication, permissions and anything a regulator will examine.
What does a good finding actually look like?: A specific input, what the system did with it, why that matters in your context, and how severe it would be. Vague findings cannot be fixed or retested. The test of whether the exercise was worth commissioning is whether each finding could be turned into a case that runs automatically afterwards.

Key takeaways

It tests how a system fails rather than confirming that it works.
The list of things worth attempting comes from what would be damaging to you, not from a generic catalogue.
People and automated probing find different things, and the domain-specific ones need people.
A provider's testing stops at their boundary and cannot cover what you connected.
Findings are worth what they become: permanent checks protect you, a report does not.

Tools that use this

Claude
A place to run your own awkward cases before anything goes live.
n8n
Turning a finding into a check that runs on every change.
NotebookLM
Scope adherence as something testable rather than promised.

Last checked July 2026

All glossary terms