Does this affect us if we only use hosted tools?

Less directly, because your supplier is making the choices about what runs. It moves the question rather than removing it: what you can reasonably ask is where their models come from and what they check, which is the same question you would answer for yourself.

How would we ever notice?

Usually not by testing, which is the uncomfortable part. Selective misbehaviour survives general evaluation, so the practical defences are about the supply chain rather than about detection: known sources, verified downloads, and a short list of places you take models from.

Is this the same as somebody manipulating a chatbot with a document?

No, and the distinction matters because the second is much easier. Planting material where a system will read it at answer time steers the output without touching training at all, and that is the version a business is far more likely to meet.

Glossary

Model poisoning

Corrupting a model by tampering with what it learned from, so that it behaves normally except in the situations an attacker chose.

In plain terms

Interfering with what a model learns so it behaves badly in a specific situation later. The unnerving part is that it behaves perfectly the rest of the time, so nothing looks wrong until the moment somebody chose.

Why it matters

Because almost nobody buying AI trains anything, and almost everybody ends up running weights that came from somewhere else. That turns an exotic-sounding attack into an ordinary question about provenance, which is the same question sensible organisations already ask about other software.

How it works

The attack targets learning rather than running. Corrupting the material a model learns from leaves the behaviour baked in, so there is nothing to intercept at the moment of use and no request that looks unusual.

The result is selective rather than broadly broken. A poisoned model performs normally on everything except the situations it was shaped to mishandle, which is why general evaluation gives it a clean bill of health.

For most organisations the exposure is the artefact rather than the training. You download weights, you run them, and every assurance you have about what went into them comes from whoever published them, which is a supply-chain relationship whether or not anybody called it one.

Training on material you do not control recreates the problem yourself. A pipeline that learns from scraped pages, user submissions or anything else an outsider can write into is an opportunity somebody else gets to use, and it is usually built for good reasons by people not thinking about this at all.

Retrieval-side manipulation is a different thing that gets called the same. A system reading documents at the moment of answering can be steered by planting a document, which needs no access to training at all, is far easier, and is the version most businesses will actually meet.

Ordinary defences are unglamorous and effective. Take models from sources with a traceable provenance, check that what you downloaded is what was published, keep the set of places you take models from small, and evaluate on your own cases rather than on the ones the publisher chose.

Two attacks people call by one name

Conflating the two costs organisations real effort in the wrong place. The left-hand attack is genuinely difficult and the honest defence against it is procurement rather than engineering: take models from sources you can trace, verify that what arrived is what was published, and keep the list of such sources short enough that somebody knows what is on it. That is a policy decision that can be made once. The right-hand attack requires only that somebody can write into a location your system reads, which in a typical deployment includes shared drives, ticket systems, wikis and anything else connected in the name of usefulness. It is also the one where ordinary controls help immediately: knowing what a system can reach, limiting that to what the job needs, and treating retrieved material as untrusted input rather than as instructions. The uncomfortable summary is that the frightening attack is mostly somebody else's problem, and the boring one is yours.

Seen in the wild

Weights downloaded from a public repository and run without anybody asking who published them.
Hugging Face
A model pulled and run locally, where provenance is whatever the source says it is.
Ollama
A document planted where a search tool will read it, steering the answer without touching a model.
Glean

Common misconceptions

People assume

It only concerns organisations that train models.

In fact

Running weights somebody else produced puts you downstream of their training. You inherit whatever went into it and your assurance is their word, which makes this a provenance question rather than a training question.

People assume

Testing the model would reveal it.

In fact

A poisoned model is meant to behave normally except in chosen situations, so general evaluation reports it as fine. Testing narrows the risk only where your tests happen to cover the situation somebody selected.

Questions

Does this affect us if we only use hosted tools?: Less directly, because your supplier is making the choices about what runs. It moves the question rather than removing it: what you can reasonably ask is where their models come from and what they check, which is the same question you would answer for yourself.
How would we ever notice?: Usually not by testing, which is the uncomfortable part. Selective misbehaviour survives general evaluation, so the practical defences are about the supply chain rather than about detection: known sources, verified downloads, and a short list of places you take models from.
Is this the same as somebody manipulating a chatbot with a document?: No, and the distinction matters because the second is much easier. Planting material where a system will read it at answer time steers the output without touching training at all, and that is the version a business is far more likely to meet.