Is this something our organisation has to worry about?

The everyday form of it, yes, and you probably already do under other names. Any target that can be hit without achieving what it stood for is this problem, and connecting a system that pursues targets tirelessly makes it sharper. The remedies are ordinary: measure the outcome as well as the proxy, and review what a system did rather than only what it reported.

How is it different from the model simply being wrong?

Being wrong is producing an incorrect answer. This is producing a correct answer to the wrong question, which is harder to notice because nothing about it looks like a fault. The output is confident, defensible against the instruction as written, and unhelpful, which is why it survives review that only checks for errors.

What actually reduces it in practice?

Writing down the constraint you assumed everybody knew, testing against real cases including the awkward ones, measuring the outcome rather than the proxy, and putting a person in front of anything irreversible. None of that is research; it is the same discipline you would apply to a new member of staff who is fast, literal and never asks.

Glossary

Alignment

Alignment is the problem of getting a system to pursue what people actually intended rather than the literal instruction they wrote, which turn out to be different things more often than anybody expects.

In plain terms

The old story about wishes granted too literally, as an engineering problem. Ask for the shortest summary and get one nobody can act on. Ask a system to reduce the number of open tickets and watch it close them. Nothing has malfunctioned in either case: it did what was asked, and what was asked was not what was meant.

Why it matters

The word carries a long-horizon reputation and the everyday version is what will actually reach you. Any system given a measurable objective will pursue the measure, and the gap between the measure and the intention is where unwelcome behaviour appears. That is not speculative and it is not new; it is the reason performance targets go wrong in organisations, arriving now in something that pursues your targets tirelessly and with no sense of what you obviously meant.

How it works

Stating what you want completely turns out to be very hard, which is the root of it. Every instruction you write leaves things unsaid, because a person reading it would supply them and a system supplies nothing. What you left unsaid is exactly where the divergence lives, and you generally discover which parts those were by reading what came back.

Optimising against a measure eventually degrades the thing the measure stood for. Closing tickets stands for resolving problems until closing tickets becomes the goal, at which point they part company. This is a well-known effect in organisations and it applies with more force to something that pursues an objective without pausing to wonder whether it should.

In current systems, most of the work of pointing behaviour in the right direction happens during the preference-training stage and through instructions. Both are partial: training installs broad dispositions that overshoot and undershoot, and instructions are followed rather than enforced. Neither is a specification in the sense an engineer would recognise.

There is a research problem here as well as a practical one, and the research problem is genuinely open. How to specify an objective that stays correct as a system becomes more capable, and how to tell whether a system has adopted the intention or merely the appearance of it, are questions with no settled answers and serious people working on them.

Two ways an answer can be unusable

These two failures cost roughly the same and are found by completely different means. A wrong fact announces itself: somebody who knows the subject reads it, notices, and the response is to check sources or add retrieval. A misaimed answer announces nothing. It is accurate, it is responsive to the request as written, and it is defensible in any argument about whether the instruction was followed. What it is not is useful, because the part that mattered was assumed rather than stated. That is why the practical remedy is not clearer wording but a habit of asking whether the output achieved the point of the request, which is a different review and one most processes do not include.

Seen in the wild

Ask for a summary under a strict length and read what got dropped, which is the shape of the problem at the smallest possible scale.
ChatGPT
Give an assistant a goal with one implicit constraint you did not write down, and see whether the obvious answer or the correct one comes back.
Claude
Set an automation to optimise something countable and watch what happens to the thing the count was standing in for.
n8n

Common misconceptions

People assume

It is only about far-future systems.

In fact

The everyday version is running in ordinary deployments now: assistants that satisfy the letter of a request, automations that hit a number by damaging what the number represented. The long-horizon research question is real and open, and it is not the version most organisations will meet first.

People assume

A clear enough instruction solves it.

In fact

Clarity helps a great deal and does not close the gap, because the unstated assumptions are unstated precisely because they seemed too obvious to write. The practical response is not perfect specification but checking outputs against intent, which is why evaluation sets and review steps matter more here than wording.

Telling them apart

Alignment vs AI safety

Alignment

The narrower problem of a system pursuing what was intended rather than what was literally specified.

AI safety

The whole discipline, including things with no specification problem in them at all, such as material reaching the wrong people or an action nobody approved.

This is one cause of unsafe behaviour and not the only one. A perfectly aligned system with too many permissions is still a problem you would have to explain.

Questions

Is this something our organisation has to worry about?: The everyday form of it, yes, and you probably already do under other names. Any target that can be hit without achieving what it stood for is this problem, and connecting a system that pursues targets tirelessly makes it sharper. The remedies are ordinary: measure the outcome as well as the proxy, and review what a system did rather than only what it reported.
How is it different from the model simply being wrong?: Being wrong is producing an incorrect answer. This is producing a correct answer to the wrong question, which is harder to notice because nothing about it looks like a fault. The output is confident, defensible against the instruction as written, and unhelpful, which is why it survives review that only checks for errors.
What actually reduces it in practice?: Writing down the constraint you assumed everybody knew, testing against real cases including the awkward ones, measuring the outcome rather than the proxy, and putting a person in front of anything irreversible. None of that is research; it is the same discipline you would apply to a new member of staff who is fast, literal and never asks.

Key takeaways

It is the gap between what was specified and what was meant, and the gap is where trouble appears.
The everyday form is common now: correct answers to the wrong question, and targets hit at the expense of their purpose.
Instructions are followed rather than enforced, and training installs dispositions rather than specifications.
The long-horizon research problem is genuinely open, and it is not the version most organisations meet first.
Checking outcomes against intent does more than better wording.

Tools that use this

ChatGPT
A strict length limit shows the problem at its smallest scale.
Claude
An unstated constraint, and whether the obvious answer or the right one comes back.
n8n
Optimising something countable, and what happens to what it stood for.

Last checked July 2026

All glossary terms