Which version should we ask a vendor for?

Name it rather than using the word. Ask whether an answer shows its sources, and separately whether the system records what it retrieved and which steps ran. Those are two different features with different availability, and asking for explainability in the abstract reliably produces a demonstration of whichever one the vendor has.

Is this required of us?

It depends on the decision and the jurisdiction, and that is a question for someone qualified rather than a glossary. What is worth knowing before the conversation is that an account of a decision is a much harder thing to supply than a list of sources, so establishing which is being asked for changes the size of the problem considerably.

What should be recorded to make an account possible later?

What went in, what was retrieved, which version answered and what came out. That set is ordinary logging rather than anything specific to AI, and it is almost always cheaper to record from the start than to reconstruct once somebody is asking. The version matters more than people expect, because the same question can be answered differently after a model changes.

Do simpler models help?

For a genuine account of a decision, yes, and that trade is real. Systems built from rules and simpler statistical models can be inspected in a way a large language model cannot. The cost is capability, and the choice is usually settled by whether anybody has to defend an individual outcome to the person it affected.

Glossary

Explainability

Explainability is being able to say why a system produced a given output, and in practice it splits into showing the sources an answer rests on and accounting for how a decision was reached.

In plain terms

Being able to answer why did it say that. Two quite different answers hide behind one word. The first is where this came from, which good products now show as sources you can open. The second is how the model arrived at it, which is a much harder question and is usually not answerable at all in the way people expect.

Why it matters

Because the version you are promised is often not the version you were asked for. Someone affected by an automated decision, or a regulator, or an internal reviewer wants an account of the reasoning. A product demonstration shows citations. Both are called explainability, the first is frequently unavailable, and the gap between them is discovered at the moment it is most awkward to discover.

How it works

The available form is provenance: this answer rests on these documents, and here they are. Retrieval-based products do this natively because they genuinely have the sources to hand, and it is a real improvement over an assistant that simply asserts things. Enterprise search in this guide answers with citations and shows each person only what their permissions allow, which is provenance and access control arriving together.

The harder form is an account of the decision, and general-purpose models do not offer it in the way people assume. Asking a model why it answered as it did produces a fluent explanation, but that text is generated the same way the answer was rather than read off the process that produced it. It is a plausible account rather than an audit of the reasoning, and treating the two as equivalent is the most common error here.

Where a real decision account exists, it usually comes from the surrounding system rather than the model. Which records were retrieved, which rules applied, which steps ran, what the thresholds were: that is a trail somebody built, and it is inspectable because it is ordinary software. Frameworks that express agents as graphs with tracing exist substantially for this reason, and the observability is a deliberate feature rather than a by-product.

The standard demanded is often higher than the one the previous process ever met, and noticing that changes the conversation. The human decision this replaces was frequently unexplainable too: a judgement formed from experience, recorded as an outcome and not as reasoning. That is not an argument for demanding less, since a system applies its reasoning at a scale a person never could. It is an argument for asking what standard is actually required rather than assuming the answer is total.

A citation is evidence about sourcing and not about correctness, which is the caveat most easily lost. A tool in this guide states plainly that citations lower the risk of invention without removing it, because sources can be misread or thin. An answer can carry impeccable references and still misrepresent them, and the reader who stops at the presence of a link has checked the wrong thing.

Two answers to why did it say that

The distinction matters most in the meeting where somebody asks for an explanation and is shown a citation, and everyone present believes the request has been met. It has not, and the mismatch surfaces later when the explanation has to satisfy someone with a stake in the outcome. The left column tells you what the answer rests on, which is genuinely valuable and makes checking possible for the first time. It does not tell you why this answer rather than another, because the sources were selected by a process that left no readable account of itself. The right column can do that, and everything in it had to be built: the retrieval logged, the steps recorded, the thresholds written down. That is why the practical question is never whether a system is explainable but which of these two things exists, and the answer is usually available in a single question to whoever built it.

Seen in the wild

Enterprise search answering with citations while showing each person only what their permissions allow, so the sources are visible and correctly scoped.
Glean
A research tool whose own documentation notes that citations lower the risk of invention but do not remove it, and that load-bearing claims deserve a click through.
Perplexity
A framework expressing agents as graphs with tracing and evaluation tooling, where the inspectable trail comes from the surrounding system rather than the model.
LangChain / LangGraph

Common misconceptions

People assume

Asking the model to explain itself produces an explanation.

In fact

It produces text about the answer, generated the same way the answer was. It is often useful and it is not a record of what happened inside the model. Where an account has to hold up to a challenge, it needs to come from a trail the surrounding system recorded, not from the model's own description of itself.

People assume

Citations mean the answer is right.

In fact

They mean the answer has a stated basis, which is a different and lesser claim. The source can be misread, outdated or thin, and the summary can drift from what it actually says. Citations move the checking work from impossible to feasible; they do not do the checking.

Telling them apart

Explainability vs Audit trail

Explainability

Why this output, in terms a person can evaluate.

Audit trail

What happened and when, recorded as it occurred.

The trail proves the sequence; the explanation justifies the outcome. Buyers ask for the second and are frequently sold the first.

Questions

Which version should we ask a vendor for?: Name it rather than using the word. Ask whether an answer shows its sources, and separately whether the system records what it retrieved and which steps ran. Those are two different features with different availability, and asking for explainability in the abstract reliably produces a demonstration of whichever one the vendor has.
Is this required of us?: It depends on the decision and the jurisdiction, and that is a question for someone qualified rather than a glossary. What is worth knowing before the conversation is that an account of a decision is a much harder thing to supply than a list of sources, so establishing which is being asked for changes the size of the problem considerably.
What should be recorded to make an account possible later?: What went in, what was retrieved, which version answered and what came out. That set is ordinary logging rather than anything specific to AI, and it is almost always cheaper to record from the start than to reconstruct once somebody is asking. The version matters more than people expect, because the same question can be answered differently after a model changes.
Do simpler models help?: For a genuine account of a decision, yes, and that trade is real. Systems built from rules and simpler statistical models can be inspected in a way a large language model cannot. The cost is capability, and the choice is usually settled by whether anybody has to defend an individual outcome to the person it affected.

Key takeaways

One word, two demands: showing the sources, and accounting for the decision.
Provenance is widely available; a real account of the reasoning usually is not.
A model's explanation of itself is generated text, not a record of what happened.
Where a decision account exists it comes from the surrounding system, which is inspectable.
Citations are evidence about sourcing, never about correctness.

Tools that use this

Glean
Citations plus permission-aware scoping, so sources are visible and correct.
Perplexity
States that citations lower the risk of invention without removing it.
LangChain / LangGraph
Tracing and evaluation, where the inspectable trail actually lives.