How do we find the stage that is failing?

Read the retrieved passages instead of the answers. For a dozen questions that went wrong, look at what came back and ask whether the answer was in it. That single question splits every failure into two piles, and each pile has a completely different remedy, which is why it is worth ten minutes before any larger investigation.

Which stage is most often at fault?

Reading and cutting, in the systems this guide covers, followed by refresh schedules. All three are invisible, all three are set by defaults during setup, and none of them announces a problem. The stages people examine first, ranking and the model, are the ones most likely to be working correctly.

We use a packaged product. Does this apply?

Entirely, with the difference that the stages are configured by somebody else. The failures are the same and the diagnosis is the same; what changes is that the remedy is often a conversation with a vendor rather than a setting you can change. Knowing which stage to name makes that conversation much shorter.

Glossary

Retrieval pipeline

A retrieval pipeline is the whole path from your documents to a grounded answer, and it earns a name of its own because the complaint always arrives about the last stage while the defect is usually much earlier.

In plain terms

Everything between your documents and the answer. Getting the files in, reading their structure, cutting them into pieces, storing them, searching, and choosing what finally goes in front of the model. Six or seven stages, each capable of quietly ruining the result, and only the last one is visible to the person complaining.

Why it matters

Because attention goes to the visible end and the damage is done at the invisible one. Somebody reports a poor answer, the conversation turns to the model, and meanwhile the passage handed over never contained the answer because a table was flattened during reading or a rule was cut away from its exception. Each stage sets a ceiling that no later stage can raise.

How it works

Each stage caps everything after it, which is the property that makes the whole worth naming. Structure discarded while reading cannot be recovered by cutting, a passage cut badly cannot be repaired by searching, and material that never reached the model cannot be reasoned about however capable it is. The ceiling is set by the earliest stage that lost something.

The symptom is identical whatever the cause, and that is why diagnosis is hard. A wrong answer looks the same whether the document was parsed badly, cut in the wrong place, missed by search, filtered out by a stale permission or misread at the end, so the complaint carries no information about where to look and the instinct is always to blame the last thing.

The diagnostic is to read the passages rather than the answers, and it takes about ten minutes. Collect a handful of questions that went wrong, look at what was actually retrieved for each, and ask whether the answer was present in that material. If it was not, everything after retrieval is irrelevant and the work belongs upstream.

Most improvement effort goes to the wrong stage. Better models and better ranking are the accessible levers and the common failures live in reading, cutting and refreshing, so teams frequently spend a quarter on the parts they can change easily while the ceiling stays exactly where it was.

Where the ceiling gets set

The arrows matter more than the boxes, because each one is a place where something can be lost permanently and none of them reports a loss. What makes this shape worth carrying around is the asymmetry at the end: four stages happen in the dark and the fifth happens in front of a person, so every piece of feedback the system ever receives is attached to the last node. A team acting reasonably on that feedback will change models, adjust prompts and tune ranking, and will observe that nothing much improves, because the ceiling was set at the second arrow months earlier by a default nobody chose. The habit that breaks the cycle is small and unnatural: when an answer is wrong, look at what was retrieved before looking at what was written. That one reordering turns an unfalsifiable sense that the tool is unreliable into a specific claim about a specific stage, which is the only kind of claim anybody can act on.

Seen in the wild

A private document tool that handles taking files in, cutting them up and storing them locally, so the whole path runs on hardware you control.
AnythingLLM
Enterprise search that indexes an organisation's applications and answers with references, where every stage runs behind one box.
Glean
A research tool answering strictly from uploaded material, which is the same path over a corpus you assembled yourself.
NotebookLM

Common misconceptions

People assume

A poor answer is a model problem.

In fact

It is usually a material problem. If what reached the model did not contain the answer, no model improves the result, and that case is common enough to check first. The model is simply the only stage anybody can see, which is why it collects blame that belongs several steps earlier.

People assume

Buying a product means we do not have these stages.

In fact

You have all of them, configured by somebody else's defaults. A packaged tool makes the stages invisible rather than absent, so the same failures occur and are harder to diagnose because the knobs are not exposed. Knowing the shape of the path is useful even when you cannot adjust it.

Telling them apart

Retrieval pipeline vs RAG

Retrieval pipeline

The machinery: every stage from document to answer.

RAG

The pattern: answer from retrieved material rather than memory.

One names the approach and the other names what has to be built and maintained for the approach to work, which is where all the difficulty actually is.

Questions

How do we find the stage that is failing?: Read the retrieved passages instead of the answers. For a dozen questions that went wrong, look at what came back and ask whether the answer was in it. That single question splits every failure into two piles, and each pile has a completely different remedy, which is why it is worth ten minutes before any larger investigation.
Which stage is most often at fault?: Reading and cutting, in the systems this guide covers, followed by refresh schedules. All three are invisible, all three are set by defaults during setup, and none of them announces a problem. The stages people examine first, ranking and the model, are the ones most likely to be working correctly.
We use a packaged product. Does this apply?: Entirely, with the difference that the stages are configured by somebody else. The failures are the same and the diagnosis is the same; what changes is that the remedy is often a conversation with a vendor rather than a setting you can change. Knowing which stage to name makes that conversation much shorter.