Glossary
RAG (retrieval-augmented generation)
RAG, or retrieval-augmented generation, is a method where software searches a chosen set of documents for passages relevant to your question, then has an AI model write its answer from those passages rather than from general training.
In plain terms
Think of the difference between asking a well-read colleague what they can remember and asking the same colleague after handing them the file. RAG is the second one. The system finds the few paragraphs that bear on your question, puts them in front of the model, and asks it to answer from those. What comes back can point at where each claim came from, and it can be wrong in a way you can check.
Why it matters
Most of what an organisation needs answered is not on the open web. It is in policies, contracts, product manuals, past tickets and meeting notes, and it changes. A model trained months ago cannot know any of it. RAG is the ordinary way to close that gap without retraining anything: point the system at the documents that count, and the answers start coming from your material instead of from a general impression of the world. It is also the main practical brake on confident invention, because an answer tied to a passage can be opened and checked.
How it works
The documents are prepared once, before anyone asks anything. Each file is split into passages of a few hundred words, and every passage is turned into an embedding, a list of numbers standing for its meaning. Those are stored, usually in a vector database, so that passages about similar subjects sit close together and can be found quickly.
When a question arrives, it is embedded the same way and used to search that store. The system pulls back the handful of passages that sit nearest to it. This is matching on meaning rather than on the exact words typed, so a question about annual leave can find a passage that only ever says holiday entitlement. Many systems combine this with ordinary keyword search, because exact terms such as a product code still matter.
How the documents are split matters more than it sounds. Passages cut too small lose the context that made them meaningful; passages left too large dilute the match and crowd other results out of the space available. Splitting on the document's own structure, such as headings or clauses, generally beats splitting at a fixed word count, and a table or a form sliced across the middle is one of the commonest reasons a system cannot answer something that is plainly sitting in the file.
The retrieved passages are then placed into the request alongside your question, with an instruction to answer from them. The model does the writing, but the facts it is working from are on the desk in front of it. Because the passages are identified, the answer can carry a citation back to each one.
Nothing in the retrieval itself knows who is asking. A store built from everything a company holds will find the passage that answers the question regardless of whether the person asking was ever entitled to read the document it came from, so permission has to be applied when passages are retrieved rather than trusted to the wording of the request. It is a failure in this arrangement that becomes an incident more often than teams expect.
Two limits follow directly from this shape. The system can only find what is in the documents you loaded and can only fit so many passages into the context window, so retrieval quality decides answer quality. And nothing here forces the model to stay strictly within the passages: grounding makes invention much less likely, not impossible, which is why the citations are the part worth opening.
The preparation flow, which runs once
The question flow, which runs every time, and where the two meet
Seen in the wild
Upload a set of source documents to a notebook tool and ask questions the material can answer. Replies quote the passage they came from and jump you to it, and the tool stays inside what you loaded.
NotebookLMAsk a work search tool a question and get an answer assembled from the organisation's own drives, wikis and tickets, showing each person only the material their permissions already allow.
GleanRun the same pattern on your own hardware: drop files into a workspace and question them with citations, with the splitting and the vector store handled for you and nothing leaving the machine.
AnythingLLMAsk an answer engine a current question and read the numbered citations rather than the summary. The web is the document set, and the citations are what make the answer checkable.
Perplexity
Common misconceptions
People assume
RAG teaches the model your documents.
In fact
It does not change the model at all. The documents are searched at the moment you ask, and the relevant passages are handed over with your question. Take the documents away and the model reverts to what it always knew. Fine-tuning is the method that changes the model itself, and it answers a different problem.
People assume
RAG stops the model making things up.
In fact
It reduces invention substantially and does not remove it. The model can still misread a passage, blend two of them, or fill a gap when retrieval returns nothing useful. The safeguard that actually works is the one RAG makes possible: open the citation and check that the passage says what the answer claims.
People assume
Retrieval respects the permissions our documents already have.
In fact
Not by itself. The search runs over whatever was indexed and knows nothing about who is asking, so unless permissions were carried into the index and applied at the moment of the search, a question can return a passage from a document its asker could never have opened. This is a failure that becomes an incident more often than teams expect, and it is worth asking a vendor about in those words.
People assume
More documents means better answers.
In fact
Only up to a point. Retrieval has to find the right few passages among everything loaded, and a store full of drafts, duplicates and superseded versions makes that harder rather than easier. A small, current, well-curated set usually outperforms a large stale one.
Telling them apart
RAG vs Fine-tuning
RAG
Adds knowledge at the moment of asking, by finding and supplying relevant passages. Update it by changing the documents. Answers can cite. Suits facts that change and material that must be current.
Changes the model's own behaviour by training it further on examples. Update it by training again. Answers cannot cite what they absorbed. Suits style, format and consistent handling of a domain.
If the requirement is that the answer be current and checkable, that is retrieval. If the requirement is that the output sound or behave a particular way every time, that is training.
RAG vs Semantic search
RAG
Retrieves the passages and then has a model write a single answer from them, with citations back to the sources.
Retrieves and stops there. You get the matching passages, ranked by meaning, and read them yourself.
Semantic search is the retrieval half on its own. RAG is that half plus a model writing the answer.
Questions
- What does RAG stand for?
- Retrieval-augmented generation. Retrieval is the search step that finds relevant passages in a chosen set of documents, augmented describes the request being topped up with those passages, and generation is the model writing the answer. The name is read most easily backwards: generation, augmented by retrieval.
- Is RAG the same as uploading a file to a chat assistant?
- For a single short file, the effect is similar and the mechanism is often simpler: the whole document is placed in the context window with no searching required. RAG is what becomes necessary once the material is too large to fit, because then something has to choose which passages go in.
- Does RAG send my documents to the model provider?
- It depends where the parts run. In a hosted setup, the retrieved passages travel to the provider with each question, so the retrieval step does not by itself keep material private. A fully self-hosted arrangement, with a local model and a local store, keeps everything on hardware you control.
- Why does a RAG system sometimes say it cannot find something that is in the documents?
- Because retrieval failed rather than the model. The passage may sit in a file type that was never indexed, be phrased so differently that the meaning match missed it, or be split across a boundary so no single passage carries the whole point. Rephrasing helps sometimes; fixing how documents are split and indexed helps more.
- Do I need a vector database to use RAG?
- Not always. Small collections can be searched adequately without one, and many tools include their own store so you never see it. A dedicated vector database earns its place at scale, where millions of passages have to be searched fast enough for someone to sit and wait for the answer.
- How do I tell whether a RAG system is working well?
- Test it with questions whose answers you already know, drawn from different corners of the collection, and read the citations rather than the prose. Two failures look alike and need different fixes: an answer citing the wrong passage is a retrieval problem, while one citing the right passage but misstating it is a generation problem.
Key takeaways
- RAG searches a chosen set of documents first, then has a model answer from what it found.
- It adds knowledge without changing the model, so updating the documents updates the answers.
- Answers can cite their sources, which is what makes them checkable.
- Retrieval quality sets the ceiling: the system cannot use what it fails to find.
- It reduces invention rather than removing it, so the citations still need opening.
Tools that use this
- NotebookLM
Answers strictly from documents you upload, with citations back into the passage.
- Glean
The same pattern across an organisation's own applications, respecting each person's permissions.
- AnythingLLM
Runs the whole retrieval loop on hardware you control, with a local store.
- Perplexity
Retrieval over the live web rather than a private collection, with numbered sources.
Last checked July 2026