Vector database

You will be told you need one, often by somebody selling one, and for a great many organisations it is not true. Below a certain size ordinary tools handle this perfectly well, and several general-purpose databases have added the capability, which means no new system to run, secure, back up and keep in step. Where it does become necessary the reason is scale and speed rather than capability, and knowing that lets you postpone a decision that adds a permanent operational job to your week.

It stores lists of numbers alongside an identifier and whatever labels you attach, and answers one question well: which stored items sit nearest to this one. Everything else it offers, the filtering, the batching, the housekeeping, exists to make that single question usable on real collections rather than on examples.

Comparing a query against millions of stored items exactly would be slow, so these systems use approximate methods that accept a small chance of missing the true nearest match in exchange for an enormous gain in speed. That trade is usually a setting you can move, and it is why two systems holding identical data can return slightly different results.

Filtering matters as much as proximity and is where products genuinely differ. Restricting a search to one department, one date range or one permission group while still ranking by meaning is harder than it sounds, and doing it badly means either slow searches or results that quietly ignore the filter. It is a good thing to test rather than to read about.

It is a derived store rather than a source of truth. Your documents live wherever they lived; this holds positions calculated from them at a particular moment. Keeping it in step with the originals is an ongoing job, and stale entries are a frequent reason a search that worked last quarter has started returning things nobody can find in the actual files.

Changing the model that produces the positions invalidates everything already stored, because positions from two different models are not comparable. That makes a model upgrade a rebuild rather than a setting, and it is the cost most easily overlooked in a plan to move to something newer later.

• Run a self-hosted assistant with its own built-in store and watch the collection grow as you add documents, which is the whole arrangement at a size you can inspect.

  AnythingLLM

• Build one flow that writes documents into a store and another that queries it, which separates the loading job from the searching job in a way finished products usually hide.

  n8n

• Use a workspace search where the store, the syncing and the permission checks are somebody else's operational problem, which is what most organisations should want.

  Glean

Do we actually need one?

Probably not yet. If your material runs to a few thousand passages, ordinary tools handle it and several general databases now include the capability. The threshold is about scale, update rate and how much filtering you need alongside the search, rather than about whether you want meaning-based results at all.

Why do two systems give different results on the same data?

Because both are finding approximately the nearest matches rather than exactly, and they make that approximation differently. Most also cut documents into passages differently, which changes what there is to find. Neither is wrong, and it is a reason to compare on your own material rather than on a specification.

Is our data safe in there?

It needs the same protection as the documents it came from. The stored positions cannot be read directly, which is not the same as being anonymous, and research has recovered meaningful information from stores of this kind. Treat it as a second copy of sensitive material rather than as harmless derived data.

What happens when we want a newer model?

Everything stored has to be regenerated, because positions from two models are not comparable. On a large collection that is a project rather than a maintenance task, and it is the cost most easily left out of a plan that assumes you can upgrade later. Ask about it before the collection gets big.

• It answers one question well: which stored items sit nearest to this one.
• The speed comes from approximate matching, so results are nearly the nearest rather than exactly.
• Filtering alongside proximity is where products genuinely differ and is worth testing.
• It is a derived copy, so keeping it in step with the source documents is an ongoing job.
• Below a few thousand passages most organisations do not need one at all.

• AnythingLLM

  A built-in store you can watch fill up as documents go in.

• n8n

  Loading and querying as separate steps, which finished products hide.

• Glean

  The store, the syncing and the permissions handled by somebody else.