Semantic search

It is the machinery underneath every product that offers to answer questions about your own documents, so understanding it explains both why those products feel impressive and how they go wrong. It also has a failure mode worth naming before you buy: it returns things related to what you asked rather than things that answer it, and related is a wide category that includes the passage saying the opposite. Knowing that changes what you check.

Everything searchable is converted in advance into a position that encodes its meaning, and those positions are stored. When you search, your question is converted the same way and the system returns whatever sits nearest. All the apparent understanding is that comparison, which is arithmetic and very fast even across millions of items.

Longer documents are cut into passages first, and the size of those passages decides what can be found at all. Cut too large and results arrive padded with irrelevance around the useful sentence. Cut too small and passages lose the context that made them meaningful, so the right one no longer looks like an answer to anything.

Ranking is by proximity, and proximity means relatedness rather than agreement. Two passages sit close together when they concern the same subject, whether or not they say the same thing, which is why a question about what is permitted can return the paragraph explaining what is forbidden. Nothing in the arithmetic has a notion of negation.

It and ordinary word matching fail in opposite directions, which is why capable systems run both. Meaning-based matching finds the passage that never uses your words and loses the exact reference number you typed. Word matching does precisely the reverse. Combining the two and then reordering what both returned covers considerably more than either alone, and has become close to standard practice for that reason.

Permissions are a separate problem and the one most likely to cause an incident. A search that indexes everything and ranks by meaning will happily surface a document somebody is not supposed to see, because nothing in the proximity calculation knows about access. Any system searching company material needs permissions applied at the point of retrieval, and that is a question worth asking a vendor early and specifically.

• Search a company workspace for something every team words differently and watch results from all of them come back together, with what you are not cleared to see left out.

  Glean

• Ask a notebook tool a question using none of the words in the document you uploaded, and see it surface the right passage anyway.

  NotebookLM

• Run the same search on a self-hosted assistant where the retrieved passages are shown to you, so you can see what the answer was built from before you trust it.

  AnythingLLM

Why does it return something related but wrong?

Because closeness in meaning is not agreement. A passage about the same subject sits near your question whether it confirms, contradicts or supersedes what you asked about. That is the arithmetic working as designed, which is why any serious use of it has a step that reads the passage rather than trusting its position in the list.

Should we replace our keyword search with it?

Run both. The two miss different things, and swapping one for the other usually produces a system that finds the policy nobody could find before and can no longer find an invoice by its number. Combining them and reordering the merged results is more work and materially better than either alone.

Can it find an exact reference number?

Poorly, and this surprises people. A code carries almost no meaning, so it has nothing to sit near, and a meaning-based search will return things about the general subject instead. This is the clearest case for keeping ordinary word matching in the system alongside it.

Does it respect who is allowed to see what?

Only if it was built to. Nothing in the matching itself knows about permissions, so an index built without them will rank a confidential document exactly as readily as a public one. Ask a vendor whether access is checked at retrieval time and how it stays current when somebody changes team.

• It compares the meaning of your question with stored material, and returns whatever sits nearest.
• Passage size decides what can be found: too large adds noise, too small removes context.
• Proximity is relatedness, not agreement, so the passage saying the opposite ranks well.
• It and word matching miss opposite things, which is why capable systems run both and reorder.
• Permissions have to be applied at retrieval, because nothing in the matching knows who you are.

• Glean

  Workspace search where access is applied to what comes back.

• NotebookLM

  The clearest small demonstration: right passage, none of your words.

• AnythingLLM

  Self-hosted, with the retrieved passages visible before you trust the answer.