How do we tell whether ours is wrong?

Read the passages rather than the answers. Take a handful of questions the system got wrong, look at what was actually retrieved, and ask whether the answer was even present in what came back. If it was not, the search did its job and the cutting is where the work belongs, which is a very different remedy.

Is there a size we should use?

There is no size that survives contact with a mixed corpus, which is the honest answer. Contracts, transcripts and support articles all have their own natural joints, so a single figure applied across all of them will suit whichever kind dominates. Cutting on the document's own structure beats any number.

Does overlapping the pieces fix it?

It reduces the boundary problem without removing it, and it costs storage and search volume for the privilege. Repeating a little of each passage in the next means a qualifier near a cut travels with its rule more often. It is worth having and it is not a substitute for cutting in sensible places.

Why is changing it such a large job?

Because everything downstream was built from the current pieces. A new scheme means reprocessing the whole corpus and rebuilding what search reads, which is why a choice made quickly during setup tends to outlive several rounds of model upgrades and remains in place long after the documents have changed character.

Glossary

Chunking

Chunking cuts long documents into passages small enough to search and to hand to a model, and because only the passage travels onward, where the cuts fall decides what an answer can possibly know.

In plain terms

Cutting long documents into pieces small enough to work with. Nobody discusses it and it decides a great deal, because only the piece gets handed onward and never the whole document. A cut in the wrong place separates a rule from its exception, and no amount of clever searching afterwards puts them back together.

Why it matters

Because it sets a ceiling that nothing downstream can lift. If the passage handed over does not contain the qualifying sentence, the answer will be confidently incomplete however good the search was at finding that passage, and the resulting complaint always arrives as a search problem. Teams therefore spend months improving the wrong stage.

How it works

Pieces have to be small enough to search usefully and large enough to make sense on their own, and those two pressures pull in opposite directions throughout. A short passage matches a query precisely and frequently arrives stripped of the context that made it mean anything, while a long one carries its context and matches less sharply because it is about several things at once. Every scheme is a position between those, chosen once and rarely revisited.

Where the cut falls matters far more than how big the piece is. Splitting between a rule and the exception that qualifies it, or between a table and the heading that says what its columns are, produces a passage that is locally coherent and globally misleading, which is the worst possible failure because nothing about it looks wrong. Cutting on the document's own structure rather than on a length avoids most of this.

Overlapping the pieces is the standard defence and it is a blunt one. Repeating a little of each passage in the next means a sentence near a boundary appears in both, so the qualifier travels with the rule more often, at the cost of storing and searching more material than you have. It reduces the failure rather than removing it.

The right scheme follows the shape of your documents, not a recommendation. Contracts split cleanly at clauses, technical documentation at headings, transcripts at speaker turns, and support articles at their existing sections, so a single setting applied across a mixed corpus will suit whichever kind happens to dominate it and quietly fail the rest.

It is difficult to change later, which is why the decision deserves more care than it gets. Altering the scheme means reprocessing everything and rebuilding what search reads, so a choice made in an afternoon during setup tends to survive for years, long after the corpus it suited has been joined by three other kinds of document.

Where a complaint about search actually comes from

The diagnostic habit that separates these takes about ten minutes and is almost never performed, because the natural place to look when answers are poor is at the thing labelled search. Collect a dozen questions the system handled badly, and for each one look at the passages that were actually retrieved rather than at the answer that was produced from them. Two patterns emerge immediately. Either the material needed to answer was in front of the model and it did something unhelpful with it, which is a genuine problem in ranking or in the model, or the material was simply not there, in which case no amount of work on either will help. That second case is common enough to be worth checking first, and it points at a decision made months earlier by somebody accepting a default during setup. Nothing about the failure announces its origin, which is precisely why it survives so many rounds of improvement aimed at the wrong stage.

Seen in the wild

A private document tool that handles ingestion, splitting and local storage for you, so the decision is made by a default rather than by anybody in particular.
AnythingLLM
A research tool answering strictly from uploaded material, where a reference jumping to the exact passage is only useful if that passage was cut somewhere sensible.
NotebookLM
Enterprise search across an organisation's applications, where documents of wildly different shapes pass through the same processing.
Glean

Common misconceptions

People assume

It is a technical setting somebody sensible has already tuned.

In fact

It is usually a default that arrived with the product and was never examined against your documents. Defaults are chosen to be inoffensive across every corpus rather than good for yours, and since nothing surfaces the consequences, the setting survives on the strength of nobody having questioned it.

People assume

Poor answers mean the search needs improving.

In fact

Frequently the search worked perfectly and handed over a passage that never contained the answer. The symptom is identical either way, which is why teams reach for better ranking and better models while the actual defect sits one stage earlier, in a decision they do not think of as a decision.

People assume

Bigger pieces are safer.

In fact

They carry more context and match far less precisely, so search starts returning long passages that are broadly about the right subject and specifically about something else. The trade never disappears; it only moves. What changes with size is which failure you get, not whether you get one.

Telling them apart

Chunking vs Document parsing

Chunking

Where the document is cut into pieces.

Document parsing

Whether its structure survived being read at all.

Parsing comes first and decides what the cutting has to work with, so a document flattened into undifferentiated text cannot be cut on its headings however good the scheme is.

Questions

How do we tell whether ours is wrong?: Read the passages rather than the answers. Take a handful of questions the system got wrong, look at what was actually retrieved, and ask whether the answer was even present in what came back. If it was not, the search did its job and the cutting is where the work belongs, which is a very different remedy.
Is there a size we should use?: There is no size that survives contact with a mixed corpus, which is the honest answer. Contracts, transcripts and support articles all have their own natural joints, so a single figure applied across all of them will suit whichever kind dominates. Cutting on the document's own structure beats any number.
Does overlapping the pieces fix it?: It reduces the boundary problem without removing it, and it costs storage and search volume for the privilege. Repeating a little of each passage in the next means a qualifier near a cut travels with its rule more often. It is worth having and it is not a substitute for cutting in sensible places.
Why is changing it such a large job?: Because everything downstream was built from the current pieces. A new scheme means reprocessing the whole corpus and rebuilding what search reads, which is why a choice made quickly during setup tends to outlive several rounds of model upgrades and remains in place long after the documents have changed character.

Key takeaways

Only the passage travels onward, so the cut caps what any answer can know.
Where the cut falls matters more than how large the piece is.
The symptom of a bad cut is indistinguishable from a search problem.
Overlap reduces boundary loss; it does not remove it.
Changing the scheme means reprocessing everything, so defaults survive for years.