What should we measure before rolling it out?

Two things: the share of documents that need no person, and the share of those that are wrong. Together they decide the business case, and either alone misleads. A high pass rate with a poor error rate is worse than a lower pass rate that sends the doubtful cases to somebody, because the errors travel silently.

Why does it struggle with one particular supplier?

Because layout is inferred rather than declared, and that supplier's arrangement is unlike the ones the system reads easily. It is not a configuration you have got wrong. The practical answers are a rule for that sender, an example or two of their format, or routing them to a person and accepting that as the cost.

Is handwriting realistic?

Sometimes, and it belongs firmly in the doubtful pile. Clear printing on a good scan is often handled; anything cursive, faint or written across a field boundary is where confidence should fall and a person should see it. If a system reports high confidence on handwriting, that is itself worth investigating.

Glossary

Document understanding

Document understanding pulls the structure and meaning out of a file rather than only its words, so an invoice becomes fields and a contract becomes clauses instead of a wall of text.

In plain terms

Getting the meaning out of a document, not just the words. An invoice becomes a supplier, a date and a total; a contract becomes clauses you can ask about. It works beautifully on the examples you try first. The difficulty is that real documents arrive in every shape their senders felt like using.

Why it matters

Because the pilot and the rollout test different things and everybody runs the pilot. A handful of clean, familiar documents will be handled almost perfectly, and the production population contains scans at an angle, a supplier who puts the total in a different place and a form somebody filled in by hand. Variety is the workload, and a sample does not contain it.

How it works

Structure is inferred rather than declared, which is what makes it flexible and unreliable in the same breath. Nothing tells the system that this box is the total; it works that out from position, wording and what the surrounding text implies, so a supplier who lays things out unusually is a genuinely harder case rather than a configuration problem.

Confidence is the number that matters more than accuracy. A system that is right most of the time and knows which cases it is unsure about is far more useful than a slightly more accurate one that is uniformly certain, because the first can route the doubtful ones to a person and the second cannot. Ask what happens to the cases it is unsure about.

The failure mode is a plausible wrong field, not a blank one. A date picked from the wrong line, a total taken from a subtotal, a party name read from the letterhead rather than the signature block: each arrives looking exactly like a correct extraction and flows onward. Nothing downstream has any reason to question it.

Two numbers decide whether it works in production, and neither is in a demonstration. What share of documents goes through without a person, and what share of those is wrong, are the whole of the business case, because a high pass rate with a poor error rate is worse than a lower pass rate that routes doubt to somebody. Measure both on real post.

The documents you tested against, and the ones that arrive

What makes this trap so hard to avoid is that assembling the pilot set is a reasonable act performed by a careful person. Somebody gathers a folder of examples, and the examples they can lay hands on quickly are the ones that were already filed properly, which means they are already the tractable cases. Nothing about that process is careless, and it systematically excludes exactly the population that decides whether the project works: the ones that came in badly, from the sender nobody has standardised, photographed at a desk rather than scanned. The remedy is cheap and slightly tedious, which is to take a month of genuine incoming documents without filtering them at all, run the system against the lot, and count two things. Everybody who does that gets a lower number than the pilot promised and a plan that survives contact with the post room.

Seen in the wild

Document intelligence placed inside the reader everybody already has, answering questions grounded in the pages with references pointing back into them.
Adobe Acrobat AI Assistant
A research tool answering strictly from uploaded material, with references that jump to the passage rather than to the file.
NotebookLM
A general assistant that takes documents and spreadsheets directly into a conversation for comparison, summary and transformation.
Claude

Common misconceptions

People assume

The pilot proves it works.

In fact

The pilot proves it handles documents like the ones in the pilot. Production populations contain angled scans, unfamiliar layouts, handwriting and the supplier who does everything differently, and none of that is in a curated sample. Run it against a month of real post before believing a number.

People assume

Errors will look like errors.

In fact

They look like extractions. A date read from the wrong line and a total taken from a subtotal both arrive correctly formatted in the right field, so nothing downstream has cause to question them. That is why the confidence signal matters more than the headline accuracy.

Telling them apart

Document understanding vs OCR

Document understanding

What the document means and where things sit.

OCR

What characters are on the page.

The second is usually a step inside the first, which is why a system can transcribe a page perfectly and still put the wrong number in the total field.

Questions

What should we measure before rolling it out?: Two things: the share of documents that need no person, and the share of those that are wrong. Together they decide the business case, and either alone misleads. A high pass rate with a poor error rate is worse than a lower pass rate that sends the doubtful cases to somebody, because the errors travel silently.
Why does it struggle with one particular supplier?: Because layout is inferred rather than declared, and that supplier's arrangement is unlike the ones the system reads easily. It is not a configuration you have got wrong. The practical answers are a rule for that sender, an example or two of their format, or routing them to a person and accepting that as the cost.
Is handwriting realistic?: Sometimes, and it belongs firmly in the doubtful pile. Clear printing on a good scan is often handled; anything cursive, faint or written across a field boundary is where confidence should fall and a person should see it. If a system reports high confidence on handwriting, that is itself worth investigating.

Key takeaways

Structure is inferred, so an unusual layout is a hard case rather than a setting.
Confidence matters more than accuracy, because it routes doubt to a person.
Errors arrive as well-formed fields and travel silently.
Pass rate and error rate together are the business case; a pilot tests neither.

Tools that use this

Adobe Acrobat AI Assistant
Answers grounded in the pages, inside the reader itself.
NotebookLM
Answers strictly from uploads, references jumping to the passage.
Claude
Documents and spreadsheets taken directly into the conversation.