What happens to something that fits no category?

Whatever you designed, and most designs place it in the nearest one without ever saying so. An explicit none-of-these category that somebody reviews weekly is better, because it turns a confident error into a visible question about whether your categories are right.

When is this better than asking a general model?

When the categories are stable and the volume is high. A purpose-built sorter is cheaper, faster, repeatable and easier to check, and it gives the same answer twice, which a general model asked the same question will not reliably do.

How should accuracy be reported?

Per category rather than overall, always. A system that is right almost always and wrong on the rare expensive category reports an excellent aggregate figure, and that figure is describing the easy cases you were never worried about in the first place.

Glossary

Classification

Sorting items into categories decided in advance, such as routing an incoming message as a complaint, a query or a renewal.

In plain terms

Putting things into boxes you decided on beforehand. The technique is old and well understood. The difficulty is almost never the sorting: it is that somebody had to choose the boxes, and the awkward items are the ones that arrive later.

Why it matters

Because it is the quiet workhorse behind a great deal of what gets called AI at work, and because it is frequently the right answer where something more elaborate is being considered. Sorting into known categories is a solved problem with predictable costs.

How it works

The categories are decided in advance, which is the defining property. That is what makes the output predictable and countable, and it is also the constraint everything else follows from.

Choosing them is the actual work. Categories that overlap produce inconsistent results, categories that are too broad produce useless ones, and both problems are cheaper to find on a hundred examples by hand than after deployment.

The item that fits nothing is the first hard case. Most designs quietly place it in the nearest category, which is confidently wrong, and the better arrangement is an explicit none-of-these that somebody looks at weekly.

The item that fits several is the second. Deciding in advance whether an item can carry more than one category, and what happens downstream if it does, prevents a system that answers arbitrarily on exactly the cases where a person would have hesitated.

It is often the better choice than something generative. Where the categories are stable and the volume is high, this is cheaper, faster, repeatable and far easier to check than asking a general model to decide, and it does not vary between runs.

Confidence and correctness are separate, and most systems report the first. A score attached to a placement says how strongly the model favoured that category over the others, which is not the same as how likely it is to be right, and treating them as one is how thresholds get set badly.

Categories drift while the system stays still. What arrives changes as products, customers and language change, so a set of categories agreed two years ago quietly stops describing the work, and nothing in the accuracy figure reports that.

Its accuracy is not one number. A system can be excellent overall and poor on the category that matters most, and an aggregate figure hides exactly that, so the useful report is per category rather than in total.

The two cases that decide whether it works

The reason both go undecided is that they are invisible during the part of the work that feels like the project. Building a sorter is done against a sample, and a sample assembled by a reasonable person contains items that fit exactly one category, because those are the ones that come to mind as examples. The awkward items are the ones that turn up later, at volume, and by then the behaviour on them is whatever fell out of the implementation rather than anything chosen. Two questions, asked once, prevent it. What should happen to something that belongs in none of these, and can something belong in two. The answers are usually easy and occasionally reveal that the categories themselves are wrong, which is much cheaper to discover in a conversation than in a queue of misrouted work that somebody has to unpick by hand.

Seen in the wild

Incoming messages routed to the right queue before anybody reads them.
Sierra
Records sorted into categories at volume, where consistency matters more than nuance.
Julius AI
A general model asked to sort things that a purpose-built sorter would do more cheaply.
ChatGPT

Common misconceptions

People assume

The difficulty is in the sorting.

In fact

The sorting is the settled part. The difficulty is choosing categories that do not overlap, are not too broad, and have somewhere sensible to put an item that fits none of them.

People assume

A high overall accuracy means it works.

In fact

It can be excellent overall and poor on the category that costs you most, and an aggregate figure conceals precisely that. The number worth reporting is per category, especially for the rare and expensive ones.

Questions

What happens to something that fits no category?: Whatever you designed, and most designs place it in the nearest one without ever saying so. An explicit none-of-these category that somebody reviews weekly is better, because it turns a confident error into a visible question about whether your categories are right.
When is this better than asking a general model?: When the categories are stable and the volume is high. A purpose-built sorter is cheaper, faster, repeatable and easier to check, and it gives the same answer twice, which a general model asked the same question will not reliably do.
How should accuracy be reported?: Per category rather than overall, always. A system that is right almost always and wrong on the rare expensive category reports an excellent aggregate figure, and that figure is describing the easy cases you were never worried about in the first place.