Why not just send the data to a server?

Sometimes there is no connection, sometimes the round trip is slower than the situation allows, and sometimes the bandwidth for continuous streaming costs more than the analysis is worth. Where none of those bite, sending it away is simpler and gets you a more capable model, so the honest answer is that most of what you are considering probably should.

How capable can a model on a device be?

Considerably more than a few years ago and still well short of what a data centre runs. For narrow, well-defined jobs such as detecting one class of thing reliably, the small models are entirely adequate. For open-ended language work they are noticeably weaker, which is why mixed designs are common.

What is the hidden cost?

Keeping a fleet current. Every device holds a copy of the model, so improving it means distributing a new one to all of them, confirming they took it, and handling the ones that were offline. That work does not exist in a server deployment, where changing the model is a single act.

Glossary

Edge AI

Edge AI means the processing happens on or beside the device producing the data, such as a phone, a camera or a vehicle, instead of the data being sent somewhere else to be handled.

In plain terms

The thinking moves to where the data is made. A camera that decides for itself whether it is looking at a person, a phone that transcribes what you said without sending the recording anywhere, a machine on a production line that spots a defect between one item and the next. Nothing travels, which is the point in every case and for slightly different reasons.

Why it matters

The reasons are usually practical before they are about privacy. A vehicle cannot wait for a round trip to a data centre before braking. A site with poor connectivity cannot depend on one. A thousand cameras streaming continuously would cost more in bandwidth than the analysis is worth. Where any of those apply, this stops being an option among several and becomes the only arrangement that works, which is why you meet it in industrial and physical settings far more often than in the office ones most people are buying for.

How it works

The model runs on hardware at the device or very near it, which means it has to be small enough to fit and quick enough to be useful there. Purpose-built processors for exactly this job now appear in phones, cameras and industrial equipment, and the models are usually reduced in size and precision to suit them.

The result is what travels, not the raw material. A camera sends the fact that something was detected rather than the video, which is the whole bandwidth argument in one sentence. It is also the strongest privacy property available anywhere in this subject, because the recording genuinely never existed outside the device.

Response time is measured in the time light takes to get nowhere. Nothing leaves, so nothing waits for a network, which is what makes this viable for anything reacting to the physical world. That advantage disappears entirely the moment a design starts calling out for a second opinion.

Updating a fleet is the hard part and the part nobody mentions in the brochure. A thousand devices each carrying a model need a way of receiving a new one, a way of confirming they did, and a plan for the ones that were switched off that week. That is a fleet-management problem wearing an AI hat, and it lands on you rather than on a provider.

How far the data actually travels

This one line organises the whole deployment vocabulary, and every distinction people struggle with is a position on it. At the far left the data never exists anywhere but the sensor that made it, which is the strongest privacy claim available in this subject and is not a policy but a physical fact. A phone doing its own transcription is barely further along. A box on site serving the devices around it is the arrangement most industrial deployments actually use, because one reasonably capable machine can serve many modest ones without anything leaving the premises. The far right is where nearly all office AI sits. Knowing which mark a proposal is describing settles most of the argument, because the privacy, cost and reaction-time properties all follow from the position rather than from the label attached to it.

Seen in the wild

Run a small model on a machine with the network disconnected entirely and notice which capabilities survive the disconnection, which is the constraint every device deployment lives under.
Ollama
Look at a published model family that includes deliberately small members, since those are what make running on modest hardware possible at all.
Qwen
Cap yourself to the smallest model a modest machine will hold and measure the wait, which is the same trade a camera or a vehicle is making permanently.
LM Studio

Common misconceptions

People assume

It is the same thing as running a model on my laptop.

In fact

The mechanism is similar and the situation is not. A laptop is one person's choice, made for one person's reasons, with nothing to administer. This is an architecture decision about a fleet of devices, usually driven by connectivity, bandwidth or response time, and it brings a fleet's worth of update and monitoring problems with it.

People assume

It means nothing ever goes to a server.

In fact

Most real designs are mixed. The device handles the immediate decision and sends summaries, exceptions or difficult cases onward for something larger to look at. Ask a vendor which parts stay on the device and which are forwarded, because their answer is where the privacy claim you are being sold actually lives.

Telling them apart

Edge AI vs Local AI

Edge AI

An architecture decision about devices in the field, driven by connectivity, bandwidth or reaction time, and carrying a fleet's worth of update problems.

Local AI

One person running a model on their own machine, for their own reasons, with nothing to administer.

If somebody has to plan how a thousand of them get the next version, it is the first. If it is a laptop and a downloaded file, it is the second.

Questions

Why not just send the data to a server?: Sometimes there is no connection, sometimes the round trip is slower than the situation allows, and sometimes the bandwidth for continuous streaming costs more than the analysis is worth. Where none of those bite, sending it away is simpler and gets you a more capable model, so the honest answer is that most of what you are considering probably should.
How capable can a model on a device be?: Considerably more than a few years ago and still well short of what a data centre runs. For narrow, well-defined jobs such as detecting one class of thing reliably, the small models are entirely adequate. For open-ended language work they are noticeably weaker, which is why mixed designs are common.
What is the hidden cost?: Keeping a fleet current. Every device holds a copy of the model, so improving it means distributing a new one to all of them, confirming they took it, and handling the ones that were offline. That work does not exist in a server deployment, where changing the model is a single act.

Key takeaways

The processing happens on or beside the device producing the data, so the data itself does not travel.
The motives are usually connectivity, bandwidth and reaction time before privacy.
Only the result is sent, which is both the bandwidth argument and the strongest privacy property available.
Models have to be small enough to fit, so capability is narrower than a data centre offers.
Updating a fleet of devices is the cost that does not exist in a server deployment.

Tools that use this

Ollama
Running disconnected, which is the constraint devices live under.
Qwen
A published family with deliberately small members.
LM Studio
Capping model size and measuring the wait, as a device permanently does.

Last checked July 2026

All glossary terms