Self-hosted

For a number of organisations this is not an optimisation but a precondition, because a regulator, a client contract or a data-residency rule says material must not leave a boundary. Where that applies, the question is not whether it is worth it but what capability you can get inside the constraint. Where it does not apply, it is a genuine trade rather than an obvious win, and the costs that get underestimated are people and time rather than hardware.

You need a model you are allowed to run, which in practice means one whose parameters have been published. Not every capable model is available this way, so the first thing self-hosting decides is which models are in scope at all, and that list is usually shorter than the one a provider offers through its service. Published parameters are what make this possible, and they do not mean you are doing it, since plenty of organisations use published models through somebody else's hosted service.

You need hardware, and specifically enough memory on the right kind of processor to hold the model while it runs. Larger models need considerably more, which is the main reason self-hosted deployments tend to run smaller models than the hosted ones people are comparing them against. Reduced-precision versions exist and trade some quality for fitting on less.

Somebody has to run it, and this is the cost that is routinely left out. Updates, monitoring, capacity, access control, backups and the person who gets called when it stops at an awkward moment are all now yours. It is an ordinary operational burden of the kind your organisation already carries elsewhere, and it is not zero.

Idle capacity costs what busy capacity costs, which inverts the economics people expect. A provider's service charges you nothing when nobody is asking anything. Hardware you own is paid for whether it is working or not, so the arrangement rewards steady heavy use and punishes the occasional bursts most organisations actually have.

Private cloud sits between the two and is where most real deployments land. You rent capacity that is yours alone, in a region you choose, with the model running under your control and nothing shared with a provider's general service. It answers most residency and confidentiality requirements without anybody having to buy a machine.

• Download a published model and run it on a machine you own, which is the smallest possible version of the whole arrangement.

  Ollama

• Load a document collection into a self-hostable assistant, so both the material and the searching stay inside your own network.

  AnythingLLM

• Run an automation platform on your own infrastructure so that the steps, the data passing through them and the logs are all somewhere you control.

  n8n

Is it cheaper than paying a provider?

Only with steady, high, predictable volume. Owning capacity converts a variable cost into a fixed one, so it wins when the machines are busy and loses when they are idle. Include the staff time to run and update it, and the crossover point sits further out than most first estimates put it.

Does it satisfy our data-residency obligations?

It can, and it is not automatic. What matters is where the machines physically are, who can administer them, and whether any part of the arrangement, such as monitoring or an update service, reaches outside the boundary. Those are the details an auditor will ask about, so establish them before treating the requirement as met.

How much hardware do we need?

Enough memory on the right kind of processor to hold the model you intend to run, which rises steeply with model size. Reduced-precision versions fit on considerably less at some cost to quality. The practical approach is choosing the smallest model that clears your own tests and sizing to that rather than to the largest thing available.

What is the part people underestimate?

The ongoing operational work. Updates, monitoring, capacity planning, access control and somebody being responsible when it stops on a Friday evening. The hardware is a purchase you make once and the running of it is a commitment, and it is the commitment rather than the purchase that decides whether this succeeds.

• Self-hosted means the model runs on machines your organisation controls rather than a provider's.
• It requires a model whose parameters were published, which narrows the choice.
• It converts a per-use cost into a fixed one, so it rewards steady heavy use and punishes bursts.
• It means data does not leave, which is narrower than and not the same as secure.
• The underestimated cost is the people who keep it running, not the hardware.

• Ollama

  The smallest version of the arrangement, on one machine you own.

• AnythingLLM

  Documents and searching both inside your own network.

• n8n

  Steps, data and logs on infrastructure you control.