On-premise AI means running AI models on hardware your organisation controls — your own servers, in your own building or data centre — instead of sending prompts and documents to a third-party cloud API. The model weights, the retrieval index and the logs all stay inside your network perimeter.
Why organisations move AI on-premise
Three reasons come up repeatedly, and only one of them is cost.
Data that cannot leave
Legal files, patient records, engineering drawings, HR investigations, classified material, unreleased financials. For these, "we have a data processing agreement with the vendor" is not an answer — the requirement is that the material never traverses a network you do not own. This is the reason that does not have a price attached, and it is the one that decides most deployments.
Predictable cost at volume
Cloud APIs bill per token. A pilot that costs a few hundred dollars a month becomes a five-figure line item when it succeeds and the whole department starts using it. On-premise inverts the curve: high capital cost up front, then near-zero marginal cost per query. The crossover point depends on utilisation — see the comparison below.
Model stability
Hosted models are deprecated and silently updated. If you have validated a workflow against a specific model's behaviour — and especially if you have documented that validation for a regulator — a provider-side update invalidates it. Open weights on your own hardware do not change unless you change them.
On-premise vs cloud: an honest comparison
| Dimension | Cloud API | On-premise |
|---|---|---|
| Time to first result | Hours | Weeks |
| Cost shape | Per token, unbounded | Capital + power + staff |
| Frontier capability | Highest available | Open-weight tier, 6–12 months behind |
| Data egress | Leaves your network | None |
| Model stability | Provider-controlled | You freeze it |
| Scaling burst load | Elastic | Capped by your hardware |
| Ops burden | Near zero | Real — GPUs, drivers, updates |
If your data can legally sit in a hyperscaler and your volume is modest, use a cloud API. On-premise is not a general upgrade — it is a specific answer to confidentiality, cost-at-volume and stability constraints. Anyone selling it as a blanket improvement is selling hardware.
How to run AI on-premise: the four components
Every working on-premise deployment has the same four parts. Getting one wrong is what causes pilots to stall.
1 · Hardware sized to the model
Memory is the binding constraint, not compute. A model's weights must fit in VRAM, plus headroom for the KV cache that grows with context length and concurrent users. Rough guide for common open-weight sizes at typical quantisation:
| Model class | VRAM (4-bit) | VRAM (8-bit) | Typical fit |
|---|---|---|---|
| 7–9B | ~6 GB | ~10 GB | Single consumer GPU |
| 27–32B | ~20 GB | ~35 GB | One 48 GB card |
| 49–70B | ~40 GB | ~75 GB | 1–2 datacentre GPUs |
| 120B+ (MoE) | ~65 GB | ~120 GB | Multi-GPU node |
Add roughly 20–40% on top for concurrency. The common mistake is sizing for the weights alone, then discovering the system falls over at eight simultaneous users.
2 · An inference server with a standard interface
Serve the model behind an OpenAI-compatible endpoint. This is the single highest-leverage architectural decision available, and it costs nothing: every client library, evaluation harness, IDE plugin and agent framework already speaks that protocol. You keep the freedom to swap the model underneath without touching a single consumer.
3 · Retrieval over your own material
A base model knows nothing about your organisation. Value comes from grounding answers in your documents: ingest, chunk, embed, index, and retrieve at query time. Two requirements that are easy to skip and expensive to retrofit — every answer must cite the source passages that produced it, and the pipeline must handle the languages your documents are actually written in. Retrieval quality in Arabic, for example, degrades badly with tokenisers and embedding models trained predominantly on English.
4 · Governance you can hand to an auditor
The question that ends unprepared pilots is: show me who asked what, what the system answered, and which documents it used. Build for it from the start — per-query logging tied to identity, immutable records, document-level access control that survives retrieval, and a human approval gate on any action with consequences.
Air-gapped deployment
An air-gapped deployment runs on a machine with no network route to the internet at all. Weights and documents arrive physically; nothing leaves. It is the strongest form of on-premise AI and the only configuration some defence, government and regulated financial environments will accept.
What changes in practice:
- No package installs at runtime. Every dependency, model file and container image is staged in advance and transferred deliberately.
- No telemetry, no licence checks, no auto-update. Any component that phones home is disqualified — which rules out most commercial AI tooling.
- Updates are events, not background tasks. A model or dependency upgrade is a scheduled, documented, re-validated procedure.
- Verification is offline. Checksums and signed manifests for every artefact, so you can prove later exactly what was running.
"Air-gap ready" and "air-gapped" are different claims. Ask any vendor for the specific list of outbound connections their system makes with the network cable pulled — including licence validation and crash reporting. The answer should be "none", and it should be demonstrable, not asserted.
Five mistakes that stall on-premise projects
- Sizing for weights, not concurrency. Works in the demo, collapses in week one.
- Skipping citations. Answers without traceable sources cannot be trusted, defended, or signed off — and retrofitting provenance means rebuilding the retrieval layer.
- Treating retrieval quality as a model problem. Most bad answers come from bad retrieval, not a weak model. Measure retrieval separately before you blame the LLM.
- Ignoring the language reality. If half your corpus is Arabic and your embedding model was trained on English, no amount of GPU solves it.
- No frozen baseline. Without a fixed model version and a stable evaluation set, you cannot tell whether a change made things better or worse.
Common questions
What is on-premise AI?
Running AI models on hardware your organisation controls — your own servers, in your own building or data centre — rather than sending prompts and documents to a third-party cloud API. Model weights, retrieval index and logs all stay inside your network perimeter.
Is on-premise AI cheaper than cloud AI?
It depends on utilisation. Cloud bills per token, so cost scales with success and is hard to forecast. On-premise is fixed capital plus power and staff. Below roughly a few million tokens per day, cloud is usually cheaper. Above that — and wherever data cannot leave the building at any price — on-premise wins.
Which models can I run on my own hardware?
Any open-weight model: the Llama, Qwen, Mistral, Gemma and Nemotron families among others. Licences differ and some restrict commercial use at scale, so read the specific licence rather than assuming "open" means unrestricted.
How far behind frontier models are open weights?
Roughly six to twelve months on general capability, and closing. For grounded question-answering over your own documents — the most common enterprise use — the gap matters far less than retrieval quality, because the model is reading your material rather than recalling its training data.
Do I need an air gap, or is on-premise enough?
On-premise with controlled egress covers most commercial confidentiality requirements. A true air gap is for classified, defence and certain regulated environments where the requirement is physical, not contractual. Air gap costs more in operational friction — do not adopt it unless the requirement is real.
Building this in the Gulf
LocalForge designs, deploys and operates on-premise AI systems — Arabic-first, air-gap-ready, built on open models, with citation and audit trails as a baseline rather than a feature. We run the same stack in our own group companies before we put our name behind it in yours.