Running AI Locally (Ollama)
What a local AI model needs — memory, graphics card, speed and model size. Rough estimates and recommended hardware for running AI Expert with Ollama on your own server.
On this page
With Local (Ollama) the AI model runs on your own hardware and no data leaves your network. That is a real advantage — but a language model is one of the most demanding applications you can run. An ordinary office laptop without a dedicated graphics card is not enough for practical use.
This chapter gives rough estimates to help you decide whether local AI makes sense for you and what hardware to plan for.
All numbers on this page are approximate. Real requirements depend on the model, its compression (quantization), the context length, the hardware and the drivers. Always test with your own data before you buy hardware.
What makes a model demanding
Model size — given in billions of parameters (B): 3b, 8b, 14b, 32b, 70b… The larger the model, the better it understands the question and the more memory it needs. Ollama normally downloads models compressed to about 4 bits per parameter, which needs roughly 0.6 GB of memory per billion parameters. For example, llama3:8b occupies about 4.4 GB, the embedding model snowflake-arctic-embed2 about 1.1 GB.
Context window — the amount of text the model processes with one question: the question itself, your system description, active alarms, tag values, history and document passages. AI Expert sets it in Context Window Length (2048 to 65535 tokens). Every token of context needs extra memory on top of the model, and the model has to read the whole context before it starts answering.
Graphics card (GPU) — language models run many times faster on a graphics card than on the processor. The model and its context should fit completely into the memory of the graphics card (VRAM). If they do not, Ollama moves part of the work to the processor and the answers become much slower.
Tools and thinking — root cause analysis needs a model with tool support (marked tools in the Ollama library). Think mode makes answers more accurate, but the model writes much longer internal reasoning, so an answer takes several times longer.
How much memory a model needs
A simple rule of thumb:
Memory needed ≈ size of the model file + memory for the context + 1–2 GB
| Model size | Example models | Model file (approx.) | Memory with context 8k–16k (approx.) |
|---|---|---|---|
| 3–4 B | llama3.2:3b, qwen3:4b, gemma3:4b | 2–3.5 GB | 4–6 GB |
| 7–8 B | llama3.1:8b, qwen3:8b | 5 GB | 7–9 GB |
| 14 B | qwen3:14b | 9 GB | 12–14 GB |
| 20–32 B | gpt-oss:20b, gemma3:27b, qwen3:32b | 14–20 GB | 18–26 GB |
| 70 B | llama3.1:70b | 40–45 GB | 48–56 GB |
| 120 B | gpt-oss:120b | 65 GB | 70–80 GB |
Context memory grows with the context length: for an 8B model roughly 1 GB per 8 000 tokens, for a 70B model roughly 2.5 GB per 8 000 tokens. A context of 32 000 tokens therefore adds about 4 GB to an 8B model.
2048 tokens are only a few pages of text — too little for many alarms, tags and history at once. For practical use plan for 8 000–32 000 tokens and choose hardware with enough memory for it. Alternatively lower the limits in the Advanced tab.
How fast it is
Speed is measured in tokens per second (one token is roughly ¾ of an English word). A typical answer has 150–400 tokens. Before writing, the model also has to read the whole context — on a processor alone this can take tens of seconds.
| Hardware | 7–8 B model | 14 B model | 30 B model | 70 B model |
|---|---|---|---|---|
| Office laptop, no graphics card | 2–8 tokens/s | 1–4 tokens/s | impractical | impossible |
| Graphics card 8 GB | 40–70 tokens/s | partly on CPU, slow | impractical | impossible |
| Graphics card 16 GB | 50–80 tokens/s | 30–50 tokens/s | partly on CPU | impossible |
| Graphics card 24 GB | 80–120 tokens/s | 50–70 tokens/s | 25–40 tokens/s | partly on CPU |
| 48–80 GB VRAM (one or more cards) | very fast | very fast | 40–60 tokens/s | 15–30 tokens/s |
| Apple Silicon with 32–128 GB unified memory | 30–60 tokens/s | 20–40 tokens/s | 10–25 tokens/s | 5–12 tokens/s (64 GB +) |
In practice: on an office laptop a single answer can take one to several minutes and the answers of the small models that fit are often incomplete. With a suitable graphics card the same answer arrives in a few seconds.
Recommended hardware
| Level | Typical hardware | Suitable models | What to expect |
|---|---|---|---|
| Not recommended | Office laptop or industrial PC without a dedicated graphics card, 8–16 GB RAM | at most 3–4 B | Only for trying it out. Slow, short and often inaccurate answers, no reliable root cause analysis. |
| Minimum | PC or workstation with a graphics card with 8 GB VRAM (e.g. GeForce RTX 4060 / 3060), 32 GB RAM | 7–8 B, context up to ~16k | Simple questions about alarms and values. Limited understanding of complex questions. Root cause analysis is impractical — in our test on a laptop with an RTX 5070 Laptop GPU (8 GB) and qwen2.5:7b it ran 29 minutes at full load and found no cause. |
| Recommended | Workstation with a graphics card with 16–24 GB VRAM (e.g. GeForce RTX 4060 Ti 16 GB, RTX 4090, RTX A4000/A5000), 64 GB RAM | 14–32 B with tool support, context 16–32k | Good answers, root cause analysis, work with documents. A good balance of price and quality. |
| High-end | Server with 48–80 GB VRAM (e.g. RTX 6000 Ada, A100/H100, or two cards), or Apple Mac Studio with 64–128 GB | 70 B and more, long context | Answers close to cloud models, several users at once. |
Other requirements:
- Disk — a fast SSD; each model takes 2–65 GB, plan space for several models.
- Operating system — Ollama runs on Linux, Windows and macOS. Graphics cards: NVIDIA (CUDA), AMD (ROCm), Apple Silicon (Metal).
- Power and cooling — under load a graphics card draws 100–450 W. A server running 24/7 needs proper cooling and a quiet place.
Where to run Ollama
- On the same computer as DataTalk — the simplest option when that computer has a suitable graphics card.
- As a plugin of DataTalk Manager — DataTalk Manager installs Ollama as the plugin
ollama(with graphics card) orollamacpu(processor only, much slower). The plugin is protected by a token: copy the bearer token from Plugins → Plugin settings → Ollama in DataTalk Manager into Authorization token in Ollama Settings. See Plugins. - On a separate AI server — usual in practice. DataTalk keeps running on its industrial PC or server, and the model runs on a machine with a graphics card in the same network. Enter its address, port and optional authorization token in Ollama Settings.
- Embeddings on another server — the embedding model is small (about 1 GB) and can run on a different Ollama server, see Use secondary Ollama for embeddings.
Ollama processes the questions of several users one after another, or in parallel when it has enough memory — every parallel request needs its own context memory. Plan more VRAM if several operators will ask at the same time. While the server is busy — for example with a long root cause analysis on a small server — new questions are answered with Wait in que. and have to be asked again later.
Local or cloud?
| Local (Ollama) | Cloud (OpenAI, Claude, Gemini) | |
|---|---|---|
| Data | Stays in your network | Sent to the provider |
| Answer quality | Depends on the hardware; large models need expensive hardware | The best available models |
| Costs | Hardware and electricity, no fee per question | No hardware, fee per question |
| Internet | Not needed (except for downloading models) | Required |
If you do not have a computer with a suitable graphics card, start with a cloud provider — it gives good answers immediately. Choose local AI when data must not leave the site, and plan hardware at least at the Recommended level.
The comparison of a local and a cloud model shows how big the difference in quality is on the same questions.