Ollama Complete Guide: Setup, Models, Commands, GPU & Troubleshooting

Updated 24 October 2026 · Sources & corrections

When you hear “I run LLMs locally,” what people almost always mean is Ollama. It has become the default way to run open-weight models on a laptop or workstation, and yet most guides stop at ollama run llama3.2 and call it a day. This one does not. It assumes you actually want to operate Ollama day to day — load models into the right amount of VRAM, serve them to other devices, hand-tune context and sampling, and recover when something breaks.

Everything below is organized the way real work flows: understand the moving parts, install correctly per OS, learn the CLI, tune with Modelfiles, budget your VRAM, expose the API, and diagnose failures. The linked diagnostic guides explain individual failure modes in more detail.

1. How Ollama Works: the daemon, the GGUF file, and why VRAM matters

Ollama is a client–server application. The installer launches a background HTTP server bound to 127.0.0.1:11434, and every command you type in the ollama CLI is just a request to that server. Keep that model in mind, because it explains a huge share of real-world confusion: “the CLI works but my app can’t connect” and “why is port 11434 open?” are all consequences of a separate server process.

Quantization labels are not memory guarantees

Quantized builds store model weights using different numerical representations. Compare the exact artifact and its runtime behavior rather than treating a label such as Q4_K_M as a universal quality score or default.

The official Llama 3.1 tag listing, for example, distinguishes several quantized builds of the same family. Select an explicit tag and record its model identity. Two similarly named tags can also differ in base checkpoint, instruction tuning or template, so a difference in answers is not automatically caused by quantization.

A model-fit estimate must account for weights, context/KV cache, working buffers and other users of memory. These terms vary with the model architecture, backend and workload. Section 5 gives a measurement process rather than a fixed extra-gigabytes rule.

2. Cross-platform installation, done deliberately

The install itself is one line on most systems, but the environment you install into changes the outcome more than the command does. Here is what to think about on each OS before you run anything.

Windows: the native installer vs. WSL2

OllamaSetup.exe installs the daemon as a background tray app and adds ollama to your PATH. Two things trip people up on Windows specifically:

• The native build uses CUDA for NVIDIA GPUs, but if you have an older GPU or no NVIDIA card, it silently falls back to CPU. Run ollama --version and check ollama ps after loading a model — inspect the PROCESSOR column to see reported CPU/GPU placement.
• If you plan to call Ollama from inside WSL2 while running it natively in Windows, remember the Windows daemon listens on 127.0.0.1 only, and WSL2 has its own network namespace. When both halves are on the same machine you can usually reach the Windows daemon via host.docker.internal or the machine’s LAN IP with OLLAMA_HOST set appropriately.

For the full native-install, first-run, and CUDA-verification walkthrough, see How to Install Ollama on Windows.

Linux and Ubuntu server: you get a real service

On Linux the installer registers a proper systemd service, which is the biggest difference from Windows and macOS — you get auto-start, logging via journalctl -u ollama, and sane permission handling.

curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama

The important follow-ups most guides omit: the service runs as a dedicated ollama user (so models live under /usr/share/ollama/.ollama, not ~/.ollama), and on a headless box nothing listens on a reachable interface until you set OLLAMA_HOST. Remote access, firewalling to a specific LAN, and hardening are covered in How to Install Ollama on Ubuntu.

macOS: Metal and unified memory

Apple Silicon runs models through Metal directly in unified memory, which is why an 8 GB RAM Mac can sometimes run an 8B quant that would be tight on a 6 GB Nvidia card — there is no separate VRAM pool to overflow. Homebrew is the clean route:

brew install ollama
brew services start ollama

A practical tip: on 8–16 GB base-model Macs you will hit a wall faster than the GPU spec suggests, because system RAM is shared with the OS and other apps. Budget the model plus a few GB for macOS itself. More in How to Install Ollama on macOS.

3. The CLI, and what each command actually tells you

These are the commands you will use weekly, with the detail that matters.

Where models land and how to relocate them to a bigger disk (a common early mistake — pulling a 70B quant fills your system drive overnight) is the subject of Where Ollama Stores Models & How to Move Them.

4. Modelfiles: packaging repeatable behavior, not just prompts

A Modelfile packages a base model, prompt/template configuration and runtime parameters. Record the file with your evaluation so changes in prompts or context are not mistaken for changes in the underlying model.

FROM qwen2.5-coder:7b

# A system prompt that changes the model's working style.
SYSTEM """You write production-grade Go and Python.
Prefer explicit error handling. Never return pseudo-code."""

# Sampling: lower temperature reduces randomness; it does not guarantee determinism.
PARAMETER temperature 0.2
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.1

# Example context setting; choose a value your model and hardware can support.
PARAMETER num_ctx 32768

# Inspect actual CPU/GPU placement with ollama ps.

Two directives are worth understanding more than the rest:

• num_ctx requests a context allocation. Check client overrides and the running allocation before changing it; a larger value consumes more memory and does not guarantee better retention or answers.
• GPU placement depends on supported hardware, backend and runtime configuration. Inspect the running model instead of assuming a layer-count setting establishes CPU-only operation.

Build and launch your tuned model with two commands:

ollama create repo-assistant -f ./Modelfile
ollama run repo-assistant

The three server-side variables you will actually reach for day to day are OLLAMA_HOST (where the daemon binds), OLLAMA_MODELS (where weights live), and OLLAMA_KEEP_ALIVE (how long loaded models stay resident). Each is covered where it applies below; for moving where models are stored, see Ollama Model Storage.

5. Budget memory and verify GPU placement

Start with a supported model build and a modest context setting that fits the task. The official context guide documents that larger context increases memory requirements. A download-size table cannot supply a universal VRAM requirement for every context, image input or concurrent workload.

1. Record the exact model tag, quantization, runner version and hardware.
2. Keep prompt, context, concurrency and generation settings fixed while comparing builds.
3. Load one candidate, inspect its placement and observe memory with the platform’s normal system tools.
4. Check correctness on tasks with known outcomes before accepting a smaller or faster build.
5. Retain the working configuration so a failed comparison can be reversed.

Use ollama ps on the server receiving your request. The PROCESSOR column describes CPU/GPU placement. CPU use or partial offload is evidence to investigate; it does not by itself prove the bottleneck, driver failure or expected token rate. Compare actual timings and workload before deciding whether to reduce context, choose a different build or investigate the backend.

The running-model API also exposes context and build metadata. This PowerShell example is read-only and expects a local server with a model already loaded:

(Invoke-RestMethod -Uri 'http://localhost:11434/api/ps').models |
    ForEach-Object {
        [pscustomobject]@{
            model = $_.name
            quantization = $_.details.quantization_level
            context_length = $_.context_length
            reported_size_bytes = $_.size
            reported_vram_bytes = $_.size_vram
        }
    }

The response is a snapshot, not a peak-memory profiler. An empty list may mean the model unloaded or that you queried another server. Record the endpoint and observation time; avoid another client’s request changing the setup during a comparison. Do not treat the reported size fields as the operating system’s complete memory total.

Use our context-override diagnostic guide for settings discrepancies and the reload-delay guide for idle-gap delays. Broader failures are covered in Ollama troubleshooting. No model fit, speed or accuracy result is claimed by this checklist.

6. The REST API: calling Ollama from your own code

The daemon exposes two families of endpoints on 11434: the native /api/* routes and an OpenAI-compatible /v1/* surface. If you are starting fresh, prefer the OpenAI-compatible route — it means you can point existing OpenAI SDK code at Ollama by swapping the base_url, and swapping back to a hosted model later is a one-line change.

If your Python client reports JSON Extra data or returns only part of an answer, follow our streamed JSON parsing guide with locally tested completion and error checks.

Non-streaming generation via cURL

curl http://127.0.0.1:11434/api/generate -d '{
  "model": "qwen2.5:7b",
  "prompt": "Write a compliant .gitignore for a Node project in 40 words.",
  "stream": false
}'

OpenAI-compatible chat (the portability win)

import openai

client = openai.OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="ollama")
resp = client.chat.completions.create(
    model="qwen2.5:7b",
    messages=[{"role": "user", "content": "Explain RAG in two sentences."}],
)
print(resp.choices[0].message.content)

Two practical notes. First, streaming (stream: true, or stream=True in the SDK) is what makes long generations feel responsive in a UI — otherwise you wait for the whole completion. Second, keep_alive controls how long a model stays in memory after a request: the default unloads after 5 minutes, which is a common source of mysteriously slow “first” requests as a model reloads. To put a GUI front-end on top of this API for chat and document workflows, see Connecting AnythingLLM and Open-WebUI to Ollama.

7. Serving Ollama to other machines (and doing it safely)

By default OLLAMA_HOST binds to 127.0.0.1, so only localhost can reach it — which is correct and safe until you explicitly want LAN access. To expose it, set the bind address and restart:

# Linux server: listen on all interfaces
sudo systemctl stop ollama
sudo systemctl edit ollama
# add: [Service]
#      Environment="OLLAMA_HOST=0.0.0.0:11434"
sudo systemctl daemon-reload
sudo systemctl start ollama

Before you do this, note the honest security reality: Ollama’s API has no built-in authentication. Binding to 0.0.0.0 on a public interface lets anyone on the network run your GPU. Only expose it if you have blocked the port at the firewall, put it behind a reverse proxy with auth, or trust every host on that LAN. OLLAMA_ORIGINS helps with browser-based CSRF-style calls from a web app, but it is not a substitute for a firewall. This is why so many published guides recommend OLLAMA_HOST=0.0.0.0 for “remote” and it quietly becomes a security issue.

8. Where Ollama fits next to the other local runtimes

Ollama is the popular middle ground, but it’s not the only player, and each alternative wins on a different axis:

• llama.cpp vs. Ollama: llama.cpp is the engine beneath Ollama. Running it directly (via llama-server) exposes every flag and lets you script or embed inference with zero packaging; Ollama gives you a model registry, a managed daemon, and a friendlier API. If you need fine-grained control or plan to ship the engine inside your own app, llama.cpp wins; for daily server hosting, Ollama’s ergonomics usually win.
• LM Studio vs. Ollama: LM Studio is a graphical app with model browsing, hardware meters, and a chat UI — ideal if you are not running a headless service. Ollama is lighter, scriptable, and better suited to API-driven and background use.
• vLLM: built for high-concurrency serving (PagedAttention, continuous batching). Running a 7B behind a REST endpoint for a handful of local users, vLLM is overkill; Ollama is the pragmatic local choice. For production multi-user throughput, vLLM leads.

If you want a full GUI on top of Ollama for chat and document workflows, see Connecting AnythingLLM and Open-WebUI to Ollama.

9. Diagnostics and fixes for the errors that actually happen

These are the failures I see most, with the real underlying cause rather than the copy-paste answer.

For the full diagnostic workflow — reading journalctl -u ollama, checking GPU offload, CORS, and network cases — work through Ollama Troubleshooting.

Frequently Asked Questions

Is Ollama free and open-source?

Yes. The runtime is MIT-licensed and the models it pulls are open-weight. You pay nothing and there are no API rate limits or telemetry-based throttling; the only cost is your own hardware.

Where does Ollama keep models, and can I move them?

Defaults are C:\Users\<you>\.ollama\models (Windows), /usr/share/ollama/.ollama/models (Linux service), and ~/.ollama/models (macOS / Linux user install). Point OLLAMA_MODELS at a new directory before starting the daemon to relocate. Details and gotchas in the model storage guide.

Can I run Ollama without a GPU?

CPU-only and partially offloaded inference are possible. Inspect placement with ollama ps, then measure the actual request on your hardware; a fixed CPU-versus-GPU speed range is not reliable across configurations.

Why is the context window truncating my conversation?

Because num_ctx defaults are small. Either raise it in the Modelfile (PARAMETER num_ctx 32768) or set it per-session with /set parameter num_ctx inside ollama run. Watch VRAM when you go up.

How do I update Ollama?

On Windows/macOS the tray/menu app prompts you. On Linux, re-run curl -fsSL https://ollama.com/install.sh | sh; it replaces the binary and leaves your downloaded models untouched.

Can I run the same on my phone or another computer on my network?

Set OLLAMA_HOST to bind beyond localhost, allow the port through the firewall, and point whichever client or app you like at http://<machine-ip>:11434. Keep auth/firewalling in mind — there is no built-in password layer.

Editorial update, 24 October 2026: replaced fixed VRAM formulas and hardware-fit promises with build-identification and allocation checks, and removed incorrect CPU-offload advice. The new PowerShell snapshot example was syntax-checked; no live model or hardware benchmark was run. The featured image is an original generated illustration.