Ollama Complete Guide: Setup, Models, Commands, GPU & Troubleshooting

When you hear “I run LLMs locally,” what people almost always mean is Ollama. It has become the default way to run open-weight models on a laptop or workstation, and yet most guides stop at ollama run llama3.2 and call it a day. This one does not. It assumes you actually want to operate Ollama day to day — load models into the right amount of VRAM, serve them to other devices, hand-tune context and sampling, and recover when something breaks.

Everything below is organized the way real work flows: understand the moving parts, install correctly per OS, learn the CLI, tune with Modelfiles, budget your VRAM, expose the API, and diagnose failures. Along the way I flag the mistakes that are easy to make and painful to undo.

1. How Ollama Works: the daemon, the GGUF file, and why VRAM matters

Ollama is a client–server application. The installer launches a background HTTP server bound to 127.0.0.1:11434, and every command you type in the ollama CLI is just a request to that server. Keep that model in mind, because it explains a huge share of real-world confusion: “the CLI works but my app can’t connect” and “why is port 11434 open?” are all consequences of a separate server process.

The single most important number in local LLMs is bytes-per-parameter. An unquantized FP16 model uses 2 bytes per parameter, so an 8B model needs ~16 GB just for weights. The GGUF format instead ships quantized weights that use far fewer bytes:

Those sizes are weights only — almost nobody warns you that the context window eats VRAM on top. A 32K-token context on an 8B model adds roughly 1–2 GB depending on architecture (the KV cache grows with both context length and layer count). So the honest rule for budgeting is: weights + ~2 GB for the KV cache + a little headroom = what you actually need in VRAM. I work through the exact numbers in Section 5.

2. Cross-platform installation, done deliberately

The install itself is one line on most systems, but the environment you install into changes the outcome more than the command does. Here is what to think about on each OS before you run anything.

Windows: the native installer vs. WSL2

OllamaSetup.exe installs the daemon as a background tray app and adds ollama to your PATH. Two things trip people up on Windows specifically:

• The native build uses CUDA for NVIDIA GPUs, but if you have an older GPU or no NVIDIA card, it silently falls back to CPU. Run ollama --version and check ollama ps after loading a model — if the GPU column is empty, you are on CPU.
• If you plan to call Ollama from inside WSL2 while running it natively in Windows, remember the Windows daemon listens on 127.0.0.1 only, and WSL2 has its own network namespace. When both halves are on the same machine you can usually reach the Windows daemon via host.docker.internal or the machine’s LAN IP with OLLAMA_HOST set appropriately.

For the full native-install, first-run, and CUDA-verification walkthrough, see How to Install Ollama on Windows.

Linux and Ubuntu server: you get a real service

On Linux the installer registers a proper systemd service, which is the biggest difference from Windows and macOS — you get auto-start, logging via journalctl -u ollama, and sane permission handling.

curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama

The important follow-ups most guides omit: the service runs as a dedicated ollama user (so models live under /usr/share/ollama/.ollama, not ~/.ollama), and on a headless box nothing listens on a reachable interface until you set OLLAMA_HOST. Remote access, firewalling to a specific LAN, and hardening are covered in How to Install Ollama on Ubuntu.

macOS: Metal and unified memory

Apple Silicon runs models through Metal directly in unified memory, which is why an 8 GB RAM Mac can sometimes run an 8B quant that would be tight on a 6 GB Nvidia card — there is no separate VRAM pool to overflow. Homebrew is the clean route:

brew install ollama
brew services start ollama

A practical tip: on 8–16 GB base-model Macs you will hit a wall faster than the GPU spec suggests, because system RAM is shared with the OS and other apps. Budget the model plus a few GB for macOS itself. More in How to Install Ollama on macOS.

3. The CLI, and what each command actually tells you

These are the commands you will use weekly, with the detail that matters.

Where models land and how to relocate them to a bigger disk (a common early mistake — pulling a 70B quant fills your system drive overnight) is the subject of Where Ollama Stores Models & How to Move Them.

4. Modelfiles: packaging repeatable behavior, not just prompts

A Modelfile is Ollama’s declarative recipe. The typical tutorial shows a system prompt and a temperature, and that’s honestly 30% of the value. The other 70% is the parts that change how the model behaves under the hood: TEMPLATE, num_ctx, and the ability to FROM an already-custom model to layer behavior.

FROM qwen2.5-coder:7b

# A system prompt that changes the model's working style.
SYSTEM """You write production-grade Go and Python.
Prefer explicit error handling. Never return pseudo-code."""

# Sampling: lower temperature for deterministic code.
PARAMETER temperature 0.2
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.1

# The context a real repo-scope task needs. 8K is the default; 32K uses more VRAM.
PARAMETER num_ctx 32768

# Which device runs the layers: 0 = auto, -1 = CPU only, 99 = push all to GPU.
PARAMETER num_gpu 99

Two directives are worth understanding more than the rest:

• num_ctx sets the context window the model is loaded with. This is the #1 cause of “it keeps forgetting our conversation.” The default is small; raise it for real tasks, but know each step up costs KV-cache VRAM.
• num_gpu controls layer offloading. 99 means “use the GPU for everything it can.” On machines where the driver is flaky, some people pin -1 (CPU-only) to force stability at the cost of speed.

Build and launch your tuned model with two commands:

ollama create repo-assistant -f ./Modelfile
ollama run repo-assistant

The three server-side variables you will actually reach for day to day are OLLAMA_HOST (where the daemon binds), OLLAMA_MODELS (where weights live), and OLLAMA_KEEP_ALIVE (how long loaded models stay resident). Each is covered where it applies below; for moving where models are stored, see Ollama Model Storage.

5. VRAM budgeting and GPU offload, with the math

The distinction that saves hours of confusion: a model either fits in VRAM and runs fast, or it spills to system RAM and runs slow — there isn’t really an in-between. Ollama will happily offload what doesn’t fit, but the moment layers hit CPU RAM you typically drop from hundreds to single-digit tokens per second.

Here is how to estimate VRAM before you download anything. A Q4_K_M 8B model is ~4.7 GB of weights. Add a 16K KV cache (~1 GB) and ~0.5 GB overhead, and you want ~6.5 GB of VRAM. On a 8 GB card that is right at the edge and will probably spill; on 12 GB it’s comfortable and fast. Use ollama ps after loading to confirm with the real number.

If your generation rate “looks slow,” the fastest diagnosis is ollama ps — not the model’s advertised benchmarks. When “PROCESSOR” shows CPU/RAM, that’s your bottleneck, and the fix is either more VRAM, a smaller quant, a smaller model, or a truncated num_ctx. For the step-by-step NVIDIA/AMD driver setup and the out-of-memory debugging flow, pair this with the Ollama Troubleshooting guide.

6. The REST API: calling Ollama from your own code

The daemon exposes two families of endpoints on 11434: the native /api/* routes and an OpenAI-compatible /v1/* surface. If you are starting fresh, prefer the OpenAI-compatible route — it means you can point existing OpenAI SDK code at Ollama by swapping the base_url, and swapping back to a hosted model later is a one-line change.

Non-streaming generation via cURL

curl http://127.0.0.1:11434/api/generate -d '{
  "model": "qwen2.5:7b",
  "prompt": "Write a compliant .gitignore for a Node project in 40 words.",
  "stream": false
}'

OpenAI-compatible chat (the portability win)

import openai

client = openai.OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="ollama")
resp = client.chat.completions.create(
    model="qwen2.5:7b",
    messages=[{"role": "user", "content": "Explain RAG in two sentences."}],
)
print(resp.choices[0].message.content)

Two practical notes. First, streaming (stream: true, or stream=True in the SDK) is what makes long generations feel responsive in a UI — otherwise you wait for the whole completion. Second, keep_alive controls how long a model stays in memory after a request: the default unloads after 5 minutes, which is a common source of mysteriously slow “first” requests as a model reloads. To put a GUI front-end on top of this API for chat and document workflows, see Connecting AnythingLLM and Open-WebUI to Ollama.

7. Serving Ollama to other machines (and doing it safely)

By default OLLAMA_HOST binds to 127.0.0.1, so only localhost can reach it — which is correct and safe until you explicitly want LAN access. To expose it, set the bind address and restart:

# Linux server: listen on all interfaces
sudo systemctl stop ollama
sudo systemctl edit ollama
# add: [Service]
#      Environment="OLLAMA_HOST=0.0.0.0:11434"
sudo systemctl daemon-reload
sudo systemctl start ollama

Before you do this, note the honest security reality: Ollama’s API has no built-in authentication. Binding to 0.0.0.0 on a public interface lets anyone on the network run your GPU. Only expose it if you have blocked the port at the firewall, put it behind a reverse proxy with auth, or trust every host on that LAN. OLLAMA_ORIGINS helps with browser-based CSRF-style calls from a web app, but it is not a substitute for a firewall. This is why so many published guides recommend OLLAMA_HOST=0.0.0.0 for “remote” and it quietly becomes a security issue.

8. Where Ollama fits next to the other local runtimes

Ollama is the popular middle ground, but it’s not the only player, and each alternative wins on a different axis:

• llama.cpp vs. Ollama: llama.cpp is the engine beneath Ollama. Running it directly (via llama-server) exposes every flag and lets you script or embed inference with zero packaging; Ollama gives you a model registry, a managed daemon, and a friendlier API. If you need fine-grained control or plan to ship the engine inside your own app, llama.cpp wins; for daily server hosting, Ollama’s ergonomics usually win.
• LM Studio vs. Ollama: LM Studio is a graphical app with model browsing, hardware meters, and a chat UI — ideal if you are not running a headless service. Ollama is lighter, scriptable, and better suited to API-driven and background use.
• vLLM: built for high-concurrency serving (PagedAttention, continuous batching). Running a 7B behind a REST endpoint for a handful of local users, vLLM is overkill; Ollama is the pragmatic local choice. For production multi-user throughput, vLLM leads.

If you want a full GUI on top of Ollama for chat and document workflows, see Connecting AnythingLLM and Open-WebUI to Ollama.

9. Diagnostics and fixes for the errors that actually happen

These are the failures I see most, with the real underlying cause rather than the copy-paste answer.

For the full diagnostic workflow — reading journalctl -u ollama, checking GPU offload, CORS, and network cases — work through Ollama Troubleshooting.

Frequently Asked Questions

Is Ollama free and open-source?

Yes. The runtime is MIT-licensed and the models it pulls are open-weight. You pay nothing and there are no API rate limits or telemetry-based throttling; the only cost is your own hardware.

Where does Ollama keep models, and can I move them?

Defaults are C:\Users\<you>\.ollama\models (Windows), /usr/share/ollama/.ollama/models (Linux service), and ~/.ollama/models (macOS / Linux user install). Point OLLAMA_MODELS at a new directory before starting the daemon to relocate. Details and gotchas in the model storage guide.

Can I run Ollama without a GPU?

Yes, but know the trade-off: an 8B Q4 model runs at roughly 3–10 tokens/sec on a modern CPU versus 60–120 tokens/sec on a current GPU. Fine for interactive light use; painful for big context or long generations.

Why is the context window truncating my conversation?

Because num_ctx defaults are small. Either raise it in the Modelfile (PARAMETER num_ctx 32768) or set it per-session with /set parameter num_ctx inside ollama run. Watch VRAM when you go up.

How do I update Ollama?

On Windows/macOS the tray/menu app prompts you. On Linux, re-run curl -fsSL https://ollama.com/install.sh | sh; it replaces the binary and leaves your downloaded models untouched.

Can I run the same on my phone or another computer on my network?

Set OLLAMA_HOST to bind beyond localhost, allow the port through the firewall, and point whichever client or app you like at http://<machine-ip>:11434. Keep auth/firewalling in mind — there is no built-in password layer.