How to Install Ollama on Ubuntu & Linux Servers (Complete Service Setup)

Linux is where Ollama behaves like the production-grade service it’s designed to be: a systemd daemon, proper GPU device access, headless operation, and Docker-friendly networking. This guide walks Ubuntu 24.04/22.04 LTS and Debian 12 through the install, the GPU permissions that actually matter, and the LAN/firewall setup for a headless box. For the command reference and architecture, see the Ollama Complete Guide.

1. Prerequisites

Before installing, make sure the system is current and that your NVIDIA driver is loaded — the number-one “Ollama runs on CPU on my Linux box” cause is a missing/old driver that systemd can’t see:

sudo apt update && sudo apt upgrade -y
nvidia-smi

If nvidia-smi errors, install the proprietary driver before proceeding. Ollama can still run on CPU, but you’ll want the GPU working first — otherwise you’ll be tearing your hair out later wondering why it’s slow.

2. Install with the official script

Run the installer:

curl -fsSL https://ollama.com/install.sh | sh

It does all of this automatically:

• Puts the binary at /usr/local/bin/ollama.
• Creates a dedicated ollama system user and group.
• Writes the unit at /etc/systemd/system/ollama.service.
• Enables and starts the daemon.

Verify the daemon is up

systemctl status ollama

You should see Active: active (running). Note: because it runs as a system service, sudo systemctl status ollama works, but ollama list from your normal shell may need the ollama binary to talk to the daemon over 127.0.0.1:11434 — that’s fine once the daemon binds localhost (default).

Test with a small model

ollama run llama3.2

3. GPU permissions that actually matter

The subtle part: the ollama service user must be able to open GPU device nodes. On a fresh Ubuntu install with proper drivers, NVIDIA GPUs work out of the box for the service user in most cases, but if you see a CPU-only fallback or a “no GPU” log line, add the service user to the GPU groups and restart:

sudo usermod -aG video,render ollama
sudo systemctl restart ollama

For AMD Radeon + ROCm, the same two groups matter (the device nodes are /dev/kfd and /dev/dri). Some consumer RDNA cards need a ROCm architecture override so the runtime doesn’t reject the chip. Edit the service environment and add something like:

sudo systemctl edit ollama.service

[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.0"

This is model/arch-specific; 11.0.0 suits RDNA3. Confirm GPU offload after a restart with ollama ps while a model is loaded — if PROCESSOR shows 100% CPU, the GPU isn’t being used.

4. Headless LAN access and firewall

By default the daemon binds only to 127.0.0.1, which is exactly right for local use but useless if you want another machine on your LAN (or a Docker container) to reach it. To listen on the network, use a systemd drop-in — never edit the packaged unit, because an Ollama update will overwrite it:

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_ORIGINS=*"

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Security note worth actually reading: if you bind to 0.0.0.0 you have just exposed Ollama’s unauthenticated API to anyone who can reach that interface — including the whole internet if the port is open in your router. Ollama has no built-in auth. Only do this on a trusted LAN, restrict it with a firewall, and consider a reverse proxy with authentication if you must expose it further. A minimal UFW rule for a private subnet:

sudo ufw allow from 192.168.1.0/24 to any port 11434 proto tcp

Leaving OLLAMA_ORIGINS as * with a wildcard can permit cross-site requests; set it to your specific UI/domain when you can. The full variable list is in the Ollama model storage and environment guide.

5. Day-to-day service commands

To reclaim disk from old models on a busy server, see How to delete and remove Ollama models.

Frequently asked questions

Where do models live on Linux?

Under the systemd service, in the service user’s home at /usr/share/ollama/.ollama/models. If you run ollama manually as yourself (not via the service), they go to ~/.ollama/models.

How do I fully uninstall from Ubuntu?

Stop and disable the service, remove the unit, binary, and data:

sudo systemctl stop ollama && sudo systemctl disable ollama
sudo rm /etc/systemd/system/ollama.service
sudo rm $(which ollama)
sudo rm -rf /usr/share/ollama
sudo systemctl daemon-reload

How do I wire up Open-WebUI or AnythingLLM?

With OLLAMA_HOST set to a reachable address, point the UI at http://<server-ip>:11434. Runnable setup is in Setting up Open-WebUI and AnythingLLM.