If Ollama is not using GPU acceleration, your inference is probably running ten times slower than it should — and the fix is usually a driver, version, or VRAM setting away. Most cases of Ollama ignoring your graphics card come down to three things: an outdated NVIDIA driver, a model that silently fell back to CPU, or a context window that exceeds your VRAM.
This guide covers the full troubleshooting path for anyone running local AI models: verifying GPU detection, fixing CUDA and driver issues, resolving VRAM overflow and out-of-memory errors on Windows, Linux, and Apple Silicon, and decoding the vague “500 internal server error.” Each section gives you the exact commands to diagnose and correct the problem.
nvidia-smi to confirm the driver sees the card, then update Ollama and your NVIDIA driver. While a model runs, check ollama ps — it should report “100% GPU.” If it shows CPU, lower num_ctx, use a smaller quantization, or reduce GPU layers to fit VRAM.Step 1: Verify Ollama Actually Detects Your GPU
Before changing anything, confirm what Ollama sees. A misconfigured system often reports a healthy GPU while Ollama quietly runs on the CPU.
Start with the NVIDIA system tool. Open a terminal and run:
nvidia-smiIf this command fails or shows no device, the problem is at the driver level, not Ollama — skip to the CUDA section below. If it lists your card, check whether Ollama loads models onto it.
Run any model, then in a second terminal execute:
ollama run llama3.1 "hello"
ollama psThe ollama ps output includes a PROCESSOR column. A healthy setup reads “100% GPU”; anything like “48%/52% CPU/GPU” or “100% CPU” means part or all of the model spilled to system memory.
You can also watch live utilization: Task Manager’s GPU graph on Windows, or watch -n 1 nvidia-smi on Linux.
Step 2: Fix CUDA Toolkit and Driver Problems
Ollama bundles its own CUDA libraries on Windows and Linux, so you usually do not need a separate CUDA Toolkit install. What it absolutely needs is a recent NVIDIA driver — version 452.39 or newer on Windows, and the matching 525+ branch on Linux.
Update in this order and reboot between steps:
- Install the latest Game Ready or Studio driver from NVIDIA’s site, or
sudo apt install nvidia-driver-535(or newer) on Ubuntu. - Upgrade Ollama itself: rerun the install script with
curl -fsSL https://ollama.com/install.sh | shon Linux, or download the latest installer on Windows and macOS. - Restart the Ollama service so it picks up the new driver:
sudo systemctl restart ollama.
Still on CPU? Check the server log: journalctl -u ollama -n 100 on Linux, or %LOCALAPPDATA%Ollamaserver.log on Windows. Lines mentioning “no compatible GPUs” pinpoint the failing layer.
On Linux, one frequent cause is the Ollama service user lacking access to the GPU device files. Adding the ollama user to the video and render groups, then restarting the service, resolves it in most distributions.
Step 3: Fix VRAM Overflow and Partial GPU Offload
When a model plus its context does not fit in VRAM, Ollama splits layers between GPU and CPU — that is the “48%/52%” split in ollama ps — or crashes outright with an out-of-memory error. Three levers fix this, in order of least quality loss.
Lower the context window
Long contexts multiply KV-cache memory. Ollama 0.3 defaults to 4096 tokens for many models, but larger values explode VRAM use. Set it explicitly:
ollama run llama3.1
>>> /set parameter num_ctx 2048For persistent settings, put the parameter in a Modelfile or export it via Ollama environment variables like OLLAMA_NUM_CTX before starting the server.
Use a smaller quantization
Quantization trades a little accuracy for big memory savings. If Q8 does not fit, drop to Q5 or Q4:
ollama run llama3.1:8b-instruct-q4_K_MApproximate VRAM needs for popular 7–8B models:
| Quantization | VRAM (8B model, 4k ctx) |
|---|---|
| Q8_0 | ~9.5 GB |
| Q6_K | ~7.5 GB |
| Q5_K_M | ~6.5 GB |
| Q4_K_M | ~5.5 GB |
Limit GPU layers manually
If you want control over the split, cap how many transformer layers go to the GPU with num_gpu. Fewer layers on the card means less VRAM, with the rest computed on CPU:
>>> /set parameter num_gpu 20If disk space is tight after experimenting with variants, you can remove Ollama models you no longer need with ollama rm.
Step 4: Out-of-Memory Errors on Apple Silicon Macs
Ollama memory usage on a MacBook follows different rules. Apple Silicon uses unified memory, so the GPU and CPU share one pool — and macOS caps how much a single process can claim, typically around 65–75% of total RAM.
Symptoms look like VRAM overflow: Metal errors, a killed process, or the whole machine swapping to a crawl. Fixes, in order:
- Quit memory-hungry apps first — browsers with many tabs, Docker, and IDEs routinely eat 10+ GB that the GPU pool cannot use.
- Drop to a smaller model or quant: a Q4 7B model runs comfortably on a 16 GB MacBook; 13B Q4 needs headroom on 24 GB+.
- Lower
num_ctxexactly as on NVIDIA systems — the KV cache counts against the same unified pool.
Check Activity Monitor’s memory pressure during inference; if the graph runs yellow or red, step down to a smaller model.
Step 5: Decoding the “500 Internal Server Error”
The Ollama 500 internal server error is a catch-all, but three causes cover nearly every case.
The server is not running. The ollama CLI talks to a background service on port 11434. If it is down, start it with ollama serve or relaunch the desktop app. Test the endpoint directly:
curl http://localhost:11434/api/versionPort conflict. Another process holding 11434 breaks the API silently. Find it with netstat -ano | findstr 11434 on Windows or lsof -i :11434 on macOS/Linux, then kill the offender or set OLLAMA_HOST to a different port.
Corrupted model pull. An interrupted download leaves broken blobs that fail at load time with a 500. Re-pull the model — Ollama resumes and verifies checksums:
ollama rm llama3.1
ollama pull llama3.1The Intel GPU Reality Check
If you are hoping Ollama will use an Intel GPU — integrated Arc graphics or an Arc discrete card — temper expectations. Ollama has no official Intel GPU support; its acceleration paths are CUDA (NVIDIA), ROCm (AMD), and Metal (Apple).
Unofficial Vulkan and SYCL builds (such as IPEX-LLM) exist but lag upstream releases. For most Intel-only machines, honest CPU inference with a small Q4 model beats fighting experimental builds.
To make CPU mode tolerable, set num_thread to your physical core count and stick to 3B–8B models at Q4_K_M. Expect 5–15 tokens per second — usable for chat, not for batch work.
Bottom Line
When Ollama is not using your GPU, work the checklist in order: confirm the driver with nvidia-smi, confirm the offload with ollama ps, then shrink num_ctx or quantization until the model fits VRAM. On Apple Silicon, manage unified memory pressure and pick models sized for your RAM.
Most GPU and memory errors are configuration problems, not hardware failures. Ten minutes with the diagnostics above resolves the large majority of them.