How to Run DeepSeek Locally: VRAM, Quantization & Setup

Learning how to run DeepSeek locally has become one of the most popular projects among enthusiasts of open-source AI models. With the right hardware and a few free tools, you can chat with DeepSeek-R1 reasoning models without an internet connection or API bills.

The catch is that “DeepSeek” covers a wide range of models, from tiny 1.5B distilled variants to the massive 671B-parameter R1. Most home setups can only handle the smaller distills, so picking the right variant is the first real decision.

This guide walks you through hardware requirements, quantization choices, and a complete Ollama-based setup, plus an honest look at when the official API makes more sense than local inference.

Which DeepSeek Models Can You Actually Run Locally?

The full DeepSeek-V3 and DeepSeek-R1 models use a Mixture-of-Experts architecture with 671B total parameters. Even heavily quantized, they demand hundreds of gigabytes of memory, which puts them firmly in server-cluster territory.

What individuals run at home are the distilled models. DeepSeek released R1-distilled versions of Qwen and Llama in sizes from 1.5B up to 70B, and these retain much of the reasoning behavior in a dense, locally-friendly package.

As a rough guide, the 7B–14B distills hit the sweet spot for quality versus hardware cost. The 1.5B model runs almost anywhere but reasons weakly, while the 70B distill needs a high-end GPU or dual-GPU setup.

VRAM Requirements at Q4 Quantization

The table below shows approximate VRAM needs for each R1 distill at Q4_K_M quantization, which is the default for most Ollama and llama.cpp builds. Add roughly 1–2 GB for context and system overhead.

These numbers are approximations and vary with context length and KV-cache settings. If a model doesn’t fit in VRAM, layers spill to system RAM and speeds drop sharply.

If you run into out-of-memory crashes or models refusing to load, our guide on how to fix Ollama GPU and memory errors covers the most common fixes.

Quantization Options Explained

GGUF for Ollama and llama.cpp

GGUF is the standard format for llama.cpp and Ollama. Quants like Q4_K_M offer the best balance of size and quality, while Q6_K or Q8_0 cost more VRAM for a small quality gain.

Below Q4, quality degrades noticeably on reasoning models. Most users should stick to Q4_K_M or Q5_K_M.

EXL2 for TabbyAPI

EXL2 is a GPU-only format used by ExLlamaV2-based servers such as TabbyAPI. It supports flexible bitrates (like 4.25 bpw) and tends to be faster than GGUF on pure-GPU inference.

The tradeoff is complexity: TabbyAPI requires manual model downloads and config files. For a first local setup, GGUF through Ollama is far simpler.

Step-by-Step: Running DeepSeek-R1 with Ollama

Step 1: Install Ollama

Download Ollama from the official site for Windows or macOS, or use the install script on Linux:

curl -fsSL https://ollama.com/install.sh | sh

Verify the installation and that your GPU is detected:

ollama --version
nvidia-smi

Step 2: Pull a DeepSeek-R1 Variant

Ollama hosts the official distill tags. Pick one that fits your VRAM from the table above:

ollama pull deepseek-r1:7b

Other available tags include 1.5b, 8b, 14b, 32b, and 70b. For example:

ollama pull deepseek-r1:14b

Step 3: Run the Model

ollama run deepseek-r1:7b

You’ll get an interactive chat prompt. R1 distills show their chain-of-thought in <think> tags before the final answer, which is normal behavior.

To check whether the model loaded fully on the GPU, run this in a second terminal:

ollama ps

The output shows processor split, e.g. “100% GPU”. Anything less means some layers are running on your CPU.

Step 4: Tune GPU Layer Offloading

If the model is slightly too large for your VRAM, you can offload only part of the layers to the GPU. Set the number of GPU layers with an environment variable before starting Ollama:

OLLAMA_NUM_GPU_LAYERS=28 ollama run deepseek-r1:14b

Lower numbers free VRAM at the cost of speed. Experiment until ollama ps reports a stable split without memory errors.

Expected Performance by Hardware Class

Generation speed depends heavily on GPU memory bandwidth. The figures below are rough, real-world ranges for Q4 GGUF inference with a moderate context.

Reasoning models generate long chains of thought, so a slow token rate hurts more than with chat models. Anything under ~10 tokens/sec feels painful for R1-style output.

When to Use the API Instead

Local distills are impressive, but they are not the same as the real DeepSeek-R1. The 671B model is significantly stronger on hard math, coding, and multi-step reasoning — we compare the full model against Meta’s flagship in our DeepSeek V3 vs Llama 3 breakdown.

Use the official API (or a hosted provider) when any of these apply:

• You need maximum reasoning quality for production work or research.
• Your GPU has less than 12 GB of VRAM and the 7B distill isn’t good enough.
• You want long contexts above roughly 32K tokens, which eats VRAM locally.
• Cost per query is low — DeepSeek’s API pricing is cheap enough that light usage costs less than the electricity for a big GPU.

Run locally when privacy matters, you want offline access, or you’re experimenting with prompts, fine-tunes, or tooling around the model.

Bottom Line

Running DeepSeek locally is straightforward: install Ollama, pull a distilled R1 variant that fits your VRAM, and verify GPU offloading with ollama ps. The 7B and 14B distills offer the best balance for typical consumer GPUs.

Just keep expectations realistic — the full 671B R1 stays in the cloud for a reason. Treat local distills as a private, free sandbox, and reach for the API when you need the real thing’s full reasoning power.