Ollama is slow or running on CPU instead of GPU: model placement, context size and reloads

#Symptom

One or more of these on a host running Ollama:

• Generation is a few tokens per second when it used to be fast.
• nvidia-smi shows little or no GPU memory used by the ollama process.
• The first request after a quiet period takes 10–60 seconds, then later requests are fast.
• Requests queue behind each other or get rejected when several users hit the server.

Most of these come down to one question: where is the model loaded, and does it stay there?

#Quick checks

#1. Where is the model loaded?

ollama ps

The PROCESSOR column tells you where the model's weights went:

If ollama ps shows nothing, no model is loaded, so the next request pays the load time.

#2. Is the GPU visible to the host and the service?

nvidia-smi
journalctl -u ollama --no-pager --since "30 min ago" | grep -iE "gpu|cuda|vram|library|offload"

For Docker:

docker run --rm --gpus all ubuntu nvidia-smi   # does the container runtime see the GPU?
docker logs ollama 2>&1 | grep -iE "gpu|cuda|vram|offload"

At startup Ollama logs which GPUs it discovered and how much VRAM is available. When it loads a model it logs how many layers it offloaded to the GPU. The exact wording varies by version, so search for offload and layers.

#3. What context size is the model using?

ollama show llama3.1:8b          # architecture, parameters, max context length of the model
systemctl show ollama --property=Environment

The KV cache grows with context size and with the number of parallel requests. A model that fits at 4K context can spill onto the CPU at 32K.

#Causes and fixes

#Fix 1: GPU not discovered

Ollama falls back to CPU silently if GPU discovery fails. A known case is suspend/resume on Linux, where the NVIDIA UVM module stops working:

sudo systemctl stop ollama
sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm
sudo systemctl start ollama
ollama run llama3.1:8b "hi" >/dev/null && ollama ps

If nvidia-smi itself fails, fix the driver first (see the NVIDIA driver and CUDA mismatch post). Check that the card's compute capability and driver version meet the minimum in Ollama's GPU documentation; older cards are dropped over time.

For more detail, turn on debug logging temporarily:

sudo systemctl edit ollama.service
# add under [Service]:
#   Environment="OLLAMA_DEBUG=1"
sudo systemctl daemon-reload && sudo systemctl restart ollama
journalctl -u ollama -f

Remove the debug setting once you are done; it is noisy.

Tip

On a multi-GPU host, CUDA_VISIBLE_DEVICES controls which GPUs Ollama can use. Use GPU UUIDs from nvidia-smi -L rather than indexes, because indexes can change. An invalid value such as -1 forces CPU, which is a common copy-paste mistake.

#Fix 2: model does not fit in VRAM

Total VRAM needed is roughly weights + KV cache (context × parallel slots) + overhead. Reduce whichever is largest:

• Smaller quantization or model. Pull a smaller tag, for example a 4-bit quantized variant instead of 8-bit. Tag names differ per model; check the model page in the Ollama library.
• Smaller context. The server default comes from OLLAMA_CONTEXT_LENGTH (the default value has changed between releases; check your version's docs). Override per request with num_ctx:

curl -s http://127.0.0.1:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Summarise: disk full on /var",
  "options": { "num_ctx": 8192 },
  "stream": false
}' | jq -r .response

• Fewer parallel slots. OLLAMA_NUM_PARALLEL multiplies the KV cache. Lower it if the model only just fits.
• KV cache settings. OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 reduce KV cache memory on supported models. A quantized KV cache can slightly change output quality, so test before using it in production.

Set server-wide values in a systemd override:

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_NUM_PARALLEL=2"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KEEP_ALIVE=30m"

sudo systemctl daemon-reload && sudo systemctl restart ollama

Then load the model again and confirm ollama ps shows 100% GPU.

Warning

Lowering num_ctx too far cuts off long prompts silently: the oldest part of the prompt is dropped and answers get worse without an error. If you run RAG, make sure context size is larger than your system prompt plus retrieved chunks plus expected output.

#Fix 3: cold start after idle

By default Ollama unloads a model after a few minutes with no requests (5 minutes in current releases). The next request reloads the weights from disk.

• Set OLLAMA_KEEP_ALIVE to a longer duration, for example 30m or 24h, or -1 to keep it loaded indefinitely.
• Per request, pass "keep_alive": "1h" in the API body. This overrides the server setting.
• Preload after a restart, for example from a systemd ExecStartPost or a deploy step:

curl -s http://127.0.0.1:11434/api/generate -d '{"model":"llama3.1:8b","keep_alive":-1}'

To unload a model manually, for example to free VRAM for another job:

ollama stop llama3.1:8b

If model loads are slow even when warm-up is expected, check the disk holding OLLAMA_MODELS. Network file shares and throttled cloud disks make every load slow.

#Fix 4: several models fighting for VRAM

If apps use different models, Ollama loads each one and evicts others when memory runs out. Each switch pays a full reload. Watch it happen:

watch -n 2 ollama ps

Standardise apps on fewer models, or put the models on separate GPUs or hosts. OLLAMA_MAX_LOADED_MODELS caps how many can be loaded at once; it does not make more fit.

#Verify

ollama ps                                   # 100% GPU, UNTIL shows your keep-alive
nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu --format=csv
time ollama run llama3.1:8b "Write one sentence about disk alerts." >/dev/null

Run the timing twice. The second run shows warm performance; the first includes load time if the model was not loaded.

#Prevent it next time

• Add a health check that calls /api/ps and alerts when the expected model reports any CPU share, not only when the service is down.
• Pin the NVIDIA driver and kernel versions on Ollama hosts, and restart Ollama after any driver update.
• Disable suspend on servers and VMs that run Ollama.
• Keep server settings (OLLAMA_CONTEXT_LENGTH, OLLAMA_NUM_PARALLEL, OLLAMA_KEEP_ALIVE) in a systemd override managed by config management, not set by hand.
• Size VRAM for the largest context and parallelism you actually allow, with headroom. Retest after changing models or tags.
• If you need many concurrent users, high throughput or multi-GPU serving, consider a dedicated serving stack such as vLLM instead of stretching a single Ollama host.