Ollama will run on CPU even if your VPS has an NVIDIA GPU sitting there unused. The symptoms are familiar: slow token generation, hot CPUs, and an idle nvidia-smi.
This guide shows you how to run Ollama with a GPU on an Ubuntu VPS. You’ll install a sane driver stack, validate CUDA visibility end to end, and fix common real-world snags.
By the end, you’ll have Ollama managed by systemd and a fast verification loop. You’ll also add a few guardrails to avoid exposing your model API to the internet.
What you need (and what usually breaks)
This assumes an Ubuntu server you control (20.04+ works; 22.04/24.04 are typical in 2026) with an attached NVIDIA GPU. A single-tenant GPU VPS or a dedicated server both work fine.
- Root or sudo access
- At least 25–40 GB free disk (models add up quickly)
- RAM: 16 GB minimum for small models; 32 GB feels comfortable
- VRAM: 8 GB can work for small/quantized models; 16–24 GB removes a lot of friction
Why GPU acceleration often doesn’t kick in:
- Wrong NVIDIA driver (or installed with a conflicting method)
- Secure Boot blocks the kernel module (rare on VPS, common on bare metal)
- Ollama runs as a service user that can’t access
/dev/nvidia* - Driver/runtime mismatch (especially after upgrades)
- GPU exists on the host, but isn’t passed into a container
If you don’t already have infrastructure, start with a GPU-capable HostMyCode VPS.
If you want the “don’t page me at 2am” version (updates, security, and service reliability), managed VPS hosting is the practical option.
Step 1: Confirm the GPU is actually attached
Before you touch drivers or services, confirm the kernel can see the hardware.
lspci | grep -i nvidia || true
If an NVIDIA device shows up, keep going. If it doesn’t, your VPS likely isn’t on a GPU node or passthrough isn’t enabled.
Fix that first. Software won’t fake missing hardware.
Step 2: Install the NVIDIA driver (Ubuntu repo method)
For production servers, take the boring path. Install the driver from Ubuntu packages, not random runfiles.
sudo apt update
sudo apt -y install ubuntu-drivers-common
ubuntu-drivers devices
Install the driver Ubuntu marks as recommended. The version below is a placeholder. Use the one your system reports.
sudo apt -y install nvidia-driver-XXX
sudo reboot
After reboot, confirm the driver is alive:
nvidia-smi
You want to see your GPU, a driver version, and a process table (even if it’s empty).
If nvidia-smi fails:
- “NVIDIA-SMI has failed because it couldn't communicate…” usually means the kernel module didn’t load. Check:
lsmod | grep nvidiaanddmesg | tail -n 80. - Secure Boot can block the module on some systems. On bare metal, disable Secure Boot or enroll MOK keys. Most GPU VPS setups don’t involve Secure Boot.
Step 3: Add basic CUDA userland (minimal, not a full toolkit)
Ollama doesn’t need the full CUDA developer toolkit. You do want a few basics for sanity checks and troubleshooting.
sudo apt -y install pciutils build-essential
If your provider documents specific CUDA runtime packages for their image, follow that guidance. The driver is mandatory.
Extra CUDA components are optional unless you have other GPU workloads that depend on them.
Step 4: Install Ollama on Ubuntu
Install Ollama using the official Linux install script. On most systems, it also sets up a service and a dedicated user.
curl -fsSL https://ollama.com/install.sh | sh
Confirm it’s installed:
command -v ollama
ollama --version || true
If you prefer Docker for isolation or upgrades, that can work. Just don’t forget the one thing Docker won’t do automatically: GPU passthrough.
This tutorial stays on the host with systemd. It’s usually the most reliable way to keep CUDA accessible on a VPS.
Step 5: Make sure the Ollama service user can access the GPU
Many installs run Ollama as a dedicated user (often ollama). If that user can’t access the NVIDIA device nodes, Ollama can quietly fall back to CPU.
Start by checking device nodes and relevant groups:
ls -l /dev/nvidia*
getent group video || true
getent group render || true
Then confirm which user systemd is actually running:
systemctl status ollama --no-pager
If the service runs as ollama, add it to the common GPU groups. Then restart:
sudo usermod -aG video,render ollama
sudo systemctl restart ollama
Check service health and recent logs:
systemctl is-active ollama
journalctl -u ollama -n 100 --no-pager
Step 6: Pull a small model and verify GPU usage
Start small. This helps you separate “GPU not detected” from “model doesn’t fit in VRAM.”
ollama pull llama3
Run a prompt in one terminal:
ollama run llama3
In a second terminal, watch the GPU while it generates:
watch -n 1 nvidia-smi
During generation, you should see at least one of the following:
- Non-trivial GPU-Util
- Increasing Memory-Usage (VRAM allocation)
- An Ollama-related process listed under running processes
If VRAM stays near zero while the CPU spikes, you’re still on CPU. Skip ahead to troubleshooting.
Step 7: Set Ollama to listen safely (don’t expose it raw)
A lot of people bind Ollama broadly so they can reach it from another machine. That’s also how endpoints get scraped and abused.
Treat the Ollama API like a database port: private by default.
Check what address it’s listening on:
ss -lntp | grep -E ':11434\b' || true
If you need remote access, put it behind a reverse proxy with TLS and authentication. HostMyCode has a practical walkthrough here: Run Ollama Behind Nginx With Basic Auth.
If you’d rather not open inbound ports at all, use a tunnel. For Cloudflare-based setups, follow: Secure Ollama API With Cloudflare Tunnel on VPS.
Step 8: Quick performance sizing (CPU vs GPU, RAM vs VRAM)
GPU acceleration helps most during token generation and attention-heavy steps. It won’t help if you’re short on RAM.
It also won’t help if VRAM can’t hold the active layers you want on GPU.
- 8 GB VRAM: stick to smaller models and stronger quantization. Expect trade-offs in output quality and context length.
- 16 GB VRAM: a comfortable baseline for many mid-size instruct models with reasonable quantization.
- 24 GB+ VRAM: room for larger models, higher context, and less aggressive quantization.
Quantization is the lever you’ll pull most often. Lower-bit variants reduce VRAM/RAM pressure, but they can change output quality and tool-use reliability.
If you’re choosing GGUF variants, keep this handy: llama.cpp Quantization: Pick Q4 vs Q5 vs Q8.
Troubleshooting: Ollama still uses CPU
This is where most “it’s installed” setups turn into “it’s actually working.” Go in order, and don’t skip steps.
1) Driver OK, but service user can’t see the GPU
Symptom: nvidia-smi works as root/your user, but Ollama stays on CPU.
ps aux | grep -E 'ollama(\s|$)'
ls -l /dev/nvidia*
Fix: add the service user to video and render, then restart Ollama:
sudo usermod -aG video,render ollama
sudo systemctl restart ollama
2) Conflicting driver install methods
Symptom: the driver half-works, but upgrades break it. You may also see DKMS errors.
Check installed NVIDIA packages:
dpkg -l | grep -E 'nvidia-(driver|dkms|kernel|utils)' | head
If you previously installed via a manual .run installer, remove it using that same method’s uninstaller.
Then reinstall from Ubuntu packages.
Mixing installation methods is a reliable way to lose the kernel module after the next reboot.
3) Not enough VRAM for the model you picked
Symptom: inference is painfully slow, the model fails to load, or it offloads to CPU without making it obvious.
Try a smaller model or a more aggressively quantized variant. Then check memory pressure:
free -h
nvidia-smi
If system RAM is tight, the OOM killer may terminate the process. Look for it:
journalctl -k -n 200 --no-pager | grep -i -E 'oom|killed process' || true
4) You’re running inside Docker without GPU passthrough
Symptom: the host sees the GPU, but the container does not.
Inside the container, nvidia-smi won’t work. The fix is running with the NVIDIA container runtime and the correct flags.
If you’re not committed to Docker, running Ollama directly on the VPS with systemd (as shown here) is usually simpler to operate.
5) Kernel updates broke the module
Symptom: everything worked until an update, then nvidia-smi fails.
Reinstall the driver package for your branch and reboot.
Also confirm DKMS built the module:
sudo dkms status | head -n 20
uname -r
On production nodes, turn on automatic security updates carefully. Plan reboots instead of letting them surprise you.
For a baseline approach, see: Set Up Automated Security Updates on Ubuntu VPS.
Optional: Put the Ollama API behind Nginx with rate limits
If you’re connecting a chat UI, a coding assistant, or internal tools, you’ll want a stable URL, TLS, and access control.
Nginx is a good fit on a VPS because it behaves predictably and it’s easy to audit.
At minimum, use Basic Auth for internal access, or an auth gateway for teams. Use the HostMyCode tutorial for a working Nginx config with credentials: Run Ollama Behind Nginx With Basic Auth.
If you’re building a full app stack (chat + tools + RAG), you’ll often route multiple upstreams.
Keep your reverse proxy config tidy as the stack grows.
A Caddy-based approach can also work well with Docker stacks; see: Set Up Caddy Reverse Proxy for Docker on a VPS.
Operational checklist (what to verify after every change)
nvidia-smiworks after rebootsystemctl is-active ollamareturnsactive- During inference,
nvidia-smishows VRAM usage increasing - Port
11434is not exposed publicly unless protected - Disk has headroom for models (
df -h)
Summary
To run Ollama with a GPU on an Ubuntu VPS, three things must line up: a working NVIDIA driver (nvidia-smi), correct permissions for the Ollama service user, and a model that fits your VRAM budget.
After that, it’s routine ops. Keep the service stable, limit network exposure, and re-check GPU usage after updates.
If you want a reliable place to run this in production, start with a GPU-ready HostMyCode VPS.
If you don’t want to babysit patching and service health, managed VPS hosting is a cleaner fit for long-running AI endpoints.
Running local AI is easier when the underlying server behaves predictably. HostMyCode’s VPS hosting gives you full control over NVIDIA drivers, systemd services, and reverse proxies. If you want help keeping it patched and stable, use managed VPS hosting and operate your Ollama node like production infrastructure.
FAQ
How do I confirm Ollama is using CUDA and not CPU?
Run a prompt with ollama run <model> and watch nvidia-smi in another terminal. You should see VRAM usage rise and GPU utilization jump during generation.
Is 8 GB VRAM enough to run Ollama with a GPU?
It can be, but you’ll need smaller models and/or heavier quantization. If loads fail or performance swings, drop model size or choose a more quantized build.
Should I expose port 11434 to the internet?
No. Put Ollama behind TLS and authentication (reverse proxy) or use a tunnel. Exposing the raw API invites abuse and surprise bills.
Why does it work in my SSH session but not as a service?
It’s usually permissions. The service user may not be in video/render or can’t access /dev/nvidia*. Add group membership and restart the service.
What should I do before upgrading drivers on a production VPS?
Schedule a maintenance window and make sure you can reach the console if the module fails to load. After reboot, verify nvidia-smi and run a short test prompt while watching VRAM usage.