Back to tutorials
Tutorial

Run Ollama on a Dedicated Server With AMD GPUs

Run Ollama on a dedicated server with AMD GPUs: drivers, systemd, and Nginx TLS for secure remote inference.

By Anurag Singh
Updated on Nov 04, 2026
Category: Tutorial
Share article
Run Ollama on a Dedicated Server With AMD GPUs

You don’t choose bare metal for comfort. You choose it for predictable performance, stable GPU access, and fewer hidden variables.

This tutorial shows you how to run Ollama on a dedicated server with AMD GPUs, then publish it safely behind Nginx with TLS.

You’ll end up with an Ollama endpoint your apps (or a chat UI) can call without SSH port-forwarding.

You’ll also add guardrails to avoid the classic mistake: shipping an open LLM API to the public internet.

What you’ll build (and what you need)

  • Dedicated server with an AMD GPU supported by ROCm (common Radeon Instinct / MI-series; some Radeon PRO cards also work). Consumer Radeon support varies by kernel/ROCm combo—plan for testing.
  • OS: Ubuntu Server LTS (recommended for ROCm stability). Commands below assume Ubuntu.
  • CPU/RAM: 8+ vCPU and 32+ GB RAM is a good baseline. More helps for concurrent requests and larger models.
  • Disk: 100+ GB free if you plan to store multiple models in /var/lib/ollama.
  • Networking: a domain name pointing to your server for TLS (HostMyCode domains work fine).

Honest limitations: AMD GPU inference depends on ROCm support for your exact GPU, kernel, and ROCm version.

If the stack doesn’t line up, Ollama may fall back to CPU inference.

That’s fine for light testing, but it’s slow and expensive at real throughput.

Provision a dedicated server and prep the OS

For consistent GPU inference, bare metal avoids “noisy neighbor” variability.

It also lets you control BIOS, kernel, and the driver stack end to end.

If you don’t have hardware yet, start with a dedicated servers plan sized for your GPU and bandwidth needs.

After provisioning, update packages and install the basics you’ll need for the rest of this guide:

sudo apt update
sudo apt -y upgrade
sudo apt -y install ca-certificates curl gnupg lsb-release ufw nginx

Set your hostname (optional). It makes logs and monitoring less confusing:

sudo hostnamectl set-hostname ollama-amd-01

Check that Linux sees your AMD GPU

Before you touch ROCm, confirm the GPU shows up on the PCI bus:

lspci | grep -Ei 'vga|3d|display'

You should see an AMD device listed.

If you don’t, start with the basics. Check BIOS settings (PCIe slot enabled, Above 4G decoding on some boards) and physical power connectors.

Next, verify the kernel module is loaded (usually amdgpu):

lsmod | grep amdgpu || true
dmesg | grep -i amdgpu | tail -n 30

Install ROCm userspace (AMD compute stack)

On AMD, Ollama typically depends on ROCm libraries.

The “right” ROCm version depends on your GPU and Ubuntu release. Treat AMD’s official ROCm docs as the source of truth.

The setup path usually looks like this:

  1. Add ROCm repository keys and repo.
  2. Install ROCm runtime packages.
  3. Add your user to the right groups and reboot.

Install prerequisites:

sudo apt -y install libnuma1 pciutils

Then install ROCm packages. The exact package names vary by ROCm generation.

Many systems use a meta-package like rocm. If you have the option, a runtime-only install often keeps the footprint smaller.

After ROCm is installed, grant your user access to the GPU devices and reboot:

sudo usermod -aG video,render $USER
sudo reboot

Once the server comes back, confirm ROCm can see the hardware.

Depending on what your install includes, one of these should return useful output:

/opt/rocm/bin/rocminfo | head -n 50
/opt/rocm/bin/hipconfig --full | head -n 50

If both commands fail, assume one of two problems.

Either ROCm didn’t install cleanly, or your GPU/kernel/ROCm combination isn’t supported.

Install Ollama and run a first model

Install Ollama with the official script.

On systemd-based distros, this typically installs the binaries and creates an ollama user and service.

curl -fsSL https://ollama.com/install.sh | sh

Check that the service is running:

sudo systemctl status ollama --no-pager

Pull a small model first.

You want a quick end-to-end test before you commit to bigger downloads:

ollama pull llama3
ollama run llama3

Send a prompt and confirm you get a response.

If it feels painfully slow, you’re likely running CPU-only.

That’s still usable for light workloads, but it changes your capacity planning immediately.

Configure Ollama as a stable systemd service

Many installations already ship a service unit.

What’s often missing is a single place to set the bind address, model path, and service behavior.

Use a systemd drop-in override so upgrades won’t stomp your changes.

Create the override:

sudo systemctl edit ollama

Paste the following (adjust paths and settings to match your environment):

[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_MODELS=/var/lib/ollama"
# Optional: keep logs readable
Environment="OLLAMA_LOG_LEVEL=info"

# Safety limits. Tune if you serve large models.
LimitNOFILE=1048576
TimeoutStartSec=180
Restart=always
RestartSec=3

Reload systemd and restart Ollama:

sudo systemctl daemon-reload
sudo systemctl restart ollama
sudo systemctl status ollama --no-pager

Make sure it’s only listening on localhost. That’s intentional—Nginx will handle public traffic:

ss -ltnp | grep 11434 || true

Put Ollama behind Nginx with TLS (safe remote access)

Binding Ollama to 0.0.0.0:11434 is an easy way to invite abuse (and surprise bills).

A reverse proxy gives you TLS termination, a place to add auth, and a clean boundary.

The backend stays on localhost.

If you want a companion walkthrough for the same pattern, see: Run Ollama Behind Nginx With Basic Auth.

1) Create a DNS record

Create an A record like ollama.example.com pointing to your dedicated server IP.

If you need domain registration or DNS hosting, use domains.

2) Allow firewall ports (SSH + HTTPS only)

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status verbose

3) Get a TLS certificate

Use Certbot with Nginx. Install it and request a certificate:

sudo apt -y install certbot python3-certbot-nginx
sudo certbot --nginx -d ollama.example.com

If you prefer Caddy for automatic TLS, HostMyCode covers the approach here: Set Up Let’s Encrypt With Caddy for a VPS Site.

This guide sticks with Nginx since it’s common on dedicated servers.

4) Add Nginx reverse proxy config with basic auth

Create an htpasswd file (keep it outside web roots):

sudo apt -y install apache2-utils
sudo htpasswd -c /etc/nginx/.htpasswd ollamauser

Create an Nginx server block:

sudo nano /etc/nginx/sites-available/ollama

Use this config (replace the domain):

server {
    listen 80;
    server_name ollama.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name ollama.example.com;

    ssl_certificate /etc/letsencrypt/live/ollama.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ollama.example.com/privkey.pem;

    # Keep requests reasonable; bump if you send large prompts.
    client_max_body_size 10m;

    location / {
        auth_basic "Ollama";
        auth_basic_user_file /etc/nginx/.htpasswd;

        proxy_pass http://127.0.0.1:11434;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Streaming responses
        proxy_buffering off;
        proxy_read_timeout 300s;
    }
}

Enable the site and reload Nginx:

sudo ln -s /etc/nginx/sites-available/ollama /etc/nginx/sites-enabled/ollama
sudo nginx -t
sudo systemctl reload nginx

If you run into upload errors later (large context windows, file-based prompts), see: Nginx 413 Request Entity Too Large Fix On VPS.

5) Test the endpoint

From your laptop (or another server), run:

curl -u ollamauser 'https://ollama.example.com/api/tags'

You should get JSON back with the models you’ve pulled.

Now test a generation request:

curl -u ollamauser https://ollama.example.com/api/generate \
  -H 'Content-Type: application/json' \
  -d '{"model":"llama3","prompt":"Write a 3-line nginx troubleshooting checklist."}'

GPU sizing and model expectations on AMD

Most sizing decisions come down to two constraints: VRAM and concurrency.

VRAM determines which model sizes fit comfortably.

Concurrency determines whether users get quick responses or wait in a queue.

  • Small models (for tools, routing, short answers): tend to fit on modest VRAM with quantization. Great for internal automation.
  • Medium models (general chat quality): need more VRAM, especially if you want larger context windows.
  • Large models: often require high VRAM GPUs or multi-GPU sharding. Don’t assume a single card will carry you.

Practical advice: get one model stable first.

Measure VRAM headroom and latency under real prompts, then add models slowly.

Model sprawl is how “works fine” turns into random OOM failures.

Quantization trade-offs (what to pick first)

Quantization cuts memory use and can improve throughput.

Quality can drop depending on the model and bit-rate.

In self-hosted setups, quantization is often the difference between “fits and serves” and “thrashes memory.”

  • Lower-bit quantization: less VRAM, faster loads, sometimes worse reasoning and more repetition.
  • Higher-bit quantization / full precision: better quality, more VRAM, fewer concurrent sessions.

Pick one quantized variant that fits with margin.

Keep 10–20% free VRAM so spikes don’t knock the service over.

Hardening checklist for a public AI endpoint

Once inference is reachable over HTTPS, treat it like any production API.

The common failure modes aren’t mysterious. Expect brute forcing, scraping, leaked credentials, and cost blowups.

  • Bind Ollama to localhost (127.0.0.1) and expose only via Nginx.
  • Use TLS and disable plain HTTP.
  • Add authentication (basic auth is better than nothing; API keys are better for apps).
  • Rate-limit by IP if the endpoint is internet-accessible.
  • Log requests and watch for unusual spikes.

If you want a tighter perimeter (no public ports), use a tunnel-based approach for private access.

For reference patterns, see: Secure Ollama API With Cloudflare Tunnel on VPS.

Keep the base OS patched, too. AI boxes are attractive targets.

You don’t want to babysit security updates:

sudo apt -y install unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

HostMyCode has a step-by-step here: Set Up Automated Security Updates on Ubuntu VPS.

Optional: connect a chat UI later (without changing your API)

Once your HTTPS endpoint is solid, you can attach a web UI on the same machine or a separate server.

Many teams keep the model host isolated and treat the UI as a client.

If you want a full-featured interface, Open WebUI is a common choice. This HostMyCode guide walks through deployment: Self-Host Open WebUI With Ollama on a VPS.

The same approach works on dedicated servers.

Troubleshooting: the failures you’ll actually see

Ollama is slow (likely CPU fallback)

  • Confirm the GPU is visible to ROCm tools (for example rocminfo).
  • Watch htop during inference. If CPU pins and GPU stays idle, you’re not using the GPU path.
  • Check system logs: journalctl -u ollama -n 200 --no-pager

Nginx returns 502

  • Verify Ollama is running: sudo systemctl status ollama
  • Check the local port: curl http://127.0.0.1:11434/api/tags
  • Review Nginx error log: sudo tail -n 100 /var/log/nginx/error.log

If you want a dedicated 502 workflow, use: Nginx 502 Bad Gateway Fix on Ubuntu VPS.

Certbot fails to issue a certificate

  • Confirm DNS resolves to your server: dig +short ollama.example.com
  • Make sure ports 80/443 are reachable (no upstream firewall blocks).
  • Check Nginx config test: sudo nginx -t

Summary: a production-friendly AMD GPU Ollama host

You now have Ollama running as a systemd service on a dedicated machine.

It’s bound to localhost and reachable over HTTPS through Nginx.

This setup is boring in the best way. That’s what you want for assistants, app features, and automation.

If you plan to operate this long-term, budget time for two ongoing jobs: ROCm/driver compatibility testing, and API hardening (auth, rate limits, and logs).

Those are the difference between a stable service and a weekly fire drill.

For operators who want predictable performance and full control of GPU inference, HostMyCode dedicated servers are a clean fit.

If you’d rather offload some day-to-day infrastructure and security work, consider managed VPS hosting for the surrounding stack while you focus on the AI layer.

If you want stable inference without shared resources, run Ollama on a dedicated server (bare metal) and keep the endpoint behind TLS from day one. HostMyCode offers dedicated servers for consistent GPU performance, plus migrations help when you’re moving an existing app stack to new infrastructure.

FAQ: Run Ollama on a Dedicated Server

Can I expose Ollama directly on port 11434?

You can, but you shouldn’t. Keep it on 127.0.0.1 and expose it through Nginx with TLS and authentication. Public LLM endpoints attract abuse fast.

Do I need a dedicated server, or is a VPS enough?

For AMD GPUs, dedicated servers are often simpler because you control the entire driver and kernel stack. A GPU VPS can work, but bare metal usually means fewer constraints.

What’s the minimum RAM and disk for a practical setup?

Start with 32 GB RAM and 100 GB free disk for a single model plus logs and headroom. If you store multiple models, plan for 200–500 GB.

How do I add a web chat UI later?

Once the API works over HTTPS, you can attach a UI like Open WebUI. Keep the model server isolated and connect the UI to your secured endpoint.

What’s the easiest way to prevent cost spikes?

Use authentication, add Nginx rate limits, and monitor request volume. If it must be private, put the endpoint behind a tunnel or a VPN rather than exposing it publicly.