Run vLLM on a GPU VPS With OpenAI-Compatible API

You don’t need a managed “AI platform” to ship an LLM endpoint. You do need predictable GPU capacity, a repeatable install, and security defaults you won’t regret later.

This walkthrough shows you how to run vLLM on a GPU VPS and expose an OpenAI-compatible /v1 API that your apps can call right away.

The target is Ubuntu Server on real infrastructure (a GPU VPS or a GPU-backed dedicated server).

You’ll install NVIDIA drivers, set up an isolated Python environment, run vLLM under systemd, and put it behind Nginx. You’ll also add TLS and a few basic guardrails.

What you’ll build (and what it won’t do)

Builds: A vLLM server providing an OpenAI-compatible API on https://llm.yourdomain.com/v1 (chat/completions + embeddings depending on the model).
Includes: systemd unit, UFW rules, Nginx reverse proxy, TLS via Let’s Encrypt, optional API key header, and a basic health check.
Doesn’t include: multi-node orchestration, automatic scaling, advanced auth (OAuth), or a full RAG stack. If you want a chat UI, add it later as a separate service.

Prerequisites and sizing: pick hardware that won’t disappoint

vLLM is built for NVIDIA GPUs and high-throughput serving. Yes, it can run on CPU.

In practice, CPU token generation slows to a crawl. The “API” ends up feeling broken.

For most deployments, start with a single GPU. Get the basics stable before you move to multi-GPU or multiple nodes.

VRAM reality check (honest rules of thumb)

VRAM requirements vary by architecture and quantization. These ranges are still useful when you’re choosing a first model:

7B–8B models: practical on 8–12 GB VRAM (quantization often required at the lower end)
13B–14B models: expect 16–24 GB VRAM (quantization helps; full precision often needs more)
30B+ models: typically 40–80 GB+ VRAM or multi-GPU setups

If you’re on the fence, go smaller first. A responsive 8B model beats a larger model that OOMs, swaps, or stalls under load.

Need infrastructure that holds steady under sustained inference? A HostMyCode VPS (or a dedicated server for heavier models) keeps performance and data locality in your hands.

Step 1: Provision the GPU VPS and set DNS

Start with a fresh Ubuntu VPS and SSH access.

Point a subdomain (for example, llm.example.com) at the server’s public IP.

Create an A record: llm → your VPS IPv4.
Wait for DNS to propagate.
Confirm resolution from your laptop:

dig +short llm.example.com

If you’re cutting over from another provider, treat it like a real migration.

HostMyCode lays out a clean process in this DNS cutover tutorial.

Step 2: Update Ubuntu and install baseline packages

Do the basic housekeeping now. It saves time later, especially after you add drivers and Python packages.

sudo apt update
sudo apt -y upgrade
sudo apt -y install curl ca-certificates gnupg lsb-release unzip jq ufw nginx python3-venv

Step 3: Install NVIDIA drivers and validate the GPU

Driver installation depends on the image you started from. Some GPU VPS images already include the right driver.

Either way, treat nvidia-smi as the source of truth.

Option A (common): drivers already present

nvidia-smi

If it prints the GPU model and driver version, you’re good. Move on.

Option B: install drivers from Ubuntu repositories

If nvidia-smi isn’t available, install a recommended driver and reboot:

sudo apt -y install ubuntu-drivers-common
ubuntu-drivers devices
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot:

nvidia-smi

Pitfall: Secure Boot

On some systems, Secure Boot blocks unsigned kernel modules.

If the driver installs but won’t load, check BIOS/firmware settings (provider-dependent).

On many VPS platforms you can’t change this. In that case, you’ll need a Secure Boot-compatible image from the provider.

Step 4: Create a dedicated service user and directories

Don’t run your model server as root. Use a locked-down service account.

Keep files in predictable paths.

sudo useradd -r -m -d /opt/vllm -s /usr/sbin/nologin vllm
sudo mkdir -p /opt/vllm/app /opt/vllm/models /var/log/vllm
sudo chown -R vllm:vllm /opt/vllm /var/log/vllm

Step 5: Install vLLM in a Python virtual environment

Install into a venv under /opt/vllm/app.

That makes upgrades a controlled change, not a system-wide cleanup job.

sudo -u vllm python3 -m venv /opt/vllm/app/venv
sudo -u vllm /opt/vllm/app/venv/bin/pip install --upgrade pip wheel

Install vLLM:

sudo -u vllm /opt/vllm/app/venv/bin/pip install vllm

Choose a model (and a sane first run)

Pick a model that fits your VRAM and still leaves headroom.

For a first deployment, an instruction-tuned 7B–8B model from a trusted publisher is usually the least painful option.

On first start, vLLM will download weights into the vLLM user’s cache.

If your model requires a Hugging Face token, you’ll add it as a systemd environment variable in a later step. If your model is open access, skip that part.

Step 6: Run vLLM once manually to confirm it loads

Before you add systemd and Nginx, prove the model loads and the API starts.

Replace MODEL_ID with your model identifier.

sudo -u vllm -H bash -lc '
  source /opt/vllm/app/venv/bin/activate
  python -m vllm.entrypoints.openai.api_server \
    --host 127.0.0.1 \
    --port 8000 \
    --model MODEL_ID \
    --served-model-name llm
'

In another SSH session, query the local endpoint:

curl -s http://127.0.0.1:8000/v1/models | jq

A JSON response listing a model means the server is alive.

Stop it (Ctrl+C) and continue.

If the model fails to load

CUDA OOM: your VRAM is too small for the model/settings. Use a smaller model, lower max context, reduce batch size, or enable quantization (model-dependent).
Disk fills up: model weights can be tens of GB. Increase disk or clean old caches under the vllm user’s cache directory.
Import errors: keep the venv clean; avoid mixing system Python packages with pip packages.

Step 7: Create a systemd service for vLLM

With systemd, the service starts on boot. It also restarts on failure.

Logs go to the journal.

Create /etc/systemd/system/vllm.service:

sudo nano /etc/systemd/system/vllm.service

Paste this file and edit the placeholders:

[Unit]
Description=vLLM OpenAI-compatible API server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=vllm
Group=vllm
WorkingDirectory=/opt/vllm

# Optional: if your model download needs an HF token
# Environment="HF_TOKEN=YOUR_TOKEN_HERE"

# Keep model cache under the service user
Environment="HF_HOME=/opt/vllm/.cache/huggingface"
Environment="TRANSFORMERS_CACHE=/opt/vllm/.cache/huggingface"

ExecStart=/opt/vllm/app/venv/bin/python -m vllm.entrypoints.openai.api_server \
  --host 127.0.0.1 \
  --port 8000 \
  --model MODEL_ID \
  --served-model-name llm

Restart=on-failure
RestartSec=3

# Basic hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true
ReadWritePaths=/opt/vllm /var/log/vllm

[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now vllm
sudo systemctl status vllm --no-pager

Follow logs while it initializes:

sudo journalctl -u vllm -f

Step 8: Configure UFW so the API isn’t exposed raw

The goal is simple: only Nginx is reachable from the internet.

vLLM stays on localhost. That’s why it listens on 127.0.0.1:8000.

Apply a minimal firewall policy:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw enable
sudo ufw status verbose

If you want a broader baseline and rationale, use this UFW tutorial and tailor it to your environment.

Step 9: Put vLLM behind Nginx with TLS

Nginx terminates TLS and gives you one place to set timeouts, payload limits, and basic access controls.

You’ll proxy /v1 to 127.0.0.1:8000.

Create an Nginx site

sudo nano /etc/nginx/sites-available/llm.conf

Config (replace llm.example.com):

server {
  listen 80;
  server_name llm.example.com;

  location / {
    return 200 "ok\n";
    add_header Content-Type text/plain;
  }
}

Enable and test:

sudo ln -s /etc/nginx/sites-available/llm.conf /etc/nginx/sites-enabled/llm.conf
sudo nginx -t
sudo systemctl reload nginx

Issue a Let’s Encrypt certificate

Install Certbot and request a certificate.

For edge cases and fixes, this SSL setup guide is a solid reference.

sudo apt -y install certbot python3-certbot-nginx
sudo certbot --nginx -d llm.example.com

Upgrade Nginx to a production proxy config

Edit the TLS-enabled server block that Certbot created (usually in the same file).

Swap the location block for the proxy rules below.

sudo nano /etc/nginx/sites-available/llm.conf

server {
  listen 443 ssl http2;
  server_name llm.example.com;

  ssl_certificate /etc/letsencrypt/live/llm.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;

  # Keep uploads sane; some clients send large JSON payloads
  client_max_body_size 10m;

  # Simple rate limiting to reduce abuse
  limit_req_zone $binary_remote_addr zone=llm_limit:10m rate=10r/s;

  location /v1/ {
    limit_req zone=llm_limit burst=20 nodelay;

    proxy_pass http://127.0.0.1:8000;
    proxy_http_version 1.1;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # Timeouts: model inference can take time
    proxy_connect_timeout 10s;
    proxy_send_timeout 300s;
    proxy_read_timeout 300s;

    # Optional: protect with a static API key header
    # if ($http_x_api_key != "CHANGE_ME") { return 401; }
  }

  location = / {
    return 200 "ok\n";
    add_header Content-Type text/plain;
  }
}

server {
  listen 80;
  server_name llm.example.com;
  return 301 https://$host$request_uri;
}

Reload:

sudo nginx -t
sudo systemctl reload nginx

Rate limiting is a basic first line of defense.

If you need a more complete set of patterns (bursts, allowlists, different limits per route), use HostMyCode’s Nginx rate limiting tutorial.

Step 10: Test the OpenAI-compatible API end-to-end

From your laptop (or any machine with access), verify the models endpoint:

curl -s https://llm.example.com/v1/models | jq

Test chat completions

This hits the OpenAI-style chat route.

If your client library supports a custom base URL, point it here.

curl -s https://llm.example.com/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "llm",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Write a 2-sentence status update for a deployment."}
    ],
    "temperature": 0.2
  }' | jq

If you enabled the static API key

Uncomment the Nginx check and pass the header:

curl -s https://llm.example.com/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'X-API-Key: CHANGE_ME' \
  -d '{"model":"llm","messages":[{"role":"user","content":"ping"}]}' | jq

Static keys aren’t a full auth solution, but they shut down drive-by abuse quickly.

If this endpoint is customer-facing, put proper auth in your app.

Step 11: Make it usable from applications (SDK config)

Most OpenAI-compatible SDKs let you override the base URL. The exact setting varies by language.

In every case, you’ll need the same three inputs:

Base URL: https://llm.example.com/v1
Model name: llm (we set --served-model-name llm)
API key: if you enabled one at Nginx, store it in your app’s secret manager

If several apps will use the endpoint, keep it on its own subdomain.

That makes rotation, logging, and rate-limit/WAF rules easier to manage.

Operational checks: keep the endpoint stable under load

Most failures are predictable: VRAM pressure, requests stacking up, or inconsistent performance on shared hosts.

These checks catch common problems quickly.

Watch GPU and VRAM while testing

watch -n 1 nvidia-smi

If VRAM pegs at max and latency spikes, reduce request concurrency or choose a smaller model.
If GPU utilization is low but requests are slow, you may be CPU-bound (tokenization) or I/O-bound (weights on slow storage).

Confirm vLLM is only bound to localhost

sudo ss -lntp | grep 8000

You want 127.0.0.1:8000, not 0.0.0.0:8000.

Log locations you’ll use later

Service logs: sudo journalctl -u vllm --since "1 hour ago"
Nginx access/error: /var/log/nginx/access.log, /var/log/nginx/error.log

Basic monitoring (don’t fly blind)

Even a small LLM API benefits from alerting on CPU, RAM, disk, and GPU memory.

If you don’t already have monitoring in place, follow this HostMyCode monitoring tutorial and add GPU checks alongside it.

Security notes specific to self-hosted LLM APIs

Self-hosting only pays off if you protect the endpoint.

Public LLM APIs attract automated scraping and prompt-injection attempts quickly.

Don’t expose vLLM directly: keep it on localhost; proxy via Nginx.
Use TLS everywhere: avoid plaintext on shared networks.
Add an auth layer: at minimum, a static header; ideally JWT/OAuth in your app.
Rate limit: not just to stop bots—also to avoid GPU thrash during traffic spikes.
Separate duties: run vLLM under a dedicated user with limited write access.

If this box also hosts websites, apply standard web hardening too.

HostMyCode’s Nginx hardening guide maps well to an API-only server.

Limitations and next steps (practical upgrades)

This is a single-node deployment with production-friendly defaults.

The next upgrades tend to look like this:

Front-end chat UI: If you want a web interface for internal users, deploy a UI separately. (HostMyCode already covers this pattern in this Open WebUI tutorial; the same UI approach can sit in front of an API endpoint.)
RAG: Add document search via pgvector or Qdrant, but keep it as a separate service so upgrades don’t collide.
Backups: Back up your configuration and any local prompts/embeddings. Don’t bother backing up public model weights unless bandwidth is painful.
Isolation: For teams with mixed workloads, move the LLM endpoint onto its own GPU VPS or dedicated server.

If you’re deploying an internal assistant or an app-facing endpoint, host it somewhere you control the network path and the logs. A HostMyCode VPS is a sensible baseline. If you’d rather not manage OS updates and security maintenance yourself, managed VPS hosting covers the operational busywork.

FAQ

Can I run vLLM on a CPU-only VPS?

You can, but it’s rarely a good fit for interactive chat. Expect high latency and low throughput.

If you have to do it, use a small model and treat it as a background worker, not a real-time API.

Should I expose vLLM on port 8000 to the internet?

No. Bind to 127.0.0.1 and proxy through Nginx with TLS, rate limiting, and an auth check.

It’s easier to audit and safer to operate.

How do I pick the right model size?

Work backward from your GPU VRAM.

With 8–12 GB VRAM, start with a 7B–8B instruction model. With 24 GB+, you can try larger models, but watch VRAM in nvidia-smi while load testing.

Where should I store secrets like HF tokens or API keys?

Use systemd drop-in environment files with root-only permissions, or a secrets manager if you already run one.

Avoid hardcoding keys in Nginx configs that end up in git.

What’s the simplest way to add a UI on top of this API?

Run a separate front-end service and point it at https://llm.example.com/v1.

Keep the UI and the model server isolated so you can update each independently.

Summary

You now have vLLM running under systemd, locked down with UFW, and reachable through an OpenAI-compatible /v1 API behind Nginx TLS.

That’s a solid base for internal assistants, app features, and controlled experiments—without shipping prompts to third-party hosted APIs.

If you plan to keep this stable under real traffic, give it dedicated resources. Start with a HostMyCode VPS, then move to a dedicated server as model size and concurrency increase.