Self-Host vLLM on a GPU Dedicated Server

High-throughput chat or completions usually fails for a simple reason: you run out of VRAM, or you waste it. This walkthrough shows you how to self-host vLLM on a GPU server you control using Docker + NVIDIA Container Toolkit. You’ll also expose an OpenAI-compatible API your apps can call.

The target setup is intentionally plain: a single-GPU dedicated server with repeatable performance. You’ll add the basics (TLS, auth, and rate limiting) so you don’t publish an unprotected model endpoint to the public internet.

What you’ll deploy (and who this is for)

This guide is for teams that already call OpenAI-style endpoints and want a drop-in internal replacement. vLLM is a strong fit when throughput matters. It batches requests well, keeps the GPU busy, and handles concurrent calls without falling apart.

Server type: GPU dedicated server (recommended) or a GPU VPS if your provider supports it.
OS: Ubuntu Server LTS (commands assume Ubuntu).
Runtime: Docker with NVIDIA Container Toolkit.
API: vLLM OpenAI-compatible server (/v1/chat/completions and /v1/completions).

If you want a browser UI later, pair this with HostMyCode’s Open WebUI tutorial: Deploy Open WebUI Behind Cloudflare Tunnel on VPS. Here, we’ll keep it API-first and ops-friendly.

Infrastructure requirements and honest limitations

Size the machine around VRAM first. CPU and system RAM help with tokenization, networking, and queueing. But the model weights (and most of the pain) live in GPU memory.

GPU VRAM: 16 GB minimum for small/medium quantized models; 24–48 GB is the comfortable range for production.
System RAM: 32 GB minimum; 64 GB recommended if you handle large prompts, long context, or multiple model files.
CPU: 8+ modern cores recommended; more cores reduce queueing under load.
Storage: NVMe strongly recommended. Model pulls can be 5–50+ GB.
Network: 1 Gbps uplink is plenty for most API traffic, but latency to your app matters.

Limitations you should expect: (1) VRAM is your first hard ceiling. (2) Long contexts can cut tokens/sec sharply. (3) A public endpoint will be scanned and abused unless you add auth and rate limits.

If you need predictable behavior, dedicated GPU hardware is usually the least surprising option. HostMyCode offers dedicated hardware where you can keep the stack stable: HostMyCode dedicated servers.

Step 1: Prepare Ubuntu and install NVIDIA drivers

Start with a fresh Ubuntu server. Log in as a sudo-capable user.

sudo apt update
sudo apt -y upgrade
sudo apt -y install curl ca-certificates gnupg lsb-release htop ufw

Install the recommended NVIDIA driver package. On servers, Ubuntu’s driver tooling is often the fastest path to a working baseline.

sudo ubuntu-drivers devices
sudo ubuntu-drivers install

Reboot so the kernel picks up the driver:

sudo reboot

After reboot, confirm the GPU is visible:

nvidia-smi

If nvidia-smi fails, fix that first. vLLM can’t paper over a broken host driver.

Step 2: Install Docker and NVIDIA Container Toolkit

Run vLLM in a container so upgrades and rollbacks stay predictable. You can install Docker from Ubuntu packages or Docker’s repo.

The Ubuntu package is fine for many servers. Production teams often prefer Docker’s official repo. Either way, you want a stable Docker daemon.

sudo apt -y install docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Log out and back in so the group membership applies. Or run:

newgrp docker

Install NVIDIA Container Toolkit so your containers can see the GPU. NVIDIA’s docs are the source of truth. This is the common Ubuntu flow that works on most hosts.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt -y install nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Confirm GPU access from inside a container:

docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi

If the container prints your GPU details, you’re ready to run vLLM.

Step 3: Choose a model and estimate VRAM before you pull

vLLM can serve many Hugging Face-style models. The decision that matters is model size versus available VRAM. You also need enough space left for KV cache.

Keep these rules of thumb in mind:

Quantization reduces VRAM but can reduce quality and sometimes throughput. For 16–24 GB GPUs, it’s often the practical choice.
Leave headroom for KV cache. If weights consume all VRAM, longer prompts and concurrency can push you into OOM.
Context length costs memory. A larger --max-model-len means more KV cache allocation.

If you’re deciding between quant levels for llama.cpp-style files, HostMyCode has a practical breakdown: llama.cpp quantization: pick Q4 vs Q5 vs Q8. vLLM’s supported quantization formats differ by model, but the trade-offs are similar.

Suggested starting point: use a model you already run internally. Start with conservative limits, then tune from there.

--max-model-len 4096 to start (raise later).
--gpu-memory-utilization 0.90 (don’t jump straight to 0.99).

Step 4: Run vLLM with an OpenAI-compatible API (Docker)

Create a directory for vLLM state and cached downloads.

sudo mkdir -p /opt/vllm
sudo chown -R $USER:$USER /opt/vllm

vLLM will download model files. Cache them on disk so container restarts don’t re-pull gigabytes.

Mounting a Hugging Face cache directory is the usual approach.

mkdir -p /opt/vllm/hf-cache

If you need access to gated models, export a token. If you only use public models, skip this and remove the env var from the Docker command.

export HUGGING_FACE_HUB_TOKEN="YOUR_TOKEN_HERE"

Run the vLLM OpenAI server container. Replace MODEL_ID with the model you want (for example, an instruction-tuned model your org permits). Keep the overall structure the same.

docker run -d --restart unless-stopped \
  --name vllm \
  --gpus all \
  -p 8000:8000 \
  -e HUGGING_FACE_HUB_TOKEN="$HUGGING_FACE_HUB_TOKEN" \
  -v /opt/vllm/hf-cache:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model MODEL_ID \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90

Tail logs until the model finishes loading:

docker logs -f vllm

Quick health check:

curl -s http://127.0.0.1:8000/v1/models | jq .

If you don’t have jq installed:

sudo apt -y install jq

Step 5: Test chat and completions with curl

vLLM speaks OpenAI-compatible JSON. That means you can reuse existing SDKs and request formats.

Start with curl so you can see the raw request and response.

Chat completions:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "MODEL_ID",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Write a 3-step checklist to rotate Linux logs."}
    ],
    "temperature": 0.2
  }' | jq -r '.choices[0].message.content'

Text completions: (some clients still use this format)

curl -s http://127.0.0.1:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "MODEL_ID",
    "prompt": "Give me 5 Nginx security headers and why they matter:\n",
    "max_tokens": 200,
    "temperature": 0.3
  }' | jq -r '.choices[0].text'

If it works locally, resist the urge to open the port. Put a gate in front of it first.

Step 6: Put vLLM behind Nginx with TLS, API keys, and rate limits

Publishing port 8000 directly on an internet-facing host is asking for trouble. Put Nginx in front.

Terminate TLS, add authentication, and rate-limit requests.

This section mirrors best practices from HostMyCode’s hardening tutorial: Secure vLLM API With Nginx, Keys, and Rate Limits. The config below is a minimal baseline you can paste and run.

Install Nginx

sudo apt -y install nginx apache2-utils
sudo systemctl enable --now nginx

Allow only what you need in UFW

sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw --force enable
sudo ufw status

Do not allow 8000/tcp in the firewall. Keep vLLM behind Nginx and reachable only via localhost or a private interface.

Create an API key file

A shared secret header can be enough for internal tools and staging. Generate a strong key and store it somewhere you won’t lose it.

sudo mkdir -p /etc/nginx/keys
openssl rand -hex 32 | sudo tee /etc/nginx/keys/vllm_api_key.txt >/dev/null

Clients will be expected to send X-API-Key with this value.

Nginx site config

Create /etc/nginx/sites-available/vllm.conf:

sudo tee /etc/nginx/sites-available/vllm.conf >/dev/null <<'EOF'
limit_req_zone $binary_remote_addr zone=vllm_per_ip:10m rate=2r/s;

map $http_x_api_key $api_key_ok {
  default 0;
  "" 0;
  ~.+ 0;
}

server {
  listen 80;
  server_name ai-api.example.com;

  location /.well-known/acme-challenge/ {
    root /var/www/html;
  }

  location / {
    return 301 https://$host$request_uri;
  }
}

server {
  listen 443 ssl http2;
  server_name ai-api.example.com;

  ssl_certificate     /etc/letsencrypt/live/ai-api.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/ai-api.example.com/privkey.pem;

  client_max_body_size 10m;

  location / {
    # Basic rate limiting per IP
    limit_req zone=vllm_per_ip burst=10 nodelay;

    # Require API key
    set $expected_key "";
    if (-f /etc/nginx/keys/vllm_api_key.txt) { }

    # Read key by including it as a variable via perl/lua would be nicer,
    # but to keep this minimal, use basic auth instead (next block).

    proxy_pass http://127.0.0.1:8000;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_read_timeout 300s;
  }
}
EOF

Practical note: Stock Nginx can’t easily load a file into a variable without extra modules. For a clean baseline, use HTTP Basic Auth (it works well for server-to-server calls). Create a password file:

sudo htpasswd -c /etc/nginx/.htpasswd vllmclient

Edit the HTTPS server block and add:

sudo sed -i '/location \/ {/a\
    auth_basic "vLLM";\
    auth_basic_user_file /etc/nginx/.htpasswd;\
' /etc/nginx/sites-available/vllm.conf

Enable the site and reload Nginx:

sudo ln -s /etc/nginx/sites-available/vllm.conf /etc/nginx/sites-enabled/vllm.conf
sudo nginx -t
sudo systemctl reload nginx

Add TLS certificates

Use Let’s Encrypt with Certbot, or use Caddy if you prefer simpler TLS automation. If you want Caddy, follow: Set Up Let’s Encrypt With Caddy for a VPS Site.

With Certbot (Nginx plugin):

sudo apt -y install certbot python3-certbot-nginx
sudo certbot --nginx -d ai-api.example.com

Once certificates are installed, run nginx -t again and reload Nginx.

Step 7: Connect your app (OpenAI SDK compatibility)

Most OpenAI-compatible SDKs support a custom base URL. The option name varies by language and library version. The pattern is consistent.

Base URL: https://ai-api.example.com/v1
API key: any string (some SDKs require it), plus your Nginx Basic Auth credentials if you enabled it

If your client can’t send Basic Auth cleanly, keep vLLM on a private network and call it from inside your infrastructure. A tunnel also works.

For a tunnel-based pattern, see: Secure Ollama API With Cloudflare Tunnel on VPS. The goal is the same: don’t leave a model endpoint hanging out on the open internet.

Step 8: Sizing VRAM for throughput (quick checklist)

Don’t rely on gut feel. Measure, adjust, and keep notes.

This checklist avoids the most common failure modes:

Start with lower context: keep --max-model-len at 4096 until you’ve proven stability under load.
Watch VRAM during warmup: run watch -n 1 nvidia-smi while vLLM loads and while you send concurrent requests.
Leave 5–10% VRAM headroom: raise --gpu-memory-utilization slowly. A setting that looks fine at idle can fail under concurrency.
Prefer one model per GPU at first. Multi-model serving is possible, but debugging becomes tedious fast.

If you hit OOM repeatedly, lower --max-model-len and/or pick a smaller or quantized model. In practice, “faster” often means “small enough to batch well without paging.”

Step 9: Operational basics (restart, logs, updates, monitoring)

Most production incidents aren’t mysterious. It’s usually a stuck container, a disk quietly filled by model cache, or a traffic spike you didn’t plan for.

Restart and inspect

docker ps
docker restart vllm
docker logs --tail 200 vllm

Check disk usage (model cache grows quietly)

df -h
du -sh /opt/vllm/hf-cache

Keep security updates automatic

On an internet-connected GPU host, unattended security updates should be non-negotiable. Follow: Set Up Automated Security Updates on Ubuntu VPS.

Monitor the host

At minimum, track CPU, RAM, disk, and network. For a lightweight view with history, Netdata is a practical option: Run Netdata on a VPS for live server monitoring.

For the GPU, nvidia-smi plus vLLM logs still does most of the work.

Step 10: Optional add-ons (RAG, UI, and crawler controls)

This tutorial stays focused on serving completions. Once the endpoint is stable, these are the usual next steps:

Add a chat UI: Open WebUI works well for internal users (separate service).
Add RAG: if you want “ask my docs” with permissions, use a vector store such as pgvector. A solid starting point is Set up pgvector for RAG on a PostgreSQL VPS plus Install PostgreSQL on Ubuntu and secure remote access.
Control AI crawlers on your website: if you operate public sites, decide what you want indexed. Use Self-host an AI crawlers blocklist with robots.txt.

Summary: a production-shaped vLLM deployment

You now have vLLM running on a GPU server with Docker, an OpenAI-compatible API, and a safer network posture via Nginx + TLS. The shape is deliberate: one GPU, one model, one endpoint.

That structure stays calm when you’re debugging under pressure.

If your priority is consistent throughput and isolation from noisy neighbors, run this on dedicated hardware. Start with HostMyCode dedicated servers. If you’d rather spend less time on the surrounding plumbing, consider managed VPS hosting for the pieces around the model (reverse proxy, TLS, firewalling, and monitoring).

If you’re building internal chat, support tools, or coding assistants, running your own model endpoint keeps latency predictable and spending easier to forecast. HostMyCode can provision the right base for this tutorial, from dedicated servers to HostMyCode VPS for surrounding services like Nginx, Postgres, or a Web UI.

FAQ

Should I run vLLM on a VPS or a dedicated server?

For steady throughput, dedicated GPU hardware is usually simpler. A VPS can work if you’re guaranteed GPU access and consistent performance, but contention and driver mismatches show up more often.

Do I need to expose port 8000 to the internet?

No. Keep vLLM bound to the server and publish only a reverse proxy on 443 with auth and rate limits, or keep the endpoint private and call it over a VPN/tunnel.

Why do I get CUDA out-of-memory errors under concurrency?

You’re probably running too close to VRAM capacity. Reduce --max-model-len, lower concurrency, choose a smaller/quantized model, or reduce --gpu-memory-utilization so KV cache has room.

Can I use an OpenAI SDK with vLLM?

Yes. Point your client’s base URL to your server’s /v1 endpoint and use the OpenAI-compatible routes. If you add proxy auth, make sure your client can send those headers.

What’s the fastest way to add “chat with my docs”?

Add RAG with pgvector or Qdrant, then build a small service that retrieves relevant chunks and forwards them to vLLM. Start with pgvector on Postgres if you already run Postgres in your stack.