
Most “AI server” guides stop at “it runs.” This one goes further. You’ll end up with a local LLM endpoint you can put behind TLS and call from your app.
You’ll run llama.cpp with CUDA on Ubuntu Server. You’ll compile with GPU support and tune layer offload so you don’t burn through VRAM. You’ll also expose an OpenAI-compatible API.
This layout fits a single-GPU VPS or a small dedicated box. You’ll run one model behind one API, with predictable spend.
If you later need multi-user auth, rate limits, or a chat UI, you can add them without reworking the base setup.
What you’ll build (and what you won’t)
By the end you’ll have:
- llama.cpp compiled with CUDA on Ubuntu Server
- A GGUF model stored on disk
- An OpenAI-compatible HTTP endpoint using
llama-server - A systemd service so it starts on boot
- A basic Nginx reverse proxy you can extend with TLS and access control
This tutorial does not cover fine-tuning, multi-GPU sharding, Kubernetes, or benchmarking.
The goal is a stable, operator-friendly baseline with clear tradeoffs.
Resource planning for a single-GPU server
Pick the box first. Most “mystery failures” come from memory limits.
- VRAM is the hard ceiling for how many layers you can offload. 8–12 GB VRAM works for many 7B–8B models with partial offload. 16–24 GB VRAM gives you more headroom for heavier quantizations and larger context sizes.
- System RAM still needs to hold the full model, even with offload. Plan for 16 GB for small models. Use 32 GB+ if you expect larger GGUF files and bigger context windows.
- CPU still matters. Prompt processing and token generation use both CPU and GPU. A slow CPU will cap throughput, especially with concurrent requests.
- Disk: NVMe improves load times and reduces pain if the system swaps. It will not compensate for insufficient RAM or VRAM.
If you’re buying infrastructure for sustained load, a GPU-capable dedicated server is usually the least surprising option.
For lighter usage, a GPU VPS can be enough.
HostMyCode’s HostMyCode dedicated servers and HostMyCode VPS are a practical baseline for self-hosting models without per-request API fees.
Prerequisites
- Ubuntu Server (LTS is typical for production)
- Root or sudo access
- One NVIDIA GPU available to the server
- Inbound network plan: you’ll usually keep the API private and front it with a reverse proxy
Assumptions: the commands below use bash.
The server can also reach GitHub to clone sources.
Step 1: Update Ubuntu and install build prerequisites
Start from a clean baseline. If the kernel updates, plan a reboot.
sudo apt update
sudo apt -y upgrade
sudo apt -y install build-essential cmake git pkg-config curl ca-certificates nginx
If you deploy via Docker or something like Coolify, the install will look different.
Here, you’ll use a direct system install. It’s easier to debug and operate.
Step 2: Install NVIDIA drivers (and confirm the GPU is visible)
On Ubuntu, the least painful route is the recommended driver package.
First, ask Ubuntu what it wants to use:
ubuntu-drivers devices
Then install the recommended driver (Ubuntu will tell you which package that is). Example:
sudo apt -y install nvidia-driver-###
Reboot so the kernel modules actually load:
sudo reboot
After reboot, confirm the driver and GPU are recognized:
nvidia-smi
You should see the GPU listed, along with driver and memory details. If nvidia-smi fails, stop here.
Common causes: you didn’t reboot, the wrong driver installed, or Secure Boot is blocking unsigned modules. This is more common on physical servers than VPS.
Step 3: Install the CUDA toolkit (for building CUDA support)
To compile llama.cpp with GPU support, you need a working CUDA toolchain.
Installing the packaged toolkit keeps updates simpler.
sudo apt -y install nvidia-cuda-toolkit
Verify the compiler is available:
nvcc --version
If your host already ships CUDA, you may already be set.
The non-negotiables are a working NVIDIA driver and a working CUDA compiler toolchain for the build.
Step 4: Compile llama.cpp with CUDA support
Create a home for source builds:
sudo mkdir -p /opt/src
sudo chown -R $USER:$USER /opt/src
cd /opt/src
Clone llama.cpp:
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
Build with CMake. Enable CUDA support via GGML_CUDA.
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j"$(nproc)"
Confirm the server binary exists:
ls -la build/bin | grep -E 'llama-server|server'
On current llama.cpp, you should see llama-server (plus other utilities).
If it’s missing, review the build output for CUDA detection or compile errors.
Step 5: Put a GGUF model on the server
llama.cpp loads GGUF. You can download a GGUF model from a trusted source (often Hugging Face) or convert one yourself.
For a first pass, pick one model you know fits your RAM/VRAM budget.
Create a model directory:
sudo mkdir -p /opt/models
sudo chown -R $USER:$USER /opt/models
Download your chosen .gguf into /opt/models. Example (placeholder URL):
cd /opt/models
# Replace this with the real direct download URL for your GGUF file
curl -L -o model.gguf "https://example.com/path/to/model.gguf"
Tip: use a descriptive filename like llama-3-8b-instruct-q4_k_m.gguf. It saves time once you have multiple variants.
Step 6: Run llama.cpp with CUDA (first interactive test)
This is your end-to-end check.
You want to see the model load, CUDA initialize, and a real response generate.
Start llama-server bound to localhost only:
cd /opt/src/llama.cpp
./build/bin/llama-server \
--model /opt/models/model.gguf \
--host 127.0.0.1 \
--port 8080 \
--ctx-size 4096 \
--n-gpu-layers 999 \
--threads "$(nproc)"
Notes:
--n-gpu-layerscontrols GPU offload. A very high value tells llama.cpp to offload as many layers as fit in VRAM. If VRAM runs out, expect errors or painful paging.--ctx-sizeincreases memory use and can raise latency. Don’t start with huge context on a small VRAM card.
From another shell, hit the OpenAI-style endpoint:
curl -s http://127.0.0.1:8080/v1/models | jq
If you don’t have jq yet:
sudo apt -y install jq
Then try a chat completion:
curl -s http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "local-model",
"messages": [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "Give me a 5-line checklist for hardening an Ubuntu VPS."}
],
"temperature": 0.4
}' | jq -r '.choices[0].message.content'
If you get a response back, the fundamentals are in place.
Step 7: Tune GPU layer offload without crashing the box
“Offload everything” works until VRAM runs out.
Use a simple workflow: start high, watch VRAM, then back off until it stays stable.
In a separate shell, watch GPU memory:
watch -n 1 nvidia-smi
If you hit out-of-memory, adjust in this order:
- Lower offload: try
--n-gpu-layers 60, then40, then20. - Drop context size:
--ctx-size 2048is a reasonable starting point on smaller GPUs. - Switch to a smaller quantization (more compression). This is often the quickest fix.
If you want a deeper quantization workflow for llama-family models, see HostMyCode’s tutorial: Quantize Llama Models for Ollama on a VPS.
The same size-vs-quality tradeoff applies here, even if you aren’t using Ollama.
Step 8: Create a dedicated system user and directories
Don’t run inference as root.
Use a locked-down system account, and keep the filesystem layout predictable.
sudo useradd --system --home /opt/llama --create-home --shell /usr/sbin/nologin llama
sudo mkdir -p /opt/llama/bin /opt/llama/models /opt/llama/run
sudo chown -R llama:llama /opt/llama
Copy the built binary and your model into the service directories:
sudo cp /opt/src/llama.cpp/build/bin/llama-server /opt/llama/bin/
sudo cp /opt/models/model.gguf /opt/llama/models/model.gguf
sudo chown -R llama:llama /opt/llama
sudo chmod 0755 /opt/llama/bin/llama-server
Step 9: Run llama-server as a systemd service
Create a systemd unit at /etc/systemd/system/llama-server.service:
sudo tee /etc/systemd/system/llama-server.service > /dev/null <<'EOF'
[Unit]
Description=llama.cpp OpenAI-compatible API server
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=llama
Group=llama
WorkingDirectory=/opt/llama
ExecStart=/opt/llama/bin/llama-server \
--model /opt/llama/models/model.gguf \
--host 127.0.0.1 \
--port 8080 \
--ctx-size 4096 \
--n-gpu-layers 60 \
--threads 8
Restart=on-failure
RestartSec=2
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/llama/run
[Install]
WantedBy=multi-user.target
EOF
Adjust:
--n-gpu-layersbased on what your VRAM can sustain.--threadsto match your CPU. Don’t max it out just because you can.
Enable and start the service:
sudo systemctl daemon-reload
sudo systemctl enable --now llama-server
sudo systemctl status llama-server --no-pager
Tail logs while you test:
journalctl -u llama-server -f
Step 10: Put Nginx in front (reverse proxy + sane defaults)
Don’t expose llama-server directly.
Put Nginx in front instead. It gives you a clean place to add TLS, auth, and request limits.
Create an Nginx site config:
sudo tee /etc/nginx/sites-available/llama-api > /dev/null <<'EOF'
server {
listen 80;
server_name ai-api.example.com;
client_max_body_size 2m;
location / {
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeouts for long generations
proxy_connect_timeout 10s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
# Basic safety against slowloris-ish behavior
send_timeout 30s;
}
}
EOF
Enable the site and reload Nginx:
sudo ln -s /etc/nginx/sites-available/llama-api /etc/nginx/sites-enabled/llama-api
sudo nginx -t
sudo systemctl reload nginx
That gives you plain HTTP. For HTTPS, use your normal Let’s Encrypt flow.
If you prefer an ingress-style setup with Docker, HostMyCode’s guide Traefik With Docker Compose for HTTPS on a VPS shows a clean pattern you can adapt for AI endpoints too.
Step 11: Secure the endpoint (minimum viable production)
If the endpoint is public, assume someone will abuse it. LLM endpoints are expensive to run.
A handful of slow requests can pin your GPU.
At minimum:
- Bind llama-server to localhost (already done):
--host 127.0.0.1 - Enable HTTPS at Nginx (terminate TLS at the edge)
- Add auth (Basic Auth is fine for internal tools; OAuth2 proxy is better for teams)
- Rate limit requests at Nginx to cap abuse
- Restrict by IP if this is only for your office/VPN
Example Nginx rate limiting (add inside http {} in /etc/nginx/nginx.conf or a file in /etc/nginx/conf.d/):
limit_req_zone $binary_remote_addr zone=llama_ratelimit:10m rate=5r/s;
Then inside your server block:
location /v1/ {
limit_req zone=llama_ratelimit burst=10 nodelay;
proxy_pass http://127.0.0.1:8080;
}
If you need real login controls for a web UI, HostMyCode has a good pattern using Open WebUI with an auth proxy: Open WebUI With OAuth2 Proxy On A VPS.
Even if you skip Open WebUI, the OAuth2 proxy approach works well for internal services.
Step 12: Test from a real application (OpenAI-compatible client)
The point of an OpenAI-compatible endpoint is SDK reuse.
Here’s a minimal Python smoke test that targets your server.
Install Python venv tooling:
sudo apt -y install python3-venv
mkdir -p ~/llama-client-test && cd ~/llama-client-test
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install openai
Create test.py:
cat > test.py <<'EOF'
from openai import OpenAI
client = OpenAI(
base_url="http://ai-api.example.com/v1",
api_key="not-used"
)
resp = client.chat.completions.create(
model="local-model",
messages=[
{"role": "system", "content": "You are concise."},
{"role": "user", "content": "Write a short status update about deploying an API."},
],
temperature=0.4,
)
print(resp.choices[0].message.content)
EOF
python test.py
If you’re using HTTPS and auth, update base_url.
Handle credentials at the proxy layer.
For Basic Auth, you can often keep auth out of the client entirely.
Operational notes you’ll feel after day 2
Self-hosted inference tends to fail in the same predictable ways.
Plan for them now, and you’ll sleep better.
- Concurrency: one GPU + one model hits a practical ceiling quickly. Start with low concurrency and queue at the app layer.
- Context size inflation: bigger context costs memory and latency. Keep it small unless you can justify the tradeoff.
- Model upgrades: treat model swaps like code deploys. Keep the previous model around for fast rollback.
- Logs: keep
journalctlclose, and ship logs centrally if you manage multiple hosts. - Disk pressure: GGUF files accumulate fast. Once a VPS hits 100% disk, everything starts misbehaving. Monitor free space and enforce cleanup.
If disk-full incidents sound familiar, this is a useful companion: Disk Full on VPS: Find Large Files and Free Space Safely.
Optional: keep the endpoint private with a tunnel
If you only need personal access (or your team connects remotely), you may not want a public AI endpoint.
A tunnel lets you keep Nginx private and still reach it externally.
HostMyCode’s tutorial Set Up Cloudflare Tunnel on a VPS for Private Admin shows a straightforward approach. The same pattern works for /v1/ endpoints.
Summary: a practical single-GPU llama.cpp deployment
You now have llama.cpp built with CUDA, a GGUF model on disk, and an OpenAI-compatible endpoint your apps can call.
Day to day, you’ll mostly tune two flags: --n-gpu-layers (VRAM pressure) and --ctx-size (memory use and latency).
For reliability, leave headroom. A single GPU sitting at 95% VRAM all day is one bad prompt away from paging, stalling, or crashing.
If you’re heading toward production traffic, consider a dedicated GPU machine or a larger VPS from HostMyCode—see managed VPS hosting for an ops-light option, or HostMyCode dedicated servers when you need consistent GPU performance.
If you’re building self-hosted AI features for a website or an internal tool, keep the workload on infrastructure you control. A HostMyCode VPS is a solid fit for light inference and private endpoints. Once you depend on GPU uptime and predictable throughput, HostMyCode dedicated servers are usually the better bet.
FAQ
Do I need a GPU to use llama.cpp?
No. llama.cpp runs on CPU-only servers as well. GPU offload mostly improves latency and throughput, but you still need enough RAM to hold the model.
Why does “offload all layers” fail on my GPU?
You run out of VRAM. Lower --n-gpu-layers, reduce --ctx-size, or switch to a smaller/more-quantized GGUF model.
Is llama.cpp’s OpenAI-compatible endpoint a full OpenAI replacement?
It covers common chat/completions flows, but it doesn’t match every OpenAI feature. Validate the exact SDK calls you use (and your error handling) before shifting production traffic.
Can I expose this endpoint publicly?
You can, but expect abuse. Put it behind HTTPS, authentication, and rate limiting at minimum. For many setups, keeping it private via VPN/tunnel is the safer choice.
How do I add a web UI later?
Add a chat front-end like Open WebUI, point it at your local endpoint, and secure it with an auth proxy. HostMyCode’s Open WebUI tutorials are a good starting point.