Self-Host llama.cpp Server API on Ubuntu VPS

A CPU-only AI endpoint won’t match a GPU machine for throughput. Still, it can handle a lot. Think internal tools, low-traffic chat, support drafts, lightweight RAG glue, or a “good enough” coding helper for a small team.

The goal here is a boring, controlled deployment. Use a simple stack, set firm limits, and keep a clean perimeter.

This tutorial shows you how to self-host llama.cpp server on an Ubuntu VPS, run a quantized model, keep it running under systemd, and publish it safely behind Nginx.

By the end, you’ll have an OpenAI-style HTTP API your app can call. You’ll also have a reverse proxy for TLS, basic access control, timeouts, and rate limiting.

What you’re building (and what it is not)

You’ll run llama-server (the llama.cpp server binary) on your VPS. It loads a .gguf quantized model and exposes HTTP endpoints. Many “OpenAI-ish” client libraries can talk to it.

• Good for: CPU inference, low-to-moderate request volume, predictable cost, private network use, and straightforward ops.
• Not good for: high concurrency, long context at high token rates, or image generation. If you need that, move inference to a GPU stack (vLLM or similar) on a dedicated GPU host.

If you plan to add GPU later, keep the same pattern (systemd + Nginx). Then swap the inference backend.

HostMyCode can run both sides: a small HostMyCode VPS for routing/auth and a larger GPU machine for inference once CPU-only stops being comfortable.

Resource sizing for CPU-only llama.cpp

Set expectations before you install anything. CPU inference speed depends on CPU generation, clock speed, memory bandwidth, and how aggressively the model is quantized.

• VPS baseline: 4 vCPU / 8–16 GB RAM is a sensible starting point for one quantized 7–8B model.
• Disk: 20–40 GB free usually covers model files and logs. Put models on fast storage (NVMe helps with cold starts and avoids painful swapping).
• RAM rule of thumb: budget for model size plus overhead. A 7–8B model in Q4 may be ~4–5 GB on disk; RAM usage rises once it’s loaded and starts accumulating KV cache.
• Concurrency: plan for 1–3 active chats on a small CPU VPS without ugly tail latency. It’s better to cap and queue than let the host thrash.

On CPU, quantization choices show up fast in both speed and quality. If you’re unsure, start with Q4 and adjust from there.

For a deeper comparison, see llama.cpp quantization: pick Q4 vs Q5 vs Q8.

Prerequisites

• Ubuntu Server (LTS is a safe choice in 2026) with sudo access
• A domain or subdomain you control (for TLS), pointed at your VPS IP
• Nginx installed (we’ll show commands)

If you haven’t chosen hosting yet, a general-purpose managed VPS hosting plan fits well. It helps if you want the endpoint running without babysitting base OS updates and reboots.

Step 1: Create a dedicated user and directories

Run the service under its own user. This keeps permissions cleaner and makes systemd hardening easier.

sudo adduser --system --group --home /opt/llama --shell /usr/sbin/nologin llama
sudo mkdir -p /opt/llama/bin /opt/llama/models /var/log/llama
sudo chown -R llama:llama /opt/llama /var/log/llama

Step 2: Install build dependencies (or use a prebuilt binary)

You can compile llama.cpp quickly on most VPS nodes. Building it yourself also makes it clear what you’re running.

On Ubuntu, install the basics:

sudo apt update
sudo apt install -y build-essential cmake git pkg-config

Now fetch and build llama.cpp. Then place the CPU server binary in /opt/llama/bin:

cd /tmp
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j"$(nproc)"

# Copy the server binary
sudo cp -v build/bin/llama-server /opt/llama/bin/
sudo chown llama:llama /opt/llama/bin/llama-server

Quick diagnostic: confirm the binary runs.

sudo -u llama /opt/llama/bin/llama-server --help | head

Step 3: Download a quantized GGUF model

llama.cpp expects a GGUF file. Pick a model you’re licensed to use. Also match it to your latency and RAM budget.

On CPU-only, a 7–8B model in Q4 is often the practical upper bound for a typical VPS.

Save the model under /opt/llama/models. Example (replace the URL with your actual model):

sudo -u llama bash -lc 'cd /opt/llama/models && wget -O model.gguf "https://example.com/path/to/your-model.gguf"'

Operational note: don’t pull GGUF files from random links. Treat model files like untrusted executables. Stick to reputable registries, and verify hashes when they’re provided.

Step 4: Run llama-server manually once (smoke test)

Start conservative. Bind only to localhost, pick a moderate context size, and keep parallelism low.

Once it’s stable, tune for your workload.

sudo -u llama /opt/llama/bin/llama-server \
  --model /opt/llama/models/model.gguf \
  --host 127.0.0.1 --port 8080 \
  --ctx-size 4096 \
  --threads $(nproc) \
  --parallel 1 \
  --n-predict 256 \
  --log-file /var/log/llama/llama-server.log

In another shell, verify something is listening. Endpoints can vary by build, so treat this as a connectivity check:

curl -sS http://127.0.0.1:8080/ | head

If you get any response (even an error page), the process is up and reachable. Stop it (Ctrl+C) before moving on to systemd.

Step 5: Create a systemd service with sensible limits

Systemd turns this into a normal service. It starts at boot and restarts after crashes.

It can also enforce limits. Those limits help keep a load spike from taking out the VPS.

Create /etc/systemd/system/llama-server.service:

sudo tee /etc/systemd/system/llama-server.service > /dev/null <<'EOF'
[Unit]
Description=llama.cpp Server API
After=network.target

[Service]
Type=simple
User=llama
Group=llama
WorkingDirectory=/opt/llama

# Start with conservative defaults; tune once stable.
ExecStart=/opt/llama/bin/llama-server \
  --model /opt/llama/models/model.gguf \
  --host 127.0.0.1 --port 8080 \
  --ctx-size 4096 \
  --threads %NPROC \
  --parallel 1 \
  --n-predict 256 \
  --log-file /var/log/llama/llama-server.log

Restart=on-failure
RestartSec=2

# Hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/llama /opt/llama/models

# Resource controls (adjust to your VPS size)
CPUAccounting=true
MemoryAccounting=true
# Example: cap at ~80% of 8GB. Change or remove if too strict.
MemoryMax=6500M

# Avoid huge file descriptor usage if a client misbehaves
LimitNOFILE=4096

[Install]
WantedBy=multi-user.target
EOF

Two important edits:

• Replace %NPROC with a real number. Systemd won’t expand it automatically here.
• Adjust MemoryMax based on your VPS RAM and model choice.

Check your CPU thread count:

nproc

Then patch the service (example sets threads to 4):

sudo sed -i 's/--threads %NPROC/--threads 4/' /etc/systemd/system/llama-server.service

Reload and start:

sudo systemctl daemon-reload
sudo systemctl enable --now llama-server
sudo systemctl status llama-server --no-pager

Logs:

sudo journalctl -u llama-server -f

Step 6: Put Nginx in front (TLS, auth, timeouts, rate limits)

Don’t publish the llama.cpp port directly. Keep it bound to 127.0.0.1, and let Nginx handle public traffic.

This gives you a clean perimeter. It’s also where you add HTTPS, request limits, and access control.

Install Nginx if needed:

sudo apt install -y nginx

Create a simple rate limit zone. Add this inside http { } in /etc/nginx/nginx.conf (or a file included from it):

sudo tee /etc/nginx/conf.d/llama-rate-limits.conf > /dev/null <<'EOF'
# 10 requests/minute per IP, with small bursts.
limit_req_zone $binary_remote_addr zone=llama_per_ip:10m rate=10r/m;
EOF

Now create a site config, for example: /etc/nginx/sites-available/llama-api

sudo tee /etc/nginx/sites-available/llama-api > /dev/null <<'EOF'
server {
    listen 80;
    server_name llama-api.example.com;

    # Keep this endpoint private during setup.
    # Remove after TLS is configured.
    location / {
        return 444;
    }
}
EOF

Enable it:

sudo ln -s /etc/nginx/sites-available/llama-api /etc/nginx/sites-enabled/llama-api
sudo nginx -t && sudo systemctl reload nginx

Add TLS: The cleanest operational pattern on a VPS is automatic TLS with Caddy. If you standardize on Nginx, use Certbot.

If you prefer Caddy for fewer moving parts, follow Set up Let’s Encrypt with Caddy for a VPS site and proxy to 127.0.0.1:8080. Below is an Nginx-based approach.

Install Certbot and issue a certificate:

sudo apt install -y certbot python3-certbot-nginx
sudo certbot --nginx -d llama-api.example.com

After Certbot updates your Nginx server block, replace the config with a proxying version. This example adds:

• rate limiting
• a simple header-based API key check (good enough for many internal uses)
• timeouts so slow generations don’t pin worker processes forever
• request body cap

sudo tee /etc/nginx/sites-available/llama-api > /dev/null <<'EOF'
# Expect a header: X-API-Key: YOUR_LONG_RANDOM_VALUE
map $http_x_api_key $llama_key_ok {
    default 0;
    "CHANGE_ME_LONG_RANDOM_KEY" 1;
}

server {
    listen 80;
    server_name llama-api.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name llama-api.example.com;

    # Certbot typically manages these paths. Keep the ones it installed.
    ssl_certificate     /etc/letsencrypt/live/llama-api.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llama-api.example.com/privkey.pem;

    client_max_body_size 1m;

    # Basic abuse control
    limit_req zone=llama_per_ip burst=5 nodelay;

    location / {
        # Simple auth gate
        if ($llama_key_ok = 0) { return 401; }

        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeouts tuned for long responses
        proxy_connect_timeout 5s;
        proxy_send_timeout 300s;
        proxy_read_timeout 300s;

        # Disable buffering for streaming responses if your client uses them
        proxy_buffering off;
    }
}
EOF

Reload:

sudo nginx -t && sudo systemctl reload nginx

Security note: Header keys work well for internal tools and low-risk use. For public exposure, use a private network, Cloudflare Tunnel, or a real auth gateway.

For the “private admin” pattern, adapt Set up Cloudflare Tunnel on a VPS for private admin so the API isn’t directly reachable.

Step 7: Test the OpenAI-style API from your workstation

The exact paths depend on how your llama.cpp server build exposes routes. Many builds support OpenAI-compatible endpoints such as /v1/chat/completions.

If you’re unsure, check what the server returns at the root or under /v1.

Try a basic call (replace host and API key):

curl -sS https://llama-api.example.com/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'X-API-Key: CHANGE_ME_LONG_RANDOM_KEY' \
  -d '{
    "model": "local-gguf",
    "messages": [
      {"role": "system", "content": "You are concise."},
      {"role": "user", "content": "Write a 3-step checklist to rotate Nginx logs."}
    ],
    "temperature": 0.2,
    "max_tokens": 150
  }' | jq

If you don’t have jq installed locally, drop the | jq. If the endpoint path differs, run /opt/llama/bin/llama-server --help and adjust.

The important part is the flow. Nginx terminates TLS and proxies back to localhost.

Step 8: Tune for stability (threads, context, KV cache, timeouts)

On CPU-only hosts, predictability matters more than chasing a few extra tokens/sec. Change one setting at a time.

After each change, watch latency and RAM.

• Threads: start at your vCPU count. On noisy neighbors, fewer threads can reduce contention and improve tail latency.
• Context size (--ctx-size): larger context helps longer conversations, but it costs memory and speed. If you see OOM kills or swapping, reduce this first.
• Parallel requests (--parallel): keep it at 1 on small VPS nodes. If you raise it, expect slower tokens/sec per user.
• Nginx timeouts: raise proxy_read_timeout if generations are being cut off. Lower it if you’d rather fail fast.

Quick diagnostic commands:

# Watch CPU and memory
htop

# Watch system memory pressure
free -h
vmstat 1

# Confirm Nginx is the only public entry point
sudo ss -lntp | egrep '(:8080|:80|:443)'

# Watch llama logs
tail -f /var/log/llama/llama-server.log

Step 9: Add a firewall rule set that matches the design

Publicly, you only need ports 80/443 (plus SSH from your admin range). The llama.cpp backend should stay on localhost.

sudo apt install -y ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status verbose

If the VPS hosts other apps, add explicit rules for those services. Avoid opening broad port ranges.

Step 10: Production housekeeping (backups, updates, and rollbacks)

Back up the parts you’ll actually want to restore:

• Nginx config: /etc/nginx/
• Systemd unit: /etc/systemd/system/llama-server.service
• Model file(s): /opt/llama/models/ (optional, but it saves download time)

For change control, keep your configs in a private Git repo. If you already run internal Git, see Deploy Gitea on a VPS with SSH Git and HTTPS.

When you update llama.cpp, rebuild into a temporary directory. Then swap binaries cleanly:

sudo systemctl stop llama-server
sudo cp /opt/llama/bin/llama-server /opt/llama/bin/llama-server.bak
sudo cp -v /tmp/llama.cpp/build/bin/llama-server /opt/llama/bin/llama-server
sudo chown llama:llama /opt/llama/bin/llama-server
sudo systemctl start llama-server

If the new build misbehaves, restore llama-server.bak and restart.

Common pitfalls (and fast fixes)

• OOM kills / random restarts: reduce --ctx-size, reduce --parallel, add RAM, or loosen MemoryMax if it’s too strict.
• Requests hang behind Nginx: increase proxy_read_timeout. Also confirm proxy_buffering off; if you expect streaming.
• CPU pegged at 100% even idle: look for a client hammering the endpoint. Tighten limit_req and consider an IP allowlist.
• Model loads but answers are poor: try a different quantization or model family. CPU inference punishes over-compression if you care about quality.

Where to go next: RAG, UI, and private access

Once the API is stable, retrieval-augmented generation is usually the next step. A practical VPS-friendly route is PostgreSQL + pgvector.

Start with Add pgvector to PostgreSQL on Ubuntu for RAG and wire your app to embeddings and retrieval.

If you want a ready-made chat UI, Open WebUI is a common pick. HostMyCode has a walkthrough for running it behind a proxy: Run Open WebUI with Ollama behind Nginx proxy.

The same reverse-proxy details (timeouts, headers, TLS) apply.

FAQ

Can I run this on a 2 vCPU / 4 GB RAM VPS?

Sometimes, but it’s tight. Use a small quantized model and a smaller context size (often 2048–4096). Expect slow responses and very limited concurrency.

Should I expose llama.cpp directly on the internet?

No. Bind it to 127.0.0.1 and put Nginx in front. Add TLS, an auth layer, and rate limits. If it’s an internal tool, use a tunnel or VPN.

What’s the best quantization for CPU inference?

Q4 is a common starting point for CPU-only because it reduces RAM and improves speed. If quality is unacceptable, try a less aggressive quantization (with higher memory use and slower performance).

How do I add an AI chatbot to my website using this endpoint?

Call the API from your backend (PHP, Node.js, Python), then stream results to the browser. Don’t call the endpoint directly from frontend JavaScript unless you have strong auth controls.

When should I move to GPU hosting?

If you need multiple simultaneous users, longer context, or faster token generation, CPU-only gets frustrating quickly. At that point, move inference to a dedicated GPU machine and keep your Nginx gateway on the VPS. You can scale both on HostMyCode dedicated servers and VPS plans under the same account.

Summary

You now have a CPU-friendly llama.cpp API running under systemd and published through Nginx with TLS, basic access control, and rate limits.

It’s intentionally boring. Boring services stay up.

If you want a stable place to run it—and room to grow into RAG or a UI later—deploy it on a HostMyCode VPS and keep the backend bound to localhost behind a hardened proxy.