Back to tutorials
Tutorial

Install llama.cpp Server on Ubuntu for Local LLM API

Learn to run llama.cpp server on Ubuntu with systemd and Nginx TLS for a lightweight OpenAI-style local LLM endpoint.

By Anurag Singh
Updated on Oct 17, 2026
Category: Tutorial
Share article
Install llama.cpp Server on Ubuntu for Local LLM API

You don’t need Kubernetes or a huge platform stack to put an LLM behind HTTPS. If you want a small, OpenAI-style HTTP endpoint you can manage like any other Linux service, llama.cpp fits well.

This guide shows how to run llama.cpp server on Ubuntu without Docker, manage it with systemd, and put Nginx in front for TLS and basic access control.

These steps assume real infrastructure (a VPS or dedicated box) with root access.

For predictable latency, use NVMe storage and enough RAM to hold the model plus headroom.

If you’re aiming for production traffic, a managed VPS hosting plan can keep patching and monitoring from becoming a second job.

What you’ll build (and what it won’t do)

You’ll deploy:

  • llama.cpp “server” compiled from source on Ubuntu
  • A dedicated Linux user, model directory, and service unit
  • Nginx TLS reverse proxy for HTTPS + simple auth controls
  • An endpoint you can call from apps, cron jobs, or internal tools

Honest limitations:

  • CPU-only inference works for smaller models and low concurrency. A 4 vCPU VPS won’t handle “chat for 500 users.”
  • On GPU, VRAM sets your practical model ceiling. Without a GPU, RAM capacity and memory bandwidth become the main bottlenecks.
  • llama.cpp supports multiple backends, but real performance depends on your hardware, quantization, and context length.

Prerequisites and sizing checklist

Recommended baseline (CPU-only):

  • Ubuntu LTS (server)
  • 8 vCPU minimum for comfortable interactive use
  • 16–32 GB RAM for smaller quantized models (plan higher if you want larger contexts)
  • Fast NVMe disk (models are read frequently; cold-starts hurt on slow storage)

GPU notes (optional): If your server has an NVIDIA GPU, llama.cpp can use CUDA builds.

This is where tokens/sec usually improves and CPU load drops.

In multi-tenant environments, prefer a dedicated server or a GPU-capable VPS.

If you’re choosing infrastructure, a HostMyCode VPS is a sensible place to start for CPU-only pilots. Scale after you measure throughput and latency with your real model.

You will need:

  • A domain/subdomain you can point at the server (for TLS). If you don’t have one, register via HostMyCode domains.
  • Port 443 reachable from clients. Port 8000 (or similar) stays private behind Nginx.

Security baseline: Lock down your firewall before you publish anything.

This pairs well with this UFW web server tutorial.

Step 1: Prepare Ubuntu packages (build tools + Nginx)

Update the system and install the toolchain:

sudo apt update
sudo apt -y upgrade

sudo apt -y install \
  git build-essential cmake pkg-config \
  python3 python3-venv \
  nginx \
  certbot python3-certbot-nginx

For a CPU build, Ubuntu doesn’t need much beyond the packages above.

If you plan to build with CUDA, install the NVIDIA driver and CUDA toolkit first.

That’s outside this CPU-first walkthrough, but the service and Nginx setup stays the same.

Step 2: Create a dedicated service user and directories

Run the model server as a non-root user.

Keep everything under /opt so ownership and permissions stay simple.

sudo useradd --system --home /opt/llama --create-home --shell /usr/sbin/nologin llama

sudo mkdir -p /opt/llama/{bin,models,logs,src}
sudo chown -R llama:llama /opt/llama

This layout gives you:

  • /opt/llama/bin for compiled binaries
  • /opt/llama/models for .gguf model files
  • /opt/llama/logs for log output (systemd will also capture logs)
  • /opt/llama/src for the git checkout

Step 3: Build llama.cpp from source (CPU build)

Switch to the llama user for the build.

This keeps the repo and artifacts from ending up owned by root.

sudo -u llama -H bash -lc 'cd /opt/llama/src && git clone https://github.com/ggerganov/llama.cpp.git'

sudo -u llama -H bash -lc 'cd /opt/llama/src/llama.cpp && cmake -S . -B build -DCMAKE_BUILD_TYPE=Release'

sudo -u llama -H bash -lc 'cd /opt/llama/src/llama.cpp && cmake --build build -j $(nproc)'

After it compiles, copy the HTTP server binary into /opt/llama/bin.

The executable name can change between revisions.

List the directory first, then copy the server target you actually have (often llama-server).

sudo -u llama -H bash -lc 'ls -la /opt/llama/src/llama.cpp/build/bin'

# Example if the binary is llama-server:
sudo -u llama -H bash -lc 'cp -av /opt/llama/src/llama.cpp/build/bin/llama-server /opt/llama/bin/'

sudo -u llama -H bash -lc 'chmod 0755 /opt/llama/bin/llama-server'

If your build output doesn’t include a server binary: upstream occasionally reorganizes targets.

Re-check build/bin, find the HTTP server executable, and adjust the copy command to match.

Step 4: Add a GGUF model file (and keep it realistic)

llama.cpp typically consumes .gguf models.

You’ll bring your own model file (and follow its licensing terms).

Drop it under /opt/llama/models and make sure the llama user can read it.

# Example path (you provide the file)
sudo -u llama -H bash -lc 'ls -lh /opt/llama/models'

Quick sizing rule: even quantized GGUF models need real RAM to run.

If the model barely fits, expect swapping, latency spikes, and odd timeouts.

Don’t “solve” that with swap.

Use a smaller quantization, reduce context, or add RAM.

If you want a practical way to track CPU/RAM/disk/GPU trade-offs, set up monitoring early.

HostMyCode’s Netdata monitoring tutorial is a solid baseline for catching memory pressure and sustained load before users feel it.

Step 5: Run llama.cpp server once by hand (sanity test)

Before you involve systemd, start the server in a shell.

First, confirm the model loads cleanly and the port is listening.

Keep it bound to localhost for this first pass.

sudo -u llama -H bash -lc '/opt/llama/bin/llama-server \
  --model /opt/llama/models/YOUR-MODEL.gguf \
  --host 127.0.0.1 \
  --port 8000 \
  --ctx-size 4096'

In a second SSH session, hit the endpoint:

curl -s http://127.0.0.1:8000/ | head

If it exits immediately, read the error output.

Common causes are a bad model path, insufficient RAM, or a model format mismatch.

Step 6: Create a systemd service (auto-start, restart, logs)

Now turn it into a real service.

Create /etc/systemd/system/llama-server.service:

sudo nano /etc/systemd/system/llama-server.service

Paste this, then adjust the model filename, context size, and threads for your box:

[Unit]
Description=llama.cpp local LLM HTTP server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=llama
Group=llama
WorkingDirectory=/opt/llama

# Adjust these to match your hardware and model
ExecStart=/opt/llama/bin/llama-server \
  --model /opt/llama/models/YOUR-MODEL.gguf \
  --host 127.0.0.1 \
  --port 8000 \
  --ctx-size 4096 \
  --threads 8

Restart=on-failure
RestartSec=2

# Basic hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/llama

# Prevent accidental core dumps leaking prompts
LimitCORE=0

[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now llama-server
sudo systemctl status llama-server --no-pager

Tail logs:

journalctl -u llama-server -f

Pitfall: After changing ExecStart, run sudo systemctl daemon-reload before restarting.

Otherwise systemd keeps the old command.

Step 7: Put Nginx in front (TLS, buffering, timeouts)

Keep llama.cpp bound to 127.0.0.1.

Only Nginx should be reachable from the internet on 443.

This split lets you add TLS, rate limits, and auth without touching the model process.

Create an Nginx site config. Replace llm.example.com with your hostname:

sudo nano /etc/nginx/sites-available/llama-api.conf

Config (set up for long generations and streaming-ish responses):

server {
  listen 80;
  server_name llm.example.com;

  location / {
    return 301 https://$host$request_uri;
  }
}

server {
  listen 443 ssl http2;
  server_name llm.example.com;

  # Certbot will manage ssl_certificate/ssl_certificate_key

  # Keep request bodies reasonable; prompts can be large, but unlimited is risky
  client_max_body_size 2m;

  # Timeouts: generation can take time on CPU
  proxy_connect_timeout 10s;
  proxy_send_timeout 300s;
  proxy_read_timeout 300s;

  location / {
    proxy_pass http://127.0.0.1:8000;
    proxy_http_version 1.1;

    # Preserve client details
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # Helpful for low-latency streaming responses
    proxy_buffering off;
  }
}

Enable the site and reload Nginx:

sudo ln -s /etc/nginx/sites-available/llama-api.conf /etc/nginx/sites-enabled/llama-api.conf
sudo nginx -t
sudo systemctl reload nginx

Issue a TLS certificate with Certbot:

sudo certbot --nginx -d llm.example.com

If you want a deeper HTTPS troubleshooting checklist (wrong chain, renewal failures, validation issues), this is worth keeping around: TLS setup guide tutorial (2026).

Step 8: Add basic access control (don’t run an open LLM endpoint)

An unauthenticated generation endpoint won’t stay private for long.

Bots scan for these constantly, and abuse arrives quickly.

Pick at least one control below before you call it “done.”

Option A: IP allowlist (fast and simple)

Add this inside the location / block:

allow 203.0.113.10;   # your office IP
allow 198.51.100.0/24; # your VPN subnet
deny all;

Option B: Basic auth for internal tools

Install htpasswd tooling and create a password file:

sudo apt -y install apache2-utils
sudo htpasswd -c /etc/nginx/.llama_htpasswd youruser

Then add to location /:

auth_basic "Restricted";
auth_basic_user_file /etc/nginx/.llama_htpasswd;

Option C: Rate limiting at Nginx

Even with auth, rate limiting protects you from buggy clients, retries gone wrong, and accidental loops.

For a deeper pattern library, see Nginx rate limiting tutorial.

For a minimal setup:

# Put this inside the http {} block in /etc/nginx/nginx.conf
limit_req_zone $binary_remote_addr zone=llama_limit:10m rate=5r/s;

And inside your location /:

limit_req zone=llama_limit burst=10 nodelay;

Reload Nginx after changes:

sudo nginx -t && sudo systemctl reload nginx

Step 9: Test over HTTPS (OpenAI-style usage patterns)

The exact endpoints exposed by llama.cpp vary by revision and flags.

The workflow stays the same: send JSON, receive generated text.

Start by confirming the reverse proxy works end to end:

curl -I https://llm.example.com/

If you enabled Basic auth:

curl -u youruser -I https://llm.example.com/

Tip: If your app expects an OpenAI-compatible schema, you may need an adapter layer.

That depends on your server mode and the endpoints it exposes.

For many internal tasks (RAG glue, ticket triage, summarization), the direct llama.cpp HTTP endpoint is often enough.

Step 10: 운영 체크—keep it stable on a VPS

Once requests flow, operate it like any other production service:

  • Pin resources: set a sensible --threads value so inference doesn’t starve your web stack.
  • Watch memory: once the kernel starts swapping, latency stops being predictable.
  • Track 5xx rates: most issues show up as timeouts or upstream resets from Nginx.
  • Patch cadence: update Ubuntu packages regularly; rebuild llama.cpp during maintenance windows.

For general server hygiene that fits AI endpoints well, don’t skip recovery planning.

The runbook style in this disaster recovery tutorial helps you avoid “we’ll fix it live” at 2 a.m.

Troubleshooting: common failure modes and fast fixes

Nginx returns 502 Bad Gateway

  • Confirm llama.cpp listens locally: ss -lntp | grep 8000
  • Check the service: sudo systemctl status llama-server
  • Read logs: journalctl -u llama-server --since "10 min ago"

It starts, then gets killed

  • Likely out-of-memory. Check: dmesg -T | tail -n 50 for OOM killer lines.
  • Reduce context size (--ctx-size), choose a smaller quant, or upgrade RAM.

Responses are slow or time out

  • Increase Nginx proxy_read_timeout for CPU-only generation.
  • Reduce max tokens / generation length from the client side.
  • Set a lower concurrency expectation. One heavy request can saturate CPU.

Certbot can’t issue/renew

If you want a clean home for a private LLM endpoint, start with a HostMyCode VPS for CPU pilots, then move to managed VPS hosting when you need production-grade ops. You get predictable resources, full Linux control, and a team that’s used to running services.

FAQ

Can I run llama.cpp server alongside my website on the same VPS?

Yes, but expect contention.

Cap llama.cpp threads, keep it behind Nginx, and watch load average and memory. If your website traffic spikes, inference latency usually spikes with it.

Do I need a GPU to run llama.cpp?

No. CPU-only is fine for smaller quantized models and low concurrency.

A GPU mainly helps with throughput and latency for interactive chat-style workloads.

Why use systemd instead of running it in a tmux session?

systemd gives you automatic restarts, consistent logs via journald, and clean startup on boot.

That’s the gap between a demo process and an operated service.

Should I expose llama.cpp directly on the internet?

Avoid it.

Put it behind TLS, add auth or an allowlist, and rate-limit requests. Treat it like any other internal API with real cost and real abuse potential.

What’s the simplest way to add HTTPS?

Nginx + Certbot with a real domain name.

The steps here cover the happy path, and the HostMyCode TLS guide helps when renewals or validation fail.

Summary

You now have a llama.cpp HTTP service bound to localhost, managed by systemd, and exposed through Nginx over HTTPS.

Next steps are straightforward: add monitoring, test backup/restore, and decide whether you need GPU acceleration or a smaller model with lower context.

If you want the endpoint private, stable, and low-maintenance, host it on a VPS you control.

A HostMyCode VPS is a solid baseline, and managed VPS hosting is the low-friction option once uptime matters more than tinkering.