llama.cpp Quantization: Pick Q4 vs Q5 vs Q8

You can run a capable LLM on a plain VPS—if you treat memory as the hard limit. That’s where llama.cpp quantization options matter most. Q4, Q5, and Q8 often decide whether the model starts cleanly or crawls into swap.

Below is a practical way to pick a quant level based on your RAM/VRAM budget. You’ll also get exact commands to download and test GGUF models.

What you’ll build (and what it won’t do)

By the end, you’ll have llama.cpp running on Ubuntu. You’ll also have a repeatable way to choose Q4 vs Q5 vs Q8 for a specific machine.

You’ll finish with a one-command prompt test. Rerun it any time to sanity-check quality and latency before you wire the model into an app.

• Works well for: private admin assistants, internal support tools, lightweight site chat, code explanation, summarization.
• Not great for: high-concurrency public chat endpoints without batching; long-context workloads on small RAM; “reasoning-heavy” tasks on aggressive Q4.

If you need an OpenAI-compatible server for multi-user traffic, llama.cpp can do it. But vLLM often handles throughput better.

HostMyCode has a dedicated tutorial on securing vLLM endpoints if you go that route: Secure vLLM API With Nginx, Keys, and Rate Limits.

Prerequisites and sizing cheat-sheet

This guide assumes Ubuntu Server and a non-root sudo user. The same commands and sizing logic work on Debian.

Minimum server specs (practical)

• CPU-only: 4 vCPU+, 16 GB RAM for “small to mid” models; 32 GB RAM is a safer baseline if you’ll experiment.
• GPU-assisted: 8–16 GB VRAM can be enough for many Q4/Q5 runs if you offload most layers. System RAM still matters.
• Disk: 20–60 GB free. GGUF files are large; keep room for multiple quants.

How to think about Q4 vs Q5 vs Q8

Quantization reduces the precision of model weights. Fewer bits means lower memory use and often faster CPU inference.

The trade-off is lower fidelity. In practice, that can show up as drift, weaker instruction-following, or fragile formatting on tougher prompts.

• Q4 (4-bit): Best for tight RAM/VRAM. Fast and cheap. Expect more “rough edges” on tricky prompts.
• Q5 (5-bit): Often the sweet spot. Noticeably better stability than Q4 while staying memory-friendly.
• Q8 (8-bit): Closest to higher precision behavior. Bigger memory footprint; CPU can be slower. Great if you can afford it.

A useful rule of thumb: run Q5 if you can keep at least 20% RAM free under load.

If you’re flirting with swap or OOM kills, drop to Q4. If you have plenty of headroom and want more consistent answers, consider Q8.

Install llama.cpp on Ubuntu (build from source)

Building from source gives you current GGUF support and the CLI tools in one place. On most servers, it’s the least surprising path.

1. Install dependencies:

   sudo apt-get update
   sudo apt-get install -y git build-essential cmake python3 python3-venv

2. Clone llama.cpp:

   cd /opt
   sudo git clone https://github.com/ggerganov/llama.cpp.git
   sudo chown -R $USER:$USER /opt/llama.cpp
   cd /opt/llama.cpp

3. Build a CPU version (a solid baseline even if you later add GPU):

   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
   cmake --build build -j"$(nproc)"

Quick sanity check:

/opt/llama.cpp/build/bin/llama-cli --help | head

If you plan to keep this online for apps or multiple operators, use stable infrastructure.

A HostMyCode VPS is a clean fit for CPU inference and internal tools. A dedicated server makes more sense for sustained workloads or GPU passthrough.

Pick llama.cpp quantization options using a repeatable sizing method

Don’t pick a quant level by vibes. Do two checks: (1) your real memory budget and (2) the model’s GGUF size.

Then reserve room for KV cache and the OS so the machine stays responsive.

Step 1: measure real available memory

free -h

Use available, not just free. On a working server, “32 GB installed” may mean “18–22 GB actually available” once other services take their share.

Step 2: estimate memory needs (weights + KV cache)

With GGUF models, the file size is a solid first estimate for weight memory. From there, budget for KV cache and basic system overhead.

• Weights: roughly the GGUF size loaded in RAM (CPU) or VRAM (GPU-offloaded layers reduce RAM but increase VRAM use).
• KV cache: grows with context length and the number of layers. Longer context costs real memory, quickly.
• Overhead: leave room for the OS, reverse proxy, monitoring, and page cache. If Linux has to fight for RAM, you’ll see “mystery” slowdowns.

Practical budgeting guideline (CPU-only): keep model weights + KV cache under ~70–75% of RAM available.

If you’re hosting a website and AI on the same VPS, aim closer to 60–65%.

Step 3: decision table you can actually use

Constraint	Pick	Why
RAM/VRAM is tight, you need it to run reliably	Q4	Largest memory reduction; better odds of avoiding swap and OOM kills
You can afford a bit more memory, want better instruction following	Q5	Balanced quality vs footprint; often the best default for self-hosting
You have headroom, want fewer subtle mistakes	Q8	Higher fidelity weights; better for “admin assistant” prompts and careful tasks

If you’re building RAG or embeddings alongside llama.cpp, count that memory too.

For a PostgreSQL RAG store, pgvector is a common choice; see Set Up pgvector for RAG on a PostgreSQL VPS.

Download a GGUF model (Q4/Q5/Q8) and run a real test prompt

llama.cpp runs GGUF files, and most model authors publish several quant variants.

The workflow stays the same: download the GGUF, run llama-cli, then judge memory behavior and output quality with a fixed prompt.

Step 1: create a model directory

sudo mkdir -p /srv/llm/models
sudo chown -R $USER:$USER /srv/llm

Step 2: download GGUF files

You can grab files via a browser or CLI. If you have a direct URL to a GGUF file:

cd /srv/llm/models
# Example pattern (replace with an actual GGUF URL you trust)
curl -L -o model.Q5_K_M.gguf "https://example.com/path/to/model.Q5_K_M.gguf"

Download Q4 and Q8 variants too if you want a quick side-by-side comparison. Use explicit filenames to avoid “which file did I deploy?” problems later.

Step 3: run the same prompt across quants

Start conservative. Use context that matches your likely usage, and set threads equal to vCPUs.

You’re looking for a stable baseline, not a leaderboard score.

cd /opt/llama.cpp
./build/bin/llama-cli \
  -m /srv/llm/models/model.Q5_K_M.gguf \
  -t "$(nproc)" \
  -c 4096 \
  -n 256 \
  --temp 0.2 \
  -p "You are a Linux admin assistant. Explain why a server might start swapping heavily after deploying a new PHP app, and list 5 commands to diagnose it."

What to watch:

• Memory: in another SSH session run htop or free -h.
• Swap usage: any meaningful swapping will hurt latency.
• Output quality: does it follow instructions, keep to 5 commands, and stay coherent?

Quick diagnostic: detect swap pressure early

vmstat 1

If you see sustained non-zero values in the si/so columns, you’re swapping.

On a live web server, treat that as a reliability problem, not a “minor performance hit.”

Drop from Q8 to Q5 or Q4, reduce context (-c), or move the model to larger hardware.

Optional: convert an HF model to GGUF and quantize it yourself

If you can’t find a GGUF build at the quant level you need, you can convert and quantize it yourself.

Expect extra disk usage and a non-trivial wait. The workflow is still repeatable.

Reality check: conversion details vary by model family. The commands below show the common llama.cpp path using its conversion scripts. If your model needs special arguments, follow the model card and the llama.cpp docs.

Step 1: prepare a Python venv

cd /opt/llama.cpp
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip

Step 2: export to GGUF (FP16) as a starting point

Assuming you already downloaded a Hugging Face model into /srv/llm/hf/MyModel (weights present locally):

mkdir -p /srv/llm/gguf
python3 convert_hf_to_gguf.py /srv/llm/hf/MyModel \
  --outfile /srv/llm/gguf/mymodel-fp16.gguf \
  --outtype f16

Step 3: quantize to Q4/Q5/Q8

Use llama-quantize from the build output:

/opt/llama.cpp/build/bin/llama-quantize \
  /srv/llm/gguf/mymodel-fp16.gguf \
  /srv/llm/gguf/mymodel-q4.gguf \
  Q4_K_M

/opt/llama.cpp/build/bin/llama-quantize \
  /srv/llm/gguf/mymodel-fp16.gguf \
  /srv/llm/gguf/mymodel-q5.gguf \
  Q5_K_M

/opt/llama.cpp/build/bin/llama-quantize \
  /srv/llm/gguf/mymodel-fp16.gguf \
  /srv/llm/gguf/mymodel-q8.gguf \
  Q8_0

Then rerun the exact same prompt against each output file.

If Q4 starts bending constraints or “forgetting” requirements, Q5 is usually the clean upgrade. If Q5 still feels unreliable and you can spare RAM/VRAM, try Q8.

Honest trade-offs you’ll notice in production

The quant level doesn’t just change output quality. It also changes how the whole box behaves once real users show up.

• Latency variance: when you’re near RAM limits, token generation gets spiky. Q4 often stays stable where Q8 starts thrashing.
• Context sensitivity: if you rely on long prompts (support tickets, logs, docs), Q4 may drop details earlier.
• Instruction adherence: Q5 and Q8 generally keep structure better (lists, JSON, strict counts).
• Total cost: Q4 can turn a “needs a dedicated server” project into a “fits on a VPS” project.

If your plan includes a web UI, put the model behind something that handles auth and conversation history.

Open WebUI is a common pairing with local models, but keep it private or protect it properly.

Secure your llama.cpp endpoint (minimum viable)

Even if you only run llama-cli today, most setups eventually grow an HTTP endpoint (chatbot, internal tool, n8n workflow).

Treat it like any other private service from day one.

• Bind to localhost and reverse proxy through Nginx/Caddy with TLS.
• Add auth (basic auth or token-based) and rate limits.
• Log access and watch for abuse patterns.

Two practical references from HostMyCode that translate well to llama.cpp-style services:

• Run Ollama Behind Nginx With Basic Auth
• Set Up Let’s Encrypt With Caddy for a VPS Site

Operational checklist: keep it reliable on a VPS

Self-hosted AI usually breaks in predictable ways. Disk fills up, RAM gets squeezed, and nobody remembers which quant file is live.

This short checklist helps you avoid the avoidable outages.

• Pin your model files: store in /srv/llm/models, keep a README.txt with the chosen quant and rationale.
• Monitor memory: set alerts for swap usage and OOM events.
• Keep headroom: don’t co-host a busy database and a large model on the same small VPS.
• Test restores: if you’re generating or downloading large GGUFs, back up metadata and scripts, not just the weights.

If you want lightweight monitoring that’s easy to deploy alongside services, see Run Netdata on a VPS for Live Server Monitoring.

FAQ: llama.cpp quantization options (Q4 vs Q5 vs Q8)

Is Q4 always worse than Q5 and Q8?

Not always. For simple chat and short answers, Q4 can be perfectly acceptable.

The gap shows up on longer prompts, strict formatting, and tricky multi-step instructions.

Why does Q8 sometimes feel slower on CPU?

Q8 uses larger weight representations than Q4/Q5. That increases memory bandwidth pressure and cache misses.

As a result, tokens/sec can drop on CPU even when quality improves.

If I have a GPU, do I still care about system RAM?

Yes. Offloading layers to VRAM helps, but you still need RAM for the runtime.

You may also need RAM for remaining layers (if not fully offloaded) and for KV cache, depending on your setup and context length.

What’s the fastest way to choose between Q4 and Q5?

Download both GGUF files, run the same prompt with the same -c, and watch swap usage and instruction adherence.

If Q5 fits without swap and behaves better, keep it.

Can I run this next to a production website?

You can, but be strict about memory headroom. If the site already uses most RAM (PHP-FPM, Redis, database), run the model on a separate VPS and call it over a private network.

Summary: a practical way to pick Q4/Q5/Q8

Pick the quant based on the thing that takes servers down first: memory.

Run Q5 if it fits with headroom, drop to Q4 if you’re close to swap, and use Q8 when you can afford the footprint and want cleaner outputs.

Validate your choice with a fixed prompt before you integrate it into anything user-facing.

If you’re building this for real users, put it on infrastructure that doesn’t fight you. A managed VPS hosting plan from HostMyCode keeps the underlying server stable while you focus on the model and the application layer.