Quantize Llama Models for Ollama on a VPS

Your VPS can run useful Llama-class models, but RAM is usually the first hard limit. If the OOM killer is taking out ollama, don’t start by paying for a bigger box. Start by learning to quantize models for Ollama the right way.

The core loop is simple. Pick a GGUF quant that fits your memory budget, import it cleanly, then validate speed and output quality.

That last step matters. It’s how you learn the trade-offs instead of guessing.

This is a hands-on, VPS-focused workflow. You’ll convert a base model to GGUF (or start from an existing GGUF).

Then you’ll choose practical quant levels (Q4/Q5/Q8), create an Ollama model from the file, and run quick checks. Those checks catch common “it loads, but it’s unusable” failures.

What you’ll build (and what you need)

By the end, you’ll have a custom Ollama model that runs reliably on a memory-limited VPS. You’ll also have a repeatable way to compare quant choices.

• Target OS: Ubuntu 24.04 LTS or Debian 12 (commands shown for Ubuntu/Debian)
• Runtime: Ollama installed and working
• Build tool: llama.cpp for conversion + quantization
• Disk: plan for 2–3× the final GGUF size during conversion (temporary files + caches)

Resource expectations (honest and practical):

• CPU-only VPS: quantized 7B–8B models are realistic; 13B can be slow and memory tight.
• RAM rule of thumb: a Q4 GGUF is often about half the size of FP16 weights. You still need overhead for KV cache, runtime, and Linux page cache.
• Swap helps stability, not speed. If you’re swapping during generation, latency will spike.

If you need a stable base box for self-hosted AI, a HostMyCode VPS is a good fit for CPU inference and model serving. You control RAM, disk, and firewall rules.

If you prefer someone else to handle OS hardening and baseline monitoring, managed VPS hosting is the lower-friction route.

Step 1: Confirm Ollama works and note your RAM ceiling

Verify that Ollama is running and you can pull a small model. This smoke test confirms DNS, outbound access, and permissions.

ollama --version
sudo systemctl status ollama --no-pager

# Smoke test: pull something small
ollama pull llama3.2:1b
ollama run llama3.2:1b "Say hello in one sentence."

Now check the basics so you know what you’re optimizing for:

free -h
nproc
df -h /

Rule of thumb: on a small VPS (roughly 4–8 GB RAM), start with Q4 or Q5. Q8 can look tempting, but it often becomes unstable once your KV cache grows with longer prompts.

Step 2: Choose the right GGUF quant (Q4 vs Q5 vs Q8)

Quantization reduces weight precision to cut memory use. It often improves CPU throughput too.

The cost is output quality. You may also see weaker behavior on long, reasoning-heavy prompts.

On a VPS, quant selection is mostly about sizing. Pick the smallest quant that still answers your real prompts correctly.

• Q4 (default for tight RAM): best chance to fit on 8–16 GB boxes, with a more noticeable hit on reasoning tasks.
• Q5 (balanced): a common sweet spot; quality is often close to Q8 without the same memory pressure.
• Q8 (quality-first CPU quant): best outputs among these, but memory climbs quickly. Don’t start here if you’re already near the limit.

If you want a deeper quant breakdown, HostMyCode also has a focused explainer on how to pick Q4 vs Q5 vs Q8 in llama.cpp. Here, you’ll take that decision and apply it directly to an Ollama workflow.

Step 3: Install llama.cpp tools for conversion and quantization

Ollama can run local models. For custom GGUF builds, you usually prep the files with llama.cpp first. Then you import the result into Ollama.

Install build deps:

sudo apt update
sudo apt install -y git build-essential cmake python3 python3-venv python3-pip

Clone and build llama.cpp:

cd /opt
sudo git clone https://github.com/ggerganov/llama.cpp.git
sudo chown -R $USER:$USER /opt/llama.cpp
cd /opt/llama.cpp

cmake -S . -B build
cmake --build build -j "$(nproc)"

Confirm you have the tools you need (names can vary slightly by build, but these are common):

ls -la /opt/llama.cpp/build/bin | egrep "(quantize|convert)" || true

If you don’t see conversion scripts: llama.cpp also ships Python converters under /opt/llama.cpp (for example convert_hf_to_gguf.py). Filenames change over time.

List what you actually have:

ls -la /opt/llama.cpp | egrep "convert.*gguf|gguf"

Step 4: Get a model source (HF weights) or start from an existing GGUF

You have two realistic options:

1. Start from Hugging Face (FP16/BF16) and quantize yourself. More control, more disk and CPU time.
2. Start from a GGUF that already exists. Faster, but you’re accepting someone else’s conversion and quant choices.

This tutorial uses the self-quantization path. It’s the most predictable way to hit a specific RAM target on a VPS.

Create a working directory:

mkdir -p ~/models/hf ~/models/gguf
cd ~/models

Download model weights: The simplest method is the Hugging Face CLI, though git lfs also works. The repository name depends on the model you choose.

python3 -m venv ~/venvs/hf
source ~/venvs/hf/bin/activate
pip install -U pip
pip install -U "huggingface_hub[cli]"

# Log in if the model requires it
# huggingface-cli login

# Example pattern (replace with the model you have rights to)
# huggingface-cli download <org>/<model> --local-dir ~/models/hf/<model>

Plan for disk pressure: FP16 downloads are large. On smaller VPS plans, you’ll often run out of disk before RAM.

If you only have ~40 GB total, do the conversion on a larger machine. Then copy the final GGUF to your VPS.

Step 5: Convert HF weights to GGUF

Conversion produces an unquantized GGUF (often f16). After that, you quantize that file to Q4/Q5/Q8.

The converter script name varies by llama.cpp version. Use whatever you see in /opt/llama.cpp.

Typical flow looks like this:

cd /opt/llama.cpp

# Find the converter script
ls -1 | grep -E "convert.*gguf"

Then run conversion (example using a common script name):

# Example ONLY — adjust script name, model path, and output name
python3 ./convert_hf_to_gguf.py \
  ~/models/hf/MY_MODEL \
  --outfile ~/models/gguf/MY_MODEL-f16.gguf \
  --outtype f16

If conversion fails with Python module errors: run it in a clean venv and install the usual requirements:

python3 -m venv ~/venvs/llamacpp
source ~/venvs/llamacpp/bin/activate
pip install -U pip
pip install -U numpy sentencepiece transformers safetensors

Space check before you start:

df -h ~/models /tmp

Step 6: Quantize the GGUF to Q4, Q5, and/or Q8

Now produce one or more quantized GGUF files. On a VPS, build at least two variants.

Make one that almost certainly fits (Q4). Make a second that should look better (Q5 or Q8).

This gives you a clean comparison. It also keeps you from guessing when something fails.

First, locate the quantize binary:

ls -la /opt/llama.cpp/build/bin | grep -i quant

Then quantize:

cd ~/models/gguf

# Q4
/opt/llama.cpp/build/bin/llama-quantize \
  ./MY_MODEL-f16.gguf \
  ./MY_MODEL-q4.gguf \
  Q4_K_M

# Q5
/opt/llama.cpp/build/bin/llama-quantize \
  ./MY_MODEL-f16.gguf \
  ./MY_MODEL-q5.gguf \
  Q5_K_M

# Q8
/opt/llama.cpp/build/bin/llama-quantize \
  ./MY_MODEL-f16.gguf \
  ./MY_MODEL-q8.gguf \
  Q8_0

Why these quant names? You’ll see multiple Q4/Q5 variants. The “K” family is a common starting point because it often balances quality and speed.

Don’t over-optimize on day one. Your first goal is simpler: “fits in RAM and behaves predictably.”

Sanity check file sizes:

ls -lh *.gguf

Pitfall: if your quantized file is only a few megabytes, stop. That usually means the input GGUF is wrong or the conversion failed silently.

Step 7: Import the quantized GGUF into Ollama

Ollama creates a model from a local GGUF using a Modelfile. Put the GGUF somewhere stable. Then reference that path from the Modelfile.

Create a directory for your Ollama custom models:

sudo mkdir -p /opt/ollama-gguf
sudo chown -R $USER:$USER /opt/ollama-gguf

cp ~/models/gguf/MY_MODEL-q4.gguf /opt/ollama-gguf/
cp ~/models/gguf/MY_MODEL-q5.gguf /opt/ollama-gguf/

Create a Modelfile for Q4 (repeat for Q5):

mkdir -p ~/ollama-modelfiles
cat > ~/ollama-modelfiles/my-model-q4.Modelfile << 'EOF'
FROM /opt/ollama-gguf/MY_MODEL-q4.gguf

# Keep settings conservative on a VPS.
PARAMETER temperature 0.2
PARAMETER top_p 0.9

# Set a realistic context for your RAM.
# Larger context increases KV cache memory usage.
PARAMETER num_ctx 2048

SYSTEM "You are a concise assistant for server operations. Ask clarifying questions when needed."
EOF

Import it:

ollama create my-model:q4 -f ~/ollama-modelfiles/my-model-q4.Modelfile
ollama list

Test-run:

ollama run my-model:q4 "Write a 5-line Nginx server block that redirects HTTP to HTTPS."

If Ollama errors with a load failure: re-check paths and permissions. On many installs, Ollama runs as a service.

Keeping GGUF files under /opt with readable permissions avoids surprises.

If you haven’t set up Ollama yet (or you want a service-style install you can monitor), follow this Ollama systemd setup tutorial first.

Step 8: Verify “fits in RAM” and doesn’t degrade badly

Now test two things. First, memory behavior under load. Second, output quality for what you actually do (support replies, internal docs, code hints, and so on).

Memory check (watch for slow death by KV cache)

Run a longer prompt and watch memory in another terminal. Keep the test boring and repeatable.

That repeatability is what makes comparisons meaningful.

# Terminal 1: watch memory
watch -n 0.5 "free -h; echo; ps -o pid,cmd,%mem,rss --sort=-rss | head -n 8"

# Terminal 2: run a longer request
ollama run my-model:q4 "Summarize the steps to rotate Nginx logs with logrotate, including a sample config."

What you’re looking for:

• No OOM kill (dmesg -T | tail should not show the kernel killing Ollama).
• RSS stabilizes after load. Some growth during generation is normal.
• Latency stays steady. If the box starts swapping, you’ll notice immediately.

If you do hit OOM, don’t guess. Pull logs and confirm what happened:

sudo journalctl -u ollama --no-pager -n 200
dmesg -T | tail -n 80

Quality check (a tiny regression test you can reuse)

Pick five prompts that match your real usage. Run them against Q4 and Q5.

You’re looking for regressions you can’t tolerate, not perfect benchmark scores.

Here’s a starter set geared toward operators:

• “Explain why a VPS might return 502 from Nginx when the upstream is healthy.”
• “Given this error log line, list the top three causes and the first command to run for each.”
• “Write a systemd service for a Node app at /srv/app with environment file /etc/app.env.”
• “Turn these bullets into a customer-facing status update under 120 words.”
• “Create a bash one-liner to find the top 20 largest files under /var.”

Run them like this:

ollama run my-model:q4 "Write a systemd service for a Node app at /srv/app with environment file /etc/app.env."
ollama run my-model:q5 "Write a systemd service for a Node app at /srv/app with environment file /etc/app.env."

How to interpret the results: if Q4 is “mostly right but messy,” and Q5 cleans it up without pushing you into OOM, use Q5.

If Q5 falls over on longer prompts, stick with Q4 and reduce context (num_ctx).

Step 9: Tune context length to stop hidden memory spikes

On RAM-limited servers, context length quietly burns your budget. Even if the weights fit, a larger context increases KV cache usage.

That increase can push you into OOM.

Two practical moves:

1. Lower num_ctx in the Modelfile (start at 2048; drop to 1536 or 1024 if needed).
2. Keep prompts tighter and summarize conversation history instead of pasting it back verbatim.

Edit your Modelfile and recreate the model:

sed -i 's/PARAMETER num_ctx 2048/PARAMETER num_ctx 1024/' ~/ollama-modelfiles/my-model-q4.Modelfile
ollama create my-model:q4 -f ~/ollama-modelfiles/my-model-q4.Modelfile

Pitfall: recreating a model with the same tag replaces it. That’s fine, but do it on purpose.

Many operators use tags like my-model:q4-ctx1024 to keep variants straight.

Step 10: Keep the endpoint private (or at least not wide open)

Many people intend to run Ollama “internally,” then accidentally expose it to the public internet. That leads to abuse, noisy logs, and surprise bills.

Minimum baseline:

• Bind Ollama to localhost, then reverse proxy with auth if needed.
• Allow inbound only from your office IP/VPN, or use a tunnel.

If you need remote access to an AI UI without exposing ports, pair this with a tunnel. HostMyCode has a practical guide to Cloudflare Tunnel for private admin access, which works well for AI dashboards too.

Quick firewall baseline (UFW):

sudo ufw status
sudo ufw allow OpenSSH
sudo ufw enable

Keep Ollama on localhost and put Nginx in front if you must expose it. If you want a full, UI-based stack, Open WebUI + Ollama behind a reverse proxy is a solid combo; see Run Open WebUI with Ollama behind Nginx.

FAQ: quantize models for Ollama

Do I need to quantize if I already have a GGUF?

Not always. If the GGUF you have already fits and performs well, use it.

Quantizing it yourself matters when you’re targeting a specific memory ceiling (for example, “must run on 8 GB RAM”) or you want consistent conversion settings across models.

Why does the model load but then crash on longer chats?

Weights fitting isn’t the whole story. As prompts get longer, KV cache grows and memory spikes.

Lower num_ctx, shorten prompts, or drop from Q5/Q8 down to Q4.

Is swap a good fix for Ollama OOM on a VPS?

Swap can prevent immediate crashes, but it usually makes responses painfully slow. Treat swap as a safety net.

The real fix is a smaller quant, a lower context length, or more RAM.

Should I run Q8 on CPU for best quality?

If you have real RAM headroom, Q8 can improve output. On many VPS plans, Q8 becomes unstable once context grows.

Start with Q5, and move to Q8 only after you’ve confirmed stable memory usage under realistic prompts.

Summary: a repeatable VPS workflow

On a VPS, the “best” model is the one that stays up under load. Convert HF weights to GGUF, quantize to Q4/Q5, import using an Ollama Modelfile, then test with real prompts while watching memory.

If you still hit limits after sensible tuning, add RAM or run inference on a larger box.

For predictable performance, run Ollama on a plan with comfortable memory headroom. A HostMyCode VPS works well for CPU-based models, and managed VPS hosting helps if you want the platform maintained while you focus on your application.