Self-Host a PrivateGPT RAG Chatbot With Qdrant

A “chat with your docs” bot only stays private if you run the full pipeline yourself. That includes the UI, the model endpoint, and the vector database that stores embeddings.

This walkthrough shows you how to self-host PrivateGPT with Qdrant on a VPS using Docker Compose, persistent volumes, and an HTTPS reverse proxy with Nginx.

By the end, you’ll have a single hostname (for example, chatdocs.example.com) serving the PrivateGPT web UI over TLS.

Documents stay on disk. Embeddings live in Qdrant. Nothing needs to leave your server unless you explicitly configure it to.

What you’re building (and what it is not)

This stack fits teams that want a private assistant for internal docs.

Typical inputs include SOPs, support macros, policy PDFs, product manuals, or codebase notes.

• PrivateGPT: the web app/API that ingests documents, chunks them, creates embeddings, and runs retrieval-augmented generation (RAG).
• Qdrant: the vector database that stores embeddings and performs similarity search.
• Local LLM + embeddings: you can run these in different ways. Here we’ll use a local model server via Ollama because it’s straightforward to operate on a VPS.

What this is not: a high-concurrency, multi-tenant AI platform.

If you expect dozens of simultaneous users, split the model server onto a dedicated GPU machine.

Keep the web UI + Qdrant on a smaller VPS.

Infrastructure sizing for a PrivateGPT + Qdrant VPS

RAG setups can feel lightweight—until you ingest a few hundred documents.

Prioritize disk and RAM first. Then pick CPU/GPU based on your response-time targets.

• Minimum (small team, light docs): 4 vCPU, 8 GB RAM, 80–120 GB NVMe. Works for a small model and occasional queries.
• Recommended (most teams): 6–8 vCPU, 16 GB RAM, 160+ GB NVMe. Comfortable for indexing, background ingestion, and faster search.
• GPU (optional): Not required for RAG itself, but helps LLM response time. If you want smooth chat UX with larger models, use a GPU VPS or a dedicated server with an NVIDIA GPU.

Qdrant is efficient, but embeddings add up.

As a practical operator rule, text embeddings often land at a few KB per chunk once you include payload metadata.

If you chunk aggressively (very small chunks), your storage footprint grows fast.

If you’re starting from scratch, a HostMyCode VPS with NVMe storage is a good fit for this deployment.

If you’d rather not maintain OS hardening and the web stack yourself, managed VPS hosting is the lowest-friction route.

Prerequisites

• Ubuntu 22.04/24.04 LTS or Debian 12 on a VPS (steps assume Ubuntu/Debian-style paths and packages)
• A domain/subdomain pointing to your server (A/AAAA record), e.g. chatdocs.example.com
• Docker + Docker Compose plugin
• Nginx installed on the host (reverse proxy + TLS termination)

Step 1: Prepare the server (DNS, firewall, packages)

Start by confirming DNS points to your VPS IP.

If you’re switching DNS providers or cleaning up records, follow this DNS migration guide so you don’t get stuck waiting on old TTLs.

On the server, install the required packages:

sudo apt update
sudo apt install -y ca-certificates curl gnupg nginx

At the network edge, expose only what you need.

If you use UFW:

sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status

Keep Qdrant and the PrivateGPT container ports private (loopback/Docker network only).

Only Nginx should be reachable from the public internet.

Step 2: Install Docker and the Compose plugin

If Docker isn’t installed, use the official packages for your distro.

On Ubuntu, that typically means Docker’s apt repo.

If Docker is already set up, skip ahead.

Verify:

docker --version
docker compose version

Optionally allow your user to run Docker without sudo:

sudo usermod -aG docker $USER
newgrp docker

Step 3: Create directories for persistent storage

The fastest way to lose a RAG index is to store it inside a container filesystem.

Keep your data on the host under /opt/privategpt.

sudo mkdir -p /opt/privategpt/{qdrant_data,uploads,ollama}
sudo chown -R $USER:$USER /opt/privategpt

• qdrant_data holds the vector collections
• uploads holds ingested source documents (or your mounted doc folder)
• ollama holds downloaded model blobs

Step 4: Write a Docker Compose file (PrivateGPT + Qdrant + Ollama)

Create /opt/privategpt/compose.yml:

cd /opt/privategpt
nano compose.yml

Paste the following.

The services share an internal Docker network.

The web UI binds only to 127.0.0.1 so Nginx can proxy it.

services:
  qdrant:
    image: qdrant/qdrant:latest
    restart: unless-stopped
    volumes:
      - ./qdrant_data:/qdrant/storage
    networks:
      - private

  ollama:
    image: ollama/ollama:latest
    restart: unless-stopped
    volumes:
      - ./ollama:/root/.ollama
    networks:
      - private

  privategpt:
    image: ghcr.io/zylon-ai/privategpt:latest
    restart: unless-stopped
    depends_on:
      - qdrant
      - ollama
    environment:
      # Point PrivateGPT to Qdrant and Ollama on the Docker network
      - QDRANT_URL=http://qdrant:6333
      - OLLAMA_BASE_URL=http://ollama:11434
      # Optional: tune worker threads if your CPU is small
      # - WEB_CONCURRENCY=1
    volumes:
      - ./uploads:/app/uploads
    ports:
      # bind only to localhost; Nginx will be public
      - 127.0.0.1:8080:8001
    networks:
      - private

networks:
  private:
    driver: bridge

Notes you should understand before continuing:

• Images and defaults change. PrivateGPT’s container environment variables and paths can evolve. If the container starts but can’t see Qdrant/Ollama, check the project docs for the current variable names.
• Localhost binding matters. 127.0.0.1:8080:8001 keeps the app from being directly exposed. You’ll publish it through Nginx.

Step 5: Start the stack and pull models

Bring everything up:

cd /opt/privategpt
docker compose up -d
docker compose ps

If anything looks unhealthy, go straight to logs:

docker compose logs -f --tail=200 privategpt

Next, pull at least one chat model and one embedding model into Ollama.

The “right” models depend on your hardware.

On CPU-only VPSes, start small. Scale up after you validate retrieval.

docker compose exec -it ollama ollama pull llama3
docker compose exec -it ollama ollama pull nomic-embed-text

If you’re short on RAM, the chat model may load slowly or fail under load.

Drop to a smaller model, or move the model server to a GPU host (Ollama/vLLM) and keep the rest on your VPS.

If you want a GPU-native OpenAI-compatible API, see our vLLM GPU tutorial.

Step 6: Quick health checks (Qdrant, PrivateGPT)

From the VPS, confirm Qdrant responds on its internal port (no public exposure needed):

docker compose exec -it qdrant wget -qO- http://localhost:6333/

Then confirm PrivateGPT is reachable via localhost:

curl -I http://127.0.0.1:8080/

If you see a 200 or 302, you’re ready to put it behind HTTPS on a real hostname.

Step 7: Add Nginx as an HTTPS reverse proxy

Create an Nginx server block for your subdomain:

sudo nano /etc/nginx/sites-available/chatdocs.example.com

Use this as a starting point.

The upload limit is sized for PDFs and small doc bundles.

The timeouts help prevent large ingestion requests from getting cut off.

server {
    listen 80;
    server_name chatdocs.example.com;

    # Keep HTTP open only for ACME challenge / redirect to HTTPS later
    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        client_max_body_size 50m;
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
    }
}

Enable it and test:

sudo ln -s /etc/nginx/sites-available/chatdocs.example.com /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

If uploads fail with 413 Request Entity Too Large, fix it at the layer that’s actually enforcing the limit (server block vs http block).

This guide covers the common gotchas: Nginx 413 fix on a VPS.

Step 8: Enable TLS (Let’s Encrypt) and enforce HTTPS

Install Certbot for Nginx:

sudo apt install -y certbot python3-certbot-nginx

Request a certificate:

sudo certbot --nginx -d chatdocs.example.com

Select the redirect option so HTTP automatically becomes HTTPS.

Confirm auto-renewal:

sudo certbot renew --dry-run

If you prefer Caddy for automatic HTTPS, follow our Caddy TLS tutorial and proxy to the same 127.0.0.1:8080 backend.

Step 9: Lock down access (basic auth or SSO gateway)

A RAG bot can surface internal policies, contracts, or customer data—whatever you feed it.

Don’t put it on a public URL without authentication.

The quickest option is HTTP basic auth in Nginx.

Create a password file:

sudo apt install -y apache2-utils
sudo htpasswd -c /etc/nginx/.htpasswd admin

Edit your Nginx TLS server block (Certbot likely created/modified it under the same file) and add inside the relevant location /:

auth_basic "Restricted";
auth_basic_user_file /etc/nginx/.htpasswd;

Then reload:

sudo nginx -t
sudo systemctl reload nginx

If you need API keys, rate limits, and tighter controls around AI endpoints, apply the same defensive patterns you’d use for model servers.

Even though this tutorial focuses on PrivateGPT, the hardening approach maps closely to securing an LLM API with Nginx and rate limits.

Step 10: Ingest documents and test retrieval quality

Open https://chatdocs.example.com and log in (if you enabled basic auth).

In the PrivateGPT UI, start with a small, representative set:

• One short policy doc (2–3 pages)
• One longer PDF (20–50 pages)
• A markdown or text SOP

Then ask questions where you already know the answer. Verify citations as you go.

For example:

• “Summarize the refund rules in 5 bullet points and cite the section title.”
• “What’s the escalation path for payment failures?”
• “List the ports our app requires and where that is documented.”

If the model sounds confident but gets details wrong, assume retrieval or grounding is the problem first.

Tighten instructions. Require citations, and explicitly allow “I don’t know” when the context doesn’t contain the answer.

Operational knobs you’ll actually use (RAM, quantization, and chunking)

After the first week, most setups get tuned in the same three places.

• Model size vs RAM: CPU-only inference is usually memory-bound. If your server is swapping, chat will feel “stuck.” Add RAM or switch to a smaller model.
• Quantization trade-offs: Quantized models reduce RAM use and can speed up responses, but they can also lose precision. In RAG, retrieval quality tends to matter more than raw model capability. A smaller model with clean retrieval often beats a larger model with messy retrieval.
• Chunking: Oversized chunks bury relevant passages. Tiny chunks return fragments with no context. Target chunks that hold a complete thought (a paragraph or short section), not a rigid character count.

A practical diagnostic loop is to watch memory and CPU during ingestion and a few chat sessions.

sudo apt install -y htop
htop

If you want continuous visibility, run an agent like Netdata (especially on a dedicated server running multiple services).

HostMyCode has a walkthrough: run Netdata on a VPS.

Backups: what to copy, and how to restore

A restore only works if you have both the vector database state and the source files.

You also need any config you changed.

At minimum, back up:

• /opt/privategpt/qdrant_data
• /opt/privategpt/uploads
• /opt/privategpt/compose.yml (and any env files you add)
• /etc/nginx/sites-available/chatdocs.example.com

A quick “stop the world” backup (fine for small installs) looks like this:

cd /opt/privategpt
docker compose stop
sudo tar -czf /root/privategpt-backup.tgz /opt/privategpt /etc/nginx/sites-available/chatdocs.example.com
docker compose start

For production, schedule offsite backups with retention.

Also run restore tests. Don’t wait for an outage to find out your backup was incomplete.

Common problems and fast fixes

• Uploads fail (413): raise client_max_body_size in the correct Nginx block and reload. See the earlier 413 guide.
• Nginx shows 502: the upstream isn’t reachable. Check curl -I http://127.0.0.1:8080/ and container logs. This tutorial helps isolate the cause: Nginx 502 on Ubuntu VPS.
• Qdrant disk grows quickly: you’re chunking too small, ingesting duplicates, or storing too much payload metadata. Revisit ingestion settings and tighten document hygiene.
• Answers ignore docs: your prompt isn’t enforcing grounding or retrieval isn’t returning relevant context. Test retrieval by asking for a direct quote and where it appears.

Putting the chatbot on a real website (safely)

If your goal is “add an AI assistant to my site,” don’t embed a privileged internal bot into a public page.

Treat this as an internal tool first.

• Run a private PrivateGPT instance for staff documentation.
• If you need a public assistant, build a separate knowledge base with curated content only.
• Use a different Qdrant collection (or a different Qdrant instance) for public vs internal data.

Also think about crawler control.

If you publish AI endpoints or chat pages, limit what gets indexed and scraped.

Use robots.txt where it makes sense, and keep sensitive content behind authentication.

Summary: a private docs chatbot you can operate

You now have a working RAG stack: PrivateGPT for ingestion and chat, Qdrant for similarity search, and an LLM endpoint via Ollama.

The operational foundations are simple but important: persistent storage, private container networking, and TLS at the edge.

If you want predictable performance, start with enough RAM and NVMe.

Then, if usage grows, scale the model runtime separately.

For many teams, moving the model server to a GPU host while keeping Qdrant close to the app is the cleanest next step.

If you’re deploying this for business use, choose infrastructure you can expand without rebuilding later.

A HostMyCode VPS is a solid baseline, and managed VPS hosting helps if you want the platform maintained while you focus on the app and data.

FAQ

Do I need a GPU to self-host PrivateGPT with Qdrant?

No. Qdrant and the RAG pipeline work on CPU.

A GPU mainly improves LLM response time, especially with larger models or more concurrent users.

Should I expose Qdrant to the internet?

Usually, no. Keep Qdrant on a private Docker network and let only your app access it.

If you must expose it, put it behind authentication and strict IP allowlists.

What’s the minimum backup I should keep?

Back up /opt/privategpt/qdrant_data and your uploaded/source documents.

Without both, you’ll lose either the index or the original data needed for rebuilding.

How do I stop the bot from answering outside my documents?

Use prompts that require citations from retrieved context and instruct the model to answer “I don’t know” if context is missing.

Then verify retrieval quality with quote-based tests.