Your VPS logs are probably split between journalctl, /var/log/nginx, and a few app files you only remember exist during an incident. That’s fine—until you need a fast answer to something simple: “When did the 502s start, and which upstream was involved?” This guide walks through VPS log shipping with Vector to a single OpenSearch endpoint, with predictable parsing, TLS, and a rollback plan you can actually use.
The goal is practical: keep the VPS lean, avoid heavyweight agents, and ship only what you’ll search later. You’ll run Vector on a Debian 13 VPS and send structured events to OpenSearch (self-hosted or managed). By the end, you’ll have verified ingestion, disk buffering that behaves under backpressure, and a few guardrails so logging doesn’t become the thing that knocks over your server.
Why Vector for centralized logging on a VPS
Vector is a fast, single-binary agent that can parse, buffer, and ship logs without dragging in a large dependency stack. On small VPS plans, the “agent tax” is real: an idle log shipper burning 150–300 MB of RAM hurts. Vector usually stays light if you keep transforms simple and avoid running expensive regex against every line.
- Predictable backpressure: disk buffering keeps memory from ballooning when OpenSearch slows down.
- Good Linux coverage: journald, file tailing, and syslog are all first-class sources.
- Clean operations: one config file, one systemd service, and readable logs when things go wrong.
If you’re tightening up network access at the same time, pair this with this UFW hardening playbook so your OpenSearch endpoint and SSH exposure stay under control.
Scenario, architecture, and what you’ll build
You’ll ship logs from a single VPS running Nginx + your application (any stack) into an OpenSearch cluster. Self-hosted or managed doesn’t change the flow—you’re still sending HTTPS bulk requests into indices you can query.
- Source VPS: Debian 13, Nginx, systemd-journald
- Agent: Vector, running as a systemd service
- Destination: OpenSearch over HTTPS with Basic Auth and TLS verification
- Indices: daily indices like
vps-logs-2026.04.13
Hosting note: log shipping works best when the VPS has steady I/O and you can spare disk for buffers. A small but capable HostMyCode VPS is a good fit because you can size CPU/RAM for the app and still reserve disk space for Vector’s buffer without crowding the root filesystem.
Prerequisites (don’t skip these)
- A VPS with root or sudo access (examples use Debian 13).
- Outbound HTTPS access to your OpenSearch endpoint (port
9200or443, depending on your setup). - An OpenSearch user with permission to write indices (e.g.,
vps-logs-*). - Nginx logs enabled (default paths below), and systemd journald running.
- Basic CLI tools:
curl,jq,openssl.
If you plan to keep logs for a while, decide retention up front. Centralized logging gets expensive when nothing ever ages out. If you’re also planning backups and restore drills, this Restic + S3 backup strategy pairs nicely with OpenSearch snapshots and a versioned Vector config.
Step 1: Confirm what you’re collecting (journald + Nginx)
Start by confirming the logs exist and the formats are stable. You’ll save yourself time later.
Journald
journalctl -n 5 --no-pagerExpected output: a handful of recent system logs with timestamps and unit names.
Nginx access and error logs
sudo ls -lah /var/log/nginx/
sudo tail -n 3 /var/log/nginx/access.log
sudo tail -n 3 /var/log/nginx/error.logIf your Nginx logs live somewhere else, write down the paths. Vector tails exactly what you configure; it won’t guess.
Step 2: Install Vector on Debian 13
Use the official Vector package repository so updates stay boring. The exact repo steps vary by distro; on Debian-based systems you’ll typically add the repo and install the vector package.
sudo apt-get update
sudo apt-get install -y curl gpg ca-certificatesThen install Vector using the official package instructions for your distro. After installation, confirm the binary:
vector --versionExpected output looks like:
vector 0.4x.x (x86_64-unknown-linux-gnu ...)Pinning tip for production: once you’re happy with behavior, hold the package until you schedule a maintenance window:
sudo apt-mark hold vectorStep 3: Create a dedicated directory for buffers and state
Disk buffering keeps you from dropping logs when OpenSearch slows down or disappears for a bit. Put buffers on persistent storage and leave headroom.
sudo install -d -o vector -g vector -m 0750 /var/lib/vector
sudo install -d -o vector -g vector -m 0750 /var/lib/vector/buffers
sudo install -d -o vector -g vector -m 0750 /var/lib/vector/stateQuick sanity check:
sudo -u vector test -w /var/lib/vector/buffersStep 4: Prepare OpenSearch endpoint, credentials, and TLS
Before you touch Vector config, prove the VPS can reach OpenSearch. Most “Vector is broken” reports turn out to be networking, auth, or TLS.
Connectivity test
export OPENSEARCH_URL="https://opensearch.example.net:9200"
export OPENSEARCH_USER="vps_shipper"
export OPENSEARCH_PASS="REPLACE_ME"
curl -sS -u "$OPENSEARCH_USER:$OPENSEARCH_PASS" \
"$OPENSEARCH_URL" | jq .Expected output: a JSON object with cluster name and version. TLS errors usually mean you’re missing the right CA chain. A 401 means bad credentials. Timeouts mean you should fix routing and firewall rules before you do anything else.
If you’re weighing self-hosted vs managed, be honest about the operational load. OpenSearch needs real care (heap sizing, shard strategy, snapshots). Many teams are better off keeping the VPS as a shipper and running search elsewhere. A common setup is to run OpenSearch on a separate managed VPS hosting instance and keep your app nodes focused on serving traffic.
Step 5: Build a production-lean Vector config (journald + Nginx → OpenSearch)
Vector’s main config is typically /etc/vector/vector.yaml. This version stays readable and intentionally avoids “parse everything perfectly” on day one. You want useful fields and reliable ingestion first.
Create a backup first:
sudo cp -a /etc/vector/vector.yaml /etc/vector/vector.yaml.bak.$(date +%F)Now edit /etc/vector/vector.yaml:
sudo nano /etc/vector/vector.yamldata_dir: /var/lib/vector
sources:
journald_all:
type: journald
since_now: true
nginx_access:
type: file
include:
- /var/log/nginx/access.log
ignore_older_secs: 86400
read_from: beginning
nginx_error:
type: file
include:
- /var/log/nginx/error.log
ignore_older_secs: 86400
read_from: beginning
transforms:
add_host_and_tags:
type: remap
inputs:
- journald_all
- nginx_access
- nginx_error
source: |
.host = get_hostname!()
.env = "prod"
.service = "edge-api"
# Normalize timestamp if present; Vector will set one anyway.
if exists(.timestamp) {
.@timestamp = to_timestamp!(.timestamp)
}
parse_nginx_access:
type: remap
inputs:
- add_host_and_tags
source: |
# Only parse lines that look like Nginx access logs.
# If you use a custom log_format, adjust this.
if is_string(.message) && contains!(.message, "HTTP/") {
parsed, err = parse_regex(.message, r'^(?P<remote_addr>\S+) \S+ \S+ \[(?P<time_local>[^\]]+)\] "(?P<method>\S+) (?P<request_uri>\S+) (?P<server_protocol>\S+)" (?P<status>\d{3}) (?P<body_bytes_sent>\d+) "(?P<http_referer>[^"]*)" "(?P<http_user_agent>[^"]*)"')
if err == null {
.nginx = parsed
.nginx.status = to_int!(.nginx.status)
.nginx.body_bytes_sent = to_int!(.nginx.body_bytes_sent)
}
}
sinks:
opensearch_logs:
type: elasticsearch
inputs:
- parse_nginx_access
endpoints:
- "${OPENSEARCH_URL}"
bulk:
index: "vps-logs-%Y.%m.%d"
auth:
strategy: basic
user: "${OPENSEARCH_USER}"
password: "${OPENSEARCH_PASS}"
tls:
verify_certificate: true
verify_hostname: true
request:
timeout_secs: 30
buffer:
type: disk
max_size: 4294967296
when_full: block
compression: gzip
What this config does:
- Reads everything from journald (starting now), plus Nginx access/error log files.
- Adds consistent fields (
host,env,service). - Parses typical Nginx access logs into structured fields when the line matches.
- Sends all events to OpenSearch daily indices, with disk buffering and gzip compression.
Security note: environment variables are fine for many deployments, but treat them as secrets. Use a systemd drop-in with restricted permissions, and scope your OpenSearch user to the index pattern only.
Step 6: Add secrets via systemd environment (cleaner than hardcoding)
Instead of embedding secrets directly in vector.yaml, pass them to the service as environment variables.
Create a systemd override:
sudo systemctl edit vectorPaste:
[Service]
Environment="OPENSEARCH_URL=https://opensearch.example.net:9200"
Environment="OPENSEARCH_USER=vps_shipper"
Environment="OPENSEARCH_PASS=REPLACE_ME"Keep these values private. Root can read the unit drop-in by default; don’t loosen permissions on /etc/systemd.
Reload systemd:
sudo systemctl daemon-reloadStep 7: Validate the config before you restart anything
Vector can validate config syntax. Make this a habit.
sudo vector validate /etc/vector/vector.yamlExpected output: Validated (or no errors). If you hit VRL errors, fix them now instead of debugging a live service.
Step 8: Start Vector and watch the first ingest
sudo systemctl enable --now vector
sudo systemctl status vector --no-pagerThen tail Vector’s own logs:
sudo journalctl -u vector -f --no-pagerExpected output: Vector starts the sources and ships batches. Bad credentials show up as HTTP 401/403 errors. TLS problems surface immediately as certificate or hostname verification errors.
Step 9: Verify data landed in OpenSearch (real queries, not hope)
From the VPS (or your workstation), confirm indices exist and that you can query documents back.
List today’s index
curl -sS -u "$OPENSEARCH_USER:$OPENSEARCH_PASS" \
"$OPENSEARCH_URL/_cat/indices?v" | grep vps-logsSearch last 5 events from this host
HOSTNAME=$(hostname)
curl -sS -u "$OPENSEARCH_USER:$OPENSEARCH_PASS" \
-H 'Content-Type: application/json' \
"$OPENSEARCH_URL/vps-logs-*/_search" \
-d "{
\"size\": 5,
\"sort\": [{\"@timestamp\": \"desc\"}],
\"query\": {\"term\": {\"host.keyword\": \"$HOSTNAME\"}}
}" | jq '.hits.hits[0]._source | {"@timestamp": ."@timestamp", host: .host, service: .service, message: .message, nginx: .nginx}'Expected output: a JSON object with @timestamp, host, and message. If you see nginx populated for access lines, your parsing is doing its job.
Operational checklist (what to set before this grows)
Once logs are flowing, a few defaults prevent slow-motion chaos.
- Index lifecycle / retention: delete old indices after 7–30 days unless you have compliance needs.
- Shard sizing: too many tiny shards will hurt performance. Start with low shard counts.
- Mapping control: if you ship unpredictable JSON, you can blow up mappings. Be selective.
- Disk alerts: watch both OpenSearch disk and Vector buffer disk.
If you want better visibility into Linux internals, this eBPF monitoring playbook pairs well with centralized logs: logs show symptoms; tracing helps you pin down the cause.
Common pitfalls (and how to spot them fast)
1) You shipped logs, but timestamps look wrong
Symptom: searches look “empty” because events landed with timestamps far in the future or past. Start by checking whether @timestamp is being set consistently.
- Confirm server time:
timedatectl - Prefer Vector’s ingestion timestamp if your source timestamps aren’t reliable.
2) OpenSearch starts rejecting writes with 413 or bulk errors
Bulk requests can exceed proxy limits (Nginx/ALB) or OpenSearch settings. If you front OpenSearch with Nginx, check client_max_body_size.
Vector-side quick mitigation: reduce batch sizes (consult Vector’s sink bulk settings for your version) and keep gzip enabled.
3) CPU spikes due to regex parsing
Regex on every line adds up quickly. Parse only what needs parsing, and keep patterns tight. If your access logs are already JSON, don’t regex them—parse JSON.
4) Disk fills because buffer never drains
Disk buffering is a safety net, but it can mask a broken destination until the buffer is full.
- Watch Vector errors:
journalctl -u vector --since "10 min ago" - Check buffer directory size:
sudo du -sh /var/lib/vector/buffers - Fix the destination (auth/TLS/route) before you increase buffer size.
5) You can’t find Nginx fields in OpenSearch
Usually it’s one of two things: your Nginx log_format doesn’t match the regex, or your Nginx logs aren’t actually coming from files (some setups send them to journald).
If you run multiple backends and want cleaner analysis, your proxy layout matters. This older-but-still-relevant technique—routing multiple apps by Nginx URL paths—benefits from structured access logs because you can filter and aggregate by path prefix.
Rollback plan (keep it boring and reversible)
If log shipping causes trouble—CPU spikes, disk pressure, or unexpected OpenSearch costs—back out cleanly.
- Stop Vector:
sudo systemctl stop vector - Restore the previous config:
sudo cp -a /etc/vector/vector.yaml.bak.* /etc/vector/vector.yaml - Disable the service if you’re pausing the project:
sudo systemctl disable vector - Optionally clear buffers (only if you accept losing queued logs):
sudo rm -rf /var/lib/vector/buffers/*
After rollback, confirm the VPS is calm: top, df -h, and journalctl -u vector should show no ongoing load or repeating errors.
Next steps (small upgrades that pay off)
- Switch Nginx access logs to JSON and parse JSON instead of regex. It’s faster and less fragile.
- Add rate-limit alerts: create an OpenSearch query/alert for spikes in
nginx.status: 499,502, and504. - Ship application logs: add a file source for
/var/log/edge-api/app.logor your container logs. - Separate “system” and “app” indices if mappings start conflicting.
- Run a restore drill for OpenSearch snapshots and keep your Vector config in Git.
If you want centralized logging without turning your app VPS into a long-running experiment, start with a right-sized HostMyCode VPS and leave disk headroom for safe buffering. If you don’t want to manage patching and baseline hardening as the setup grows, managed VPS hosting keeps the server layer maintained while you focus on logs, searches, and incident response.
FAQ: VPS log shipping with Vector
Do I need to ship every journald unit?
No. Start broad for a week, then tighten it. The easiest way to control volume is to drop noisy units (like package updates) or ship only the units you care about (Nginx, your app service, SSH).
Should I parse Nginx logs with regex or JSON?
JSON is usually the better choice in 2026: lower CPU cost, fewer brittle patterns, and cleaner fields in OpenSearch. Regex works as a bridge if you can’t change log_format yet.
How much disk buffer should Vector have?
Size it for your expected outage window. On small production VPS setups, 2–8 GB is a reasonable starting range. The right number depends on log volume and how quickly you need logs searchable.
Is it safe to store OpenSearch credentials in systemd environment variables?
It’s acceptable if you keep permissions tight and don’t hand out root access. For higher-security environments, use a secret manager, short-lived credentials, or mTLS with client certs.
What if my OpenSearch endpoint is private and the VPS can’t reach it?
Fix connectivity first: routing, firewall rules, and DNS. If the endpoint is only reachable inside a private network, you’ll likely need a VPN or a dedicated network segment before log shipping will be reliable.
Summary
Centralized logs only help during incidents if ingestion is steady and the agent behaves under stress. With VPS log shipping with Vector, you get disk buffering, straightforward transforms, and a systemd-managed service you can validate and roll back without drama.
If you’re rolling this out across multiple nodes, standardize early. Build a base image, deploy Vector consistently, and tag each node. A simple starting point is a fleet of HostMyCode VPS instances with the same Vector config and per-node tags, so cross-host searches don’t turn into detective work.