Most “runbooks” fail for one boring reason: nobody tries them until everything’s on fire. Linux VPS incident response automation isn’t about fancy tooling. It’s about removing friction so you can grab the right evidence, cut off the obvious access paths, and keep the service alive while you work out what happened.
This is a practical playbook for developers and sysadmins running small-to-mid production workloads on a VPS: a SaaS API, an internal tool, a blog that suddenly attracts the wrong attention, or a worker that starts pinning CPU. You’ll get a repeatable structure, a couple of scripts that are safe to leave on the box, and sensible defaults for 2026.
What you’re trying to achieve (and what “automation” means here)
During an incident, you’re balancing three goals that naturally fight each other:
- Containment: stop the bleeding (credential misuse, data exfil, resource exhaustion).
- Continuity: keep critical endpoints alive, even degraded.
- Forensics: preserve evidence without corrupting it.
Automation here means pre-staging small, auditable actions you can trigger fast: capture system state, package key logs, freeze an account, or throw a temporary network “circuit breaker” while you investigate.
If your baseline hardening is weak, every incident costs more. Pair this runbook with your usual checklist and audits; HostMyCode has a solid starting point in Linux VPS hardening checklist in 2026 and Linux VPS security auditing in 2026.
Scenario: your Node.js API is still up, but you suspect credential abuse
Anchor this in a real situation. You run a Node.js API behind Nginx on a Debian 12 VPS. You notice a spike in 401/403s, then a burst of successful logins from IPs you don’t recognize. Load looks normal, but the pattern screams “leaked token” or “compromised SSH key.”
This is a common failure mode for teams that ship fast and tighten controls later. Your goal isn’t lab-grade forensics. It’s to preserve enough truth to explain the incident, recover safely, and close the hole.
Prerequisites you should have before an incident
You can use this mid-incident, but you’ll move a lot faster if you already have:
- Console access (provider panel, rescue console, or out-of-band). Don’t rely on SSH alone.
- Time sync via chrony or systemd-timesyncd (timestamps matter).
- Centralized logs if possible. Even a lightweight setup helps. If you’re shipping logs, see VPS log shipping with Loki.
- Backups and/or snapshots you’ve verified. For a pragmatic approach, see VPS snapshot backup automation.
Incidents feel very different when your hosting lets you scale resources, take snapshots, and redeploy cleanly. A HostMyCode VPS fits this workflow well because you can treat the server as disposable while keeping data recoverable.
Build a minimal incident kit on the VPS (small, boring, dependable)
Your “incident kit” can be simple: one directory with scripts you trust, plus a place to stash captured artifacts. Keep it readable. Avoid big frameworks that nobody will audit at 2 a.m.
Create a directory and lock down permissions:
sudo install -d -m 0700 /root/ir
sudo install -d -m 0700 /root/ir/artifactsAdd a one-file “state capture” script. Create /root/ir/capture-state.sh:
#!/usr/bin/env bash
set -euo pipefail
TS="$(date -u +%Y%m%dT%H%M%SZ)"
OUT="/root/ir/artifacts/state-${TS}.txt"
{
echo "# Incident state capture (UTC)";
echo "timestamp_utc=${TS}";
echo
echo "## Host";
hostnamectl || true
echo
echo "## Kernel / uptime";
uname -a
uptime
echo
echo "## Time sync";
timedatectl || true
echo
echo "## Users logged in";
who -a || true
echo
echo "## Recent auth";
journalctl -u ssh --since "-6 hours" --no-pager || true
echo
echo "## Processes (top offenders)";
ps -eo pid,ppid,user,cmd,%cpu,%mem --sort=-%cpu | head -n 25
echo
echo "## Listening ports";
ss -lntup
echo
echo "## Network connections (top)";
ss -tnp | head -n 80
echo
echo "## Disk";
df -h
echo
echo "## Recent system errors";
journalctl -p 0..3 --since "-2 hours" --no-pager || true
} | tee "$OUT"
echo "Wrote $OUT"Make it executable:
sudo chmod 0700 /root/ir/capture-state.shExpected output: the script prints a short report and ends with something like Wrote /root/ir/artifacts/state-20260417T114455Z.txt.
Linux VPS incident response automation: containment actions you can trigger safely
This is where many incident responses go sideways. Containment isn’t “reboot and hope.” It’s applying reversible controls that reduce damage without bulldozing evidence.
1) Capture state first (before you change anything)
Run:
sudo /root/ir/capture-state.shIf you’re forced to choose between speed and perfect data, choose speed—but capture something. This single file often answers “what changed?” later.
2) Put SSH into a “safe mode” without locking yourself out
If you suspect SSH compromise, the fastest way to make things worse is editing /etc/ssh/sshd_config live over the same SSH session.
Instead, use a reversible firewall rule that temporarily limits SSH to your current admin IP (replace 203.0.113.10 with your IP):
sudo nft add rule inet filter input tcp dport 22 ip saddr 203.0.113.10 acceptThen add a drop rule for other SSH traffic (place it after any allow rules you rely on):
sudo nft add rule inet filter input tcp dport 22 dropVerification:
sudo nft list ruleset | sed -n '1,160p'If you want a cleaner, auditable setup with rate limits and rollback patterns, keep this guide handy: VPS firewall logging with nftables.
3) Freeze the suspected account, don’t delete it
Deleting accounts destroys evidence and usually creates new problems. If you suspect a local user is compromised (say deploy):
sudo passwd -l deploy
sudo usermod -s /usr/sbin/nologin deployVerification:
sudo getent passwd deployExpected: the shell is /usr/sbin/nologin and the password hash is locked.
4) Rotate exposed application secrets, but keep the old ones temporarily
If the incident involves leaked API tokens, you want rotation without a long outage. The practical pattern is dual validation: accept two keys (current + previous) for a short window so you can roll forward safely.
If you manage secrets on-host, don’t scatter them across random .env files. Encrypt them and commit the encrypted file. In 2026, sops + age is a workable approach; HostMyCode has a hands-on piece: Linux VPS secrets management with sops + age.
Turn the runbook into repeatable automation (without turning it into a platform)
You’re aiming for a small set of commands you can run under pressure. The structure below works well for solo operators and small SRE teams because it stays obvious and easy to extend.
Create a single “incident command” wrapper
Make /root/ir/ir.sh:
#!/usr/bin/env bash
set -euo pipefail
CMD="${1:-}"
case "$CMD" in
capture)
exec /root/ir/capture-state.sh
;;
auth-last-hour)
journalctl -u ssh --since "-1 hour" --no-pager
;;
nginx-5xx)
awk '$9 ~ /^5/ {print}' /var/log/nginx/access.log | tail -n 50
;;
top-tcp)
ss -tn state established | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head
;;
help|--help|-h|"")
echo "Usage: ir.sh {capture|auth-last-hour|nginx-5xx|top-tcp}"
;;
*)
echo "Unknown command: $CMD" >&2
exit 2
;;
esacsudo chmod 0700 /root/ir/ir.shThis is intentionally small. Add commands as you learn what you actually need, but don’t let it turn into a 500-line mystery script.
Verification: quick checks that tell you if you’re stable
Once you’ve contained the blast radius, you need to answer one question: is the server safe enough to keep serving traffic right now?
Service health checks you can run from the box
Check Nginx and your app service status:
sudo systemctl status nginx --no-pager
sudo systemctl status api.service --no-pagerIf you don’t have an app service yet, put it on the roadmap. A unit file gives you restart policy, logs in journald, and a predictable place to hang health checks.
Validate the local HTTP path without DNS in the loop:
curl -fsS http://127.0.0.1:8081/healthzExpected output: something short like ok and an exit code 0. If it fails, you’ll get a non-zero exit code and a visible error.
Security sanity checks (fast, not exhaustive)
sudo last -a | head
sudo journalctl -u ssh --since "-2 hours" --no-pager | tail -n 80
sudo ss -lntupLook for logins you can’t explain and ports you never intended to expose (for example, a database suddenly listening on 0.0.0.0:5432).
Common pitfalls that waste time during incidents
- Locking yourself out with firewall changes. Add allow rules first, and keep a provider console path ready.
- Rebooting early. Reboots wipe volatile evidence (active connections, process trees) and often slow recovery.
- Editing configs without a rollback plan. Use
cp -abackups or a.d/include file you can remove cleanly. - Assuming logs are complete. Disk pressure, aggressive rotation, or misconfigured journald retention can silently drop data.
- Rotating secrets without invalidating sessions. If you rotate tokens but keep refresh tokens valid, attackers may continue.
If disk pressure is part of the incident (log storms happen), keep this troubleshooting guide around: VPS disk space troubleshooting.
Rollback: how to back out of containment safely
Containment is usually temporary. It’s a guardrail while you confirm root cause and clean up persistence.
Rollback nftables emergency rules
If you added ad-hoc rules, remove them explicitly (list handles first):
sudo nft -a list chain inet filter input | sed -n '1,220p'You’ll see rules with handle numbers. Delete by handle:
sudo nft delete rule inet filter input handle 42Verification: re-run the nft -a list and confirm the rule is gone, then test SSH access from an allowed network.
Rollback account freezes
Only unfreeze after you’ve rotated keys and reviewed authorized access.
sudo usermod -s /bin/bash deploy
sudo passwd -u deployIf you use SSH keys (recommended), you may not need to unlock the password. Consider leaving password auth disabled entirely.
Rollback application-level mitigations
If you added temporary “dual key” token validation, set a deadline and remove the old key on schedule. Leaving both keys live for weeks is how one incident becomes a recurring problem.
Where HostMyCode fits in your incident workflow
Doing incident response on aging shared infrastructure is fighting with one hand tied. A VPS gives you the controls your runbook assumes: clean systemd units, firewall policy you can reason about, snapshots, and predictable performance.
For production workloads that need consistent admin access and better guardrails, start with a HostMyCode VPS. If you want patching, monitoring, and baseline hardening handled with you, managed VPS hosting is the calmer option—especially for small teams.
If you’re standardizing how you respond to incidents, run it on infrastructure that supports snapshots, predictable networking, and clean Linux defaults. HostMyCode’s VPS plans work well for hands-on teams, and managed VPS hosting is a good fit when you want help maintaining a stable baseline while you focus on the app.
FAQ: practical questions that come up mid-incident
Should you take a snapshot before making changes?
Yes, if it won’t materially delay containment. Snapshotting gives you a restore point and preserves evidence. If the attacker is still active, contain first, snapshot second.
Is it safe to run security scanners during an incident?
Lightweight checks (like listing listening ports or reviewing auth logs) are fine. Full vulnerability scans can spike load and add noise; schedule them after containment unless you suspect a known, actively exploited CVE.
What logs matter most for credential abuse?
SSH logs (journalctl -u ssh), your reverse proxy access logs (Nginx), and application auth logs. If you use JWTs or API keys, also log token ID prefixes (not full secrets) to correlate usage.
How do you know you’re “done” with containment?
You’re done when unauthorized access stops, new credentials are in place, persistence is removed, and you’ve verified the service path (health checks, auth flow, and outbound connections) is stable.
Next steps: make this sustainable
- Test the runbook monthly. Run the capture script, confirm it writes artifacts, and confirm you can undo firewall rules without surprises.
- Add low-noise monitoring. If you don’t already have it, a minimal metrics + alerting setup catches incidents earlier; see Linux VPS monitoring with Prometheus and Grafana.
- Practice one “credentials compromised” drill. Rotate SSH keys, rotate app secrets, and verify sessions invalidate correctly.
- Document your known-good baseline. List intended open ports, systemd units, and where logs live. You’ll thank yourself later.
If you want a hosting environment that supports this style of disciplined ops, start with a HostMyCode VPS and treat your server as replaceable: automate rebuilds, keep backups verified, and keep incident response boring.