Web Application Firewall Tutorial (2026): Set Up ModSecurity + OWASP CRS on Nginx (Ubuntu VPS)

A network firewall won’t stop SQL injection, malicious file uploads, or a WordPress login being hammered with crafted payloads. That’s the job of a WAF. It sits next to your web server and inspects HTTP requests before they reach PHP or your application.

This web application firewall tutorial walks you through a practical ModSecurity v3 + OWASP Core Rule Set (CRS) setup on an Ubuntu VPS running Nginx. You’ll use a tight tune-and-test loop so you don’t break real users.

If you want someone else to own the full stack—Nginx hardening, WAF tuning, rule updates, and the inevitable “why is checkout failing?” investigation—use managed VPS hosting from HostMyCode. If you prefer hands-on control, a standard HostMyCode VPS is a solid base.

What you’ll build (and what you won’t)

• WAF in front of Nginx using ModSecurity v3 (libmodsecurity) and the Nginx connector.
• OWASP CRS running in detection mode first, then gradually enforced.
• Logging you can actually use: audit log + a repeatable “why was this blocked?” workflow.
• Per-site controls (different paranoia levels, exclusions, and body limits).

This guide doesn’t replace patching, least privilege, backups, or secure coding. Treat the WAF as an extra control.

It catches a lot of noisy, automated abuse.

Prerequisites and a quick readiness checklist

This tutorial assumes:

• Ubuntu 24.04 LTS or Ubuntu 22.04 LTS (both common on hosting VPS in 2026)
• Nginx already serving at least one site
• Root or sudo access
• Disk space for logs (audit logs grow fast if you let them)

Before you add a WAF, make sure the basics won’t bite you mid-change:

• Confirm TLS works and renewals are reliable. If renewals are flaky, fix that first using SSL renewal troubleshooting.
• Keep a second SSH session open. That way you don’t lock yourself out after an Nginx mistake.
• Know where your site logs live: usually /var/log/nginx/access.log and /var/log/nginx/error.log.

Architecture choice: Nginx module vs reverse-proxy WAF

On a VPS in 2026, you’ll usually see one of two patterns:

• ModSecurity integrated with Nginx (what we’re doing): lower latency, fewer moving parts, direct control.
• WAF as a reverse proxy (separate service or container): easier to isolate, but more wiring and more places to debug.

For a single VPS hosting a handful of sites, the integrated approach stays simple. It also performs well, especially if you keep audit logging under control.

Step 1 — Install ModSecurity v3 dependencies on Ubuntu

Update packages and install build dependencies. Ubuntu’s repos don’t always ship a ready-to-use ModSecurity v3 Nginx connector in the exact shape you want.

When you need predictable behavior, building from source is still the reliable route.

sudo apt update
sudo apt -y install git build-essential autoconf automake libtool pkgconf \
  libpcre2-dev libxml2-dev libyajl-dev liblmdb-dev libgeoip-dev \
  libcurl4-openssl-dev liblua5.4-dev libssl-dev zlib1g-dev \
  libmaxminddb-dev

Check your Nginx version and build flags:

nginx -v
nginx -V 2>&1 | tr ' ' '\n' | sed -n '1,120p'

Keep this output handy. You may need it to rebuild Nginx with a compatible dynamic module.

On many VPS installs, dynamic modules work cleanly.

Step 2 — Build and install libmodsecurity (ModSecurity engine)

Compile libmodsecurity from source. This is the inspection engine.

It is not the rule set.

cd /usr/local/src
sudo git clone --depth=1 https://github.com/owasp-modsecurity/ModSecurity.git
cd ModSecurity
sudo git submodule update --init --recursive
sudo ./build.sh
sudo ./configure
sudo make -j"$(nproc)"
sudo make install

Confirm the library is installed:

ldconfig -p | grep -i modsecurity || true

Step 3 — Build the ModSecurity Nginx connector as a dynamic module

Now build the Nginx connector. This module lets Nginx call into libmodsecurity.

Compile it against the same Nginx source version you’re running.

Install Nginx source package metadata and fetch the source:

sudo apt -y install dpkg-dev
apt-cache policy nginx | sed -n '1,120p'
mkdir -p ~/nginx-build && cd ~/nginx-build
apt-get source nginx

Clone the connector:

cd ~/nginx-build
git clone --depth=1 https://github.com/owasp-modsecurity/ModSecurity-nginx.git

Enter the extracted Nginx source directory (it will look like nginx-1.24.x or newer depending on Ubuntu updates). Then compile the module.

Replace the directory name to match what ls shows.

cd ~/nginx-build
ls
cd nginx-*

# Build dynamic module
./configure --with-compat --add-dynamic-module=../ModSecurity-nginx
make modules

Copy the module into Nginx’s modules path. On Ubuntu this is typically /usr/lib/nginx/modules/.

sudo mkdir -p /usr/lib/nginx/modules
sudo cp objs/ngx_http_modsecurity_module.so /usr/lib/nginx/modules/

Load the module by adding this at the top of /etc/nginx/nginx.conf (first line is fine):

load_module modules/ngx_http_modsecurity_module.so;

Test Nginx config syntax before restarting:

sudo nginx -t

Step 4 — Create a baseline ModSecurity config

Create a ModSecurity config directory. Then start from the recommended config shipped in the source tree.

sudo mkdir -p /etc/nginx/modsec
sudo cp /usr/local/src/ModSecurity/modsecurity.conf-recommended /etc/nginx/modsec/modsecurity.conf

Edit it:

sudo nano /etc/nginx/modsec/modsecurity.conf

Make two deliberate choices up front:

• Start in detection mode: SecRuleEngine DetectionOnly
• Turn on audit logging that won’t drown you: use RelevantOnly

Recommended minimal adjustments:

# /etc/nginx/modsec/modsecurity.conf
SecRuleEngine DetectionOnly

# Avoid filling disks during testing
SecAuditEngine RelevantOnly
SecAuditLogParts ABIJDEFHZ
SecAuditLogType Serial
SecAuditLog /var/log/modsecurity/audit.log

# Keep request body inspection on (needed for many CRS rules)
SecRequestBodyAccess On

# Tune later per site if you accept large uploads
SecRequestBodyLimit 13107200
SecRequestBodyNoFilesLimit 1048576
SecRequestBodyInMemoryLimit 131072

# Response body inspection usually off for performance unless you need it
SecResponseBodyAccess Off

Create the log directory and file with safe permissions:

sudo mkdir -p /var/log/modsecurity
sudo touch /var/log/modsecurity/audit.log
sudo chown -R www-data:adm /var/log/modsecurity
sudo chmod 750 /var/log/modsecurity
sudo chmod 640 /var/log/modsecurity/audit.log

Step 5 — Install OWASP Core Rule Set (CRS)

CRS is the curated rule set that flags (and later blocks) common attack classes.

Install it under /etc/nginx/modsec.

cd /etc/nginx/modsec
sudo git clone --depth=1 https://github.com/coreruleset/coreruleset.git
sudo cp coreruleset/crs-setup.conf.example coreruleset/crs-setup.conf

Create an include file that wires everything together:

sudo nano /etc/nginx/modsec/main.conf

# /etc/nginx/modsec/main.conf
Include /etc/nginx/modsec/modsecurity.conf

# OWASP CRS
Include /etc/nginx/modsec/coreruleset/crs-setup.conf
Include /etc/nginx/modsec/coreruleset/rules/*.conf

Step 6 — Enable ModSecurity in Nginx (global or per site)

You can enable the WAF globally in the http {} block, or per server block. On a VPS hosting multiple sites, per-site control is usually safer.

Start with one site. Verify behavior, then expand.

Edit your site config (example: /etc/nginx/sites-available/example.com) and add:

server {
  # ... existing config ...

  modsecurity on;
  modsecurity_rules_file /etc/nginx/modsec/main.conf;

  # ...
}

Reload Nginx:

sudo nginx -t && sudo systemctl reload nginx

Step 7 — Validate with controlled attack-style requests

You’re still in detection mode. Requests should succeed, but you should see rule matches in the audit log.

Run a few tests from your workstation.

1) A basic SQLi-looking string:

curl -i "https://example.com/?q=' OR 1=1 --"

2) A path traversal pattern:

curl -i "https://example.com/?file=../../../../etc/passwd"

3) A suspicious User-Agent:

curl -i -H 'User-Agent: sqlmap' https://example.com/

Now inspect the audit log:

sudo tail -n 80 /var/log/modsecurity/audit.log

If you want to search quickly by rule ID:

sudo grep -R "id \"9" -n /var/log/modsecurity/audit.log | tail -n 20

You’re checking two things:

• Rules load successfully.
• Normal browsing doesn’t generate a flood of alerts.

Step 8 — Turn on blocking safely: anomaly scoring + gradual rollout

CRS works best with anomaly scoring. Instead of blocking on a single match, it adds scores.

It blocks only when a request crosses a threshold. This approach cuts false positives on real-world apps.

Edit /etc/nginx/modsec/coreruleset/crs-setup.conf:

sudo nano /etc/nginx/modsec/coreruleset/crs-setup.conf

These are reasonable starting values for a public site with forms and a login. Tighten them later if you have time to tune exclusions.

# Paranoia level 1 is a sane start for most sites
SecAction "id:900000,phase:1,nolog,pass,t:none,setvar:tx.paranoia_level=1"

# Block threshold (inbound)
SecAction "id:900110,phase:1,nolog,pass,t:none,setvar:tx.inbound_anomaly_score_threshold=5"

# Outbound usually unused on Nginx setups
SecAction "id:900120,phase:1,nolog,pass,t:none,setvar:tx.outbound_anomaly_score_threshold=4"

Then switch ModSecurity from detection to blocking by changing:

# /etc/nginx/modsec/modsecurity.conf
SecRuleEngine On

Reload Nginx and test your key flows (login, checkout, contact forms, admin pages):

sudo nginx -t && sudo systemctl reload nginx

If you run WordPress, watch wp-admin AJAX calls and REST API endpoints.

They’re common false-positive hotspots once you start tightening rules.

Step 9 — Fix false positives without turning the WAF off

The quickest way to waste a WAF is to “solve” a false positive by disabling CRS. Instead, make narrow, explainable exceptions.

You can revisit them later.

Create a per-site overrides file:

sudo nano /etc/nginx/modsec/site-example.com.conf

Three patterns that usually hold up well:

• Disable one rule ID only for a single endpoint (best option).
• Disable rules for a specific parameter name.
• Increase request body limits for upload endpoints.

Example: exclude rule 942100 (SQLi detection) only on a known-safe search endpoint that accepts special characters:

# /etc/nginx/modsec/site-example.com.conf
SecRule REQUEST_URI "@beginsWith /search" "id:10001,phase:1,pass,nolog,ctl:ruleRemoveById=942100"

Example: exclude a rule for a specific argument (parameter) that legitimately includes JSON:

SecRule ARGS:payload "@rx .*" "id:10002,phase:1,pass,nolog,ctl:ruleRemoveTargetById=942260;ARGS:payload"

Include your site overrides after CRS so they take effect:

sudo nano /etc/nginx/modsec/main.conf

Include /etc/nginx/modsec/modsecurity.conf
Include /etc/nginx/modsec/coreruleset/crs-setup.conf
Include /etc/nginx/modsec/coreruleset/rules/*.conf

# Per-site overrides
Include /etc/nginx/modsec/site-example.com.conf

Reload and re-test the exact failing request. If you’re not sure what failed, capture it in your browser’s Network tab.

Then reproduce it with curl.

Step 10 — Make the logs actionable (and keep your disk from filling)

WAF logs pay off only if you can answer two questions fast: “what got blocked?” and “was it real?”

Start by rotating the audit log.

Create a logrotate policy:

sudo nano /etc/logrotate.d/modsecurity

/var/log/modsecurity/audit.log {
  daily
  rotate 14
  compress
  delaycompress
  missingok
  notifempty
  create 640 www-data adm
  sharedscripts
  postrotate
    systemctl reload nginx >/dev/null 2>&1 || true
  endscript
}

Force a test rotation:

sudo logrotate -f /etc/logrotate.d/modsecurity

For incident response, pair WAF logging with general server logging. If you haven’t centralized and cleaned up logs, the workflow in this VPS log auditing tutorial helps you avoid the “grep across 12 files” routine.

Step 11 — Performance tuning: keep the WAF from becoming your bottleneck

ModSecurity adds CPU work per request. On WordPress and many PHP apps, that overhead is often smaller than PHP execution time.

You should still watch it after enabling blocking.

Knobs that actually move the needle:

• Keep paranoia level at 1 unless you’re ready to tune exclusions.
• Don’t audit-log everything. Stick with RelevantOnly until you’re actively investigating.
• Disable response body inspection unless you have a specific reason to inspect responses.
• Raise request body limits only where needed (uploads), not globally.

Watch Nginx latency and CPU after the switch to blocking. If things slow down, measure before you change anything.

The workflow in this slow response troubleshooting guide is a good baseline.

Step 12 — Hardening checklist for production WAF deployments

Use this list before you call it “done”:

• Detection-only burn-in: run 24–72 hours in detection mode first on a representative traffic period.
• Rule update plan: CRS updates can change behavior. Schedule a monthly maintenance window to pull updates and review diffs.
• Rollback path: keep a toggle per site. In Nginx you can comment out modsecurity on; and reload.
• Disk headroom: ensure /var has room for logs; WAF logging is bursty during scans.
• Don’t expose admin panels: WAF is not a substitute for restricting access. If you need private access to panels, use an SSH tunnel workflow like this SSH tunnel setup guide.

Common pitfalls (and quick fixes)

• Nginx won’t start after loading the module: verify the module path and that it matches your Nginx build. Re-run nginx -t and read the exact error.
• Uploads fail with 413/403: increase client_max_body_size in Nginx andSecRequestBodyLimit for that site or endpoint.
• WordPress admin breaks: keep paranoia at 1, then add surgical exclusions for specific endpoints/rules after reviewing audit logs.
• Audit log explodes during bot scans: stay on RelevantOnly and rotate logs daily. If needed, temporarily reduce log parts.

Summary: a practical WAF that you can maintain

A WAF earns its keep when it matches your real traffic and stays easy to operate. Start in detection-only and review the audit log.

Add narrow exclusions, then enable blocking with anomaly scoring.

After that, treat CRS updates like any other change: controlled, logged, and reversible.

If you don’t want to spend nights chasing false positives or rebuilding modules after updates, HostMyCode can run this stack for you on managed VPS hosting. If you want full root access and hands-on control, deploy it on a HostMyCode VPS and keep your rule changes under version control.

FAQ

Should I run ModSecurity in DetectionOnly forever?

No. Detection-only is for burn-in and tuning. Once you’ve reviewed logs and added a few narrow exclusions, switch to blocking with anomaly scoring.

That’s when the WAF actually stops attacks.

Will a WAF fix WordPress brute-force attacks?

It helps, but it’s not the first tool I’d pick for brute-force. Rate limiting, Fail2Ban, strong passwords, and 2FA usually give better results.

Use WAF mainly for malicious payloads and exploit scanning.

Can I enable different paranoia levels per site?

Yes. Put per-site overrides in separate files and include them after CRS. Keep most sites at paranoia level 1.

Only raise it for apps you can test thoroughly.

What’s the easiest way to find why a request was blocked?

Start with /var/log/modsecurity/audit.log. Search for your timestamp, URL, and the CRS rule ID.

Then create a narrow exclusion (by rule ID and endpoint) and retest.

Do I need a WAF if I already use a CDN?

A CDN can filter a lot of junk, but it won’t always catch application-layer payloads specific to your app.

A WAF on your VPS is still useful, especially for admin endpoints and custom forms.