Nginx Bot Blocking Tutorial (2026): Stop Bad Crawlers on a VPS with Maps, User-Agent Rules, and Fail2Ban

Most “bot traffic” headaches aren’t about bandwidth. They’re about wasted CPU: repeated hits to /wp-login.php, expensive search endpoints, and 404 sweeps that keep PHP-FPM workers busy. This Nginx bot blocking tutorial shows a practical way to reduce noisy crawler traffic on a hosting VPS without breaking real users or legitimate search engines.

You’ll set up a layered defense. It includes a strict-but-maintainable map-based blocklist, a sane robots.txt, optional basic auth for non-public paths, and an escalation path with Fail2Ban for repeat offenders.

The steps assume Ubuntu 24.04/25.04-style layouts and Nginx 1.24+ as commonly shipped by distributions in 2026.

Before you start: scope, risks, and what “blocking bots” really means

Blocking by User-Agent isn’t security. Anyone can spoof it.

It’s still a clean way to cut down on low-effort scrapers and “SEO tools” that identify themselves honestly.

The biggest gains usually come from three moves:

• Rejecting obvious junk fast (return 403 or 444 before PHP runs).
• Protecting expensive URLs (search, login, XML-RPC) with targeted rules.
• Escalating repeat abuse to IP bans using logs and Fail2Ban.

If you want a managed environment where the web stack and security defaults are kept current for you, managed VPS hosting from HostMyCode is a good fit for WordPress and multi-site hosting.

It’s especially useful when you’d rather ship features than babysit rulesets.

Prerequisites (VPS, Nginx, and a safe rollback)

You should have:

• Root or sudo access to your VPS
• Nginx installed and serving your site
• A rollback plan (snapshot or config backup)

Quick config backup:

sudo cp -a /etc/nginx /etc/nginx.bak.$(date +%F)

Test config after every change:

sudo nginx -t
sudo systemctl reload nginx

If you’re still setting up baseline server hygiene, see: server hardening tutorial for a new Ubuntu VPS.

Step 1: Turn on a log format that makes bot analysis fast

If you can’t see what’s happening, you’ll end up guessing. Guessing leads to broken rules.

Add a dedicated log format that includes request time and User-Agent. This makes abusive crawlers easier to spot.

Edit /etc/nginx/nginx.conf inside the http {} block:

log_format botdiagnostic '$remote_addr - $remote_user [$time_local] '
                       '"$request" $status $body_bytes_sent '
                       'rt=$request_time ua="$http_user_agent" '
                       'ref="$http_referer"';

In your server block (or site file under /etc/nginx/sites-available/), set:

access_log /var/log/nginx/access_botdiagnostic.log botdiagnostic;

Reload and let it run for a few minutes. Then pull the top User-Agents:

sudo tail -n 2000 /var/log/nginx/access_botdiagnostic.log \
  | awk -F 'ua="' '{print $2}' \
  | awk -F '"' '{print $1}' \
  | sort | uniq -c | sort -nr | head

And check which URLs are taking the most hits:

sudo tail -n 5000 /var/log/nginx/access_botdiagnostic.log \
  | awk '{print $7}' | sort | uniq -c | sort -nr | head -n 20

Step 2: Create a maintainable User-Agent blocklist with Nginx map

Avoid scattering if rules across virtual hosts. You’ll miss one.

Scattered rules also make it easier to block something you didn’t intend.

Put the logic in one map include file. Then apply it consistently.

Create a new include file:

sudo nano /etc/nginx/conf.d/bot-blocking-map.conf

Add this (start conservative, then adjust based on your logs):

map $http_user_agent $is_bad_ua {
  default 0;

  # Empty UA is commonly used by crude scanners
  "" 1;

  # Examples of self-identified scrapers/scanners (adjust to your logs)
  ~*(?:curl|wget|python-requests|httpclient|libwww-perl) 1;
  ~*(?:go-http-client|aiohttp|scrapy) 1;

  # “SEO”/downloaders that are rarely legitimate on production sites
  ~*(?:HTTrack|SiteSucker|Teleport|WebCopier) 1;
}

Now apply it in each relevant server {} block near the top:

if ($is_bad_ua) { return 403; }

Why 403 and not 444? Use 403 while you tune. It’s easier to spot in logs and less likely to confuse upstream checks.

Once you trust your patterns, move the noisiest categories to return 444;. That drops connections with minimal work.

Step 3: Don’t rely on robots.txt, but do make it correct

robots.txt won’t stop hostile bots. It can reduce accidental crawl waste from well-behaved crawlers.

This includes some AI-related bots that follow rules.

Keep it explicit. Don’t block assets Google needs to render pages.

Example for WordPress (serve it as a static file from your web root):

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

If your search endpoint triggers expensive DB queries, this alone can cut a surprising amount of “polite” bot load.

Step 4: Block common exploit paths early (cheap 403/444 rules)

Even if you don’t run WordPress, attackers will probe WordPress paths. Respond cheaply, and keep these requests away from PHP.

Put these in the server {} block (adjust for your app):

# Quietly drop obvious probes (optional)
location = /xmlrpc.php { return 444; }

# Block common dotfile and repo leaks
location ~* /(\.git|\.env|\.ht|composer\.(json|lock)|package\.json|yarn\.lock) {
  return 403;
}

# Prevent PHP execution in uploads (WordPress hardening pattern)
location ~* ^/wp-content/uploads/.*\.php$ {
  return 403;
}

For a broader hardening pass (headers, TLS behavior, and safer defaults), pair this with: HTTP security headers tutorial.

Step 5: Add a “challenge” zone for high-cost pages (basic auth or allowlist)

Some URLs are fine for humans but expensive at scale. Common examples include internal search, preview links, admin tooling, and staging areas.

If a path doesn’t need to be public, protect it.

Example: protect a private staging area at /staging/ with basic auth.

1. Install tools and create a password file:

sudo apt update
sudo apt install -y apache2-utils
sudo htpasswd -c /etc/nginx/.htpasswd youruser

2. In the site config:

location ^~ /staging/ {
  auth_basic "Restricted";
  auth_basic_user_file /etc/nginx/.htpasswd;
}

If you run a staging workflow for WordPress updates, this pairs nicely with: WordPress staging site tutorial.

Step 6: Escalate repeat offenders with Fail2Ban (based on Nginx logs)

User-Agent rules handle the low-hanging fruit. Fail2Ban is for IPs that keep hammering known bad paths.

It’s also useful for IPs that generate high rates of 403/404 in a short window.

Install Fail2Ban:

sudo apt update
sudo apt install -y fail2ban
sudo systemctl enable --now fail2ban

Create a filter for common exploit scans (example for WordPress + generic probes):

sudo nano /etc/fail2ban/filter.d/nginx-badbots-paths.conf

[Definition]
failregex = ^<HOST> .* "(GET|POST) /(wp-login\.php|xmlrpc\.php|wp-admin/|\.env|\.git|cgi-bin/).*" (200|301|302|403|404)
ignoreregex =

Create a jail:

sudo nano /etc/fail2ban/jail.d/nginx-badbots-paths.local

[nginx-badbots-paths]
enabled = true
filter = nginx-badbots-paths
logpath = /var/log/nginx/access_botdiagnostic.log
maxretry = 15
findtime = 10m
bantime = 6h
action = iptables-multiport[name=nginx-badbots, port="http,https"]

Restart Fail2Ban and verify:

sudo systemctl restart fail2ban
sudo fail2ban-client status
sudo fail2ban-client status nginx-badbots-paths

Pitfall: if your distro uses nftables by default, the iptables action may be wrong.

Check /etc/fail2ban/action.d/ and switch to an nftables action if required.

Step 7: Add a safe allowlist for real bots (Google, Bing) without trusting User-Agent alone

If you block aggressively by UA, you need a plan to avoid collateral damage. The “correct” options are reverse DNS validation (slow) or IP range allowlists (maintenance-heavy).

For most small and mid-sized sites, the better answer is simpler. Keep UA blocking conservative, and don’t block Googlebot-style strings at all.

A pragmatic approach:

• Do not block any UA containing Googlebot, Bingbot, DuckDuckBot, Applebot.
• Do block empty UA and commodity tool UAs you actually see in your logs.

If you host client sites and need tighter control, a dedicated environment can make life easier. You can segment workloads and apply per-site rules without one noisy domain dragging down the rest.

HostMyCode offers dedicated servers for high-traffic sites where bot noise turns into a real cost line.

Step 8: Quick diagnostics: prove you reduced load (not just blocked requests)

After you deploy, don’t stop at “requests got blocked.” Check operational signals like response codes, request time, and upstream pressure.

• 403/444 volume should rise (junk is getting stopped early).
• Average request time should drop for real pages.
• PHP-FPM busy workers should fall during bot spikes.

Top blocked UAs:

sudo awk '$9 ~ /403|444/ {print}' /var/log/nginx/access_botdiagnostic.log \
  | awk -F 'ua="' '{print $2}' | awk -F '"' '{print $1}' \
  | sort | uniq -c | sort -nr | head

If you want alerts when bot spikes start (instead of hearing about it from users), add lightweight monitoring and log checks.

This pairs well with: server monitoring tutorial.

Step 9: Hardening checklist (copy/paste for your runbook)

• Back up Nginx config: /etc/nginx
• Add a dedicated access log format with User-Agent and request time
• Implement map-based UA blocking in one include file
• Add cheap blocks for common exploit paths (.env, .git, xmlrpc.php)
• Protect non-public paths with basic auth
• Deploy Fail2Ban jails against repeated bad-path requests
• Review logs weekly and adjust patterns based on evidence
• Document rollback steps: restore backup + nginx -t + reload

Where this fits in a hosting workflow (WordPress, reseller, and multi-site VPS)

On a multi-site VPS, bot noise on one domain can steal resources from every other site. Bot control belongs in normal hosting operations, not in a one-off “fix it once” sprint.

If you run a reseller-style setup, keep these rules in version control. Deploy them the same way everywhere.

When you migrate sites, treat your Nginx include files as part of the move checklist. That prevents old problems from following you to the new server.

HostMyCode can help with cutovers and validation via HostMyCode migrations.

FAQ

Will User-Agent blocking hurt SEO?

It can if you block legitimate crawlers. Keep UA blocks conservative, monitor logs, and avoid blocking known search bots by string unless you also validate their IP ownership.

Should I use 403 or 444 for bad bots?

Use 403 while you tune rules because it’s easier to audit. Switch the worst offenders to 444 after you’re confident you won’t block real users.

Why add Fail2Ban if Nginx already blocks requests?

Nginx can reject quickly, but it still has to accept connections. Fail2Ban reduces repeated abuse from the same IP by banning it at the firewall layer.

How do I know which bots to block?

Start from your own logs. Sort by User-Agent and by URL volume, then block the patterns you can clearly justify. Avoid copying random blocklists from the internet.

What if the bot rotates IPs and ignores User-Agent rules?

At that point you’re dealing with a more serious adversary. Consider upstream protections (CDN/WAF) or moving heavy endpoints behind authentication. On your server, prioritize caching and reducing PHP execution paths.

Summary: a practical bot-control setup you can maintain

This Nginx bot blocking tutorial used a layered approach: logging you can trust, a single map-based UA blocklist, cheap rules for common exploit paths, and Fail2Ban to punish repeat offenders.

Nothing fancy. Just measurable changes you can roll back and keep consistent across multiple sites.

If you’re rolling this out across client domains—or you simply want more predictable resources—do it on infrastructure you control. HostMyCode’s virtual private servers are a straightforward place to standardize these Nginx policies and keep noisy traffic from spilling into the rest of your stack.