Self-Host AI Crawlers Blocklist With robots.txt

Your robots.txt file is one of the few controls you can change quickly. It’s low risk to deploy. You can also confirm changes in your access logs.

If you want to block AI crawlers from pulling pages for training or datasets, take a measured approach. Start with a small blocklist. Confirm which bots actually visit. Then roll out changes on Nginx, Apache, or cPanel. This helps you avoid accidental SEO damage.

What robots.txt can (and cannot) do for AI scraping

robots.txt is guidance, not enforcement. Compliant crawlers read it and follow the rules. Uncooperative scrapers ignore it and keep fetching pages.

What it does: Tells specific user agents which paths they may or may not crawl; can also publish sitemap URLs.
What it doesn’t do: Authenticate users, rate-limit requests, or block traffic at the network layer.
What you’ll verify: Which user agents hit your site, how frequently, and whether their behavior changes after you publish rules.

If you need enforcement, pair robots.txt with server-side controls. Use WAF rules, Nginx/Apache denies, or a CDN.

This guide focuses on a self-hosted robots.txt blocklist plus verification. It also adds a few hardening notes where they matter.

Prerequisites: access, paths, and a safe rollout plan

You’ll need at least one of these:

SSH access to your VPS/dedicated server (Nginx or Apache).
cPanel File Manager access for shared hosting.
Ability to inspect logs (at minimum, access logs for your site’s vhost).

If the server hosts multiple sites, confirm the correct document root before you change anything.

If you want predictable log locations and full control over web server config, a VPS you manage is the easiest place to work. For example, HostMyCode VPS gives you root access for edits, better log visibility, and fewer panel constraints.

Step 1: Inventory AI bot traffic from your server logs

Don’t start by pasting a giant list from the internet. First, check what actually hits your site.

This avoids blocking “ghosts.” It also gives you a baseline for after deployment.

On Nginx (common log locations)

Typical paths:

/var/log/nginx/access.log
/var/log/nginx/your-site.access.log (per-server block)

List the most frequent user agents from the last 50k lines:

sudo tail -n 50000 /var/log/nginx/access.log \
| awk -F'"' '{print $6}' \
| sed 's/^ *//;s/ *$//' \
| sort \
| uniq -c \
| sort -nr \
| head -n 30

On Apache (common log locations)

Typical paths:

/var/log/apache2/access.log (Debian/Ubuntu)
/var/log/httpd/access_log (RHEL/AlmaLinux/Rocky)
/var/log/apache2/your-site-access.log (per-vhost)

Same idea on Ubuntu/Debian:

sudo tail -n 50000 /var/log/apache2/access.log \
| awk -F'"' '{print $6}' \
| sed 's/^ *//;s/ *$//' \
| sort \
| uniq -c \
| sort -nr \
| head -n 30

Filter likely AI crawlers fast

This grep is intentionally broad. It will catch many AI-related user agents, plus a few false positives.

Use it to find candidates. Then copy the exact tokens you see in real requests.

sudo tail -n 200000 /var/log/nginx/access.log \
| grep -Ei 'gpt|openai|anthropic|claude|bytespider|ccbot|commoncrawl|ai|chatgpt' \
| head -n 50

If you’re on Apache, swap the log path. Once you have a short list of real user agents, you can make robots.txt precise instead of speculative.

If you can’t find logs easily (or rotation is too aggressive), fix that first. You need enough history to spot spikes and confirm trends.

For visibility into load patterns, dashboards help. HostMyCode customers often use lightweight monitoring as shown in live server monitoring with Netdata.

Step 2: Build a robots.txt that blocks known AI crawlers (without breaking search)

A good robots.txt does two things well. It targets specific AI user agents. It also stays out of the way of normal search indexing.

Keep the file readable. Add short comments so future-you knows why each line exists.

A practical starter robots.txt (conservative)

Put this in your site root as robots.txt. Then adjust the user agents based on what you found in your logs.

# robots.txt
# Goal: limit compliant AI training crawlers while keeping search indexing intact.

User-agent: *
Disallow:

# Common AI / dataset crawlers (add/remove based on your logs)
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: ClaudeBot
Disallow: /

# If you publish a sitemap, keep it here
Sitemap: https://example.com/sitemap.xml

Important: user-agent matching is string-based and depends on the crawler’s implementation. Names change. Formats vary. Some operators use multiple tokens.

Treat your own logs as the source of truth.

Block only sensitive paths (optional alternative)

If you don’t mind bots reading public articles, keep them out of sensitive sections. Common examples include docs, pricing, or account pages.

In that case, restrict the disallows to specific paths:

User-agent: GPTBot
Disallow: /docs/
Disallow: /pricing/
Disallow: /account/
Disallow: /wp-admin/
Disallow: /wp-login.php

This reduces collateral damage risk. It still relies on compliance.

WordPress note: don’t rely on “Discourage search engines”

WordPress can emit robots meta tags and has a “discourage indexing” option. That setting is meant for staging sites.

On production, keep indexing healthy. Use robots.txt as a targeted policy for specific bots instead.

Step 3: Deploy robots.txt safely on Nginx, Apache, or cPanel

Most robots.txt failures are basic. The file lands in the wrong document root. Permissions block reads. Or caching makes you think nothing changed.

Deploy carefully. Then verify with a fetch.

Nginx on Ubuntu/Debian (typical)

Find your server block and document root:

sudo nginx -T | grep -E "server_name|root" | head -n 40

Place robots.txt in the document root (example uses /var/www/example.com/public):
```
sudo nano /var/www/example.com/public/robots.txt
```

Fix ownership/permissions (the web server must be able to read it):

sudo chown root:root /var/www/example.com/public/robots.txt
sudo chmod 644 /var/www/example.com/public/robots.txt

Confirm it’s served correctly:

curl -sS https://example.com/robots.txt | sed -n '1,120p'

Common pitfall: the file exists on disk, but your live site uses a different root. This can happen with containers, reverse proxies, or a different vhost.

Trust curl, not your file browser.

Apache on Ubuntu/Debian (DocumentRoot)

Locate the vhost and DocumentRoot:
```
sudo apache2ctl -S
```

Edit/create robots.txt under the DocumentRoot:

sudo nano /var/www/example.com/public_html/robots.txt

Verify it’s served:

curl -sS https://example.com/robots.txt | head

cPanel shared hosting (File Manager)

Open cPanel → File Manager.
Go to public_html/ (or your addon domain’s document root).
Create robots.txt (or edit it if it exists).
Save, then fetch https://yourdomain/robots.txt in your browser to confirm it’s live.

If you manage multiple client sites, reseller hosting keeps this workflow clean. Each account gets its own document root and its own robots policy.

If that matches your setup, HostMyCode reseller hosting is a practical fit for agencies maintaining multiple domains.

Step 4: Make sure robots.txt isn’t being cached

robots.txt is commonly cached by CDNs. Some caching plugins also cache it aggressively.

If you update the file and still see the old version, you’re almost certainly hitting cache.

Browser check: use an incognito window and hard refresh.
Command check: inspect headers.

curl -I https://example.com/robots.txt

If you see a long Cache-Control max-age, shorten it for robots.txt. On Nginx you can add a location block (adjust paths to your config layout):

# inside your server { } block
location = /robots.txt {
  add_header Cache-Control "public, max-age=300";
}

Reload Nginx after changes:

sudo nginx -t && sudo systemctl reload nginx

Step 5: Verify compliance by watching access logs after deployment

robots.txt can’t force behavior. It can only change what compliant bots do.

Your logs will show whether anything actually changed.

Watch requests to /robots.txt in real time

On Nginx:

sudo tail -f /var/log/nginx/access.log | grep -E "GET /robots\.txt"

On Apache (Ubuntu/Debian):

sudo tail -f /var/log/apache2/access.log | grep -E "GET /robots\.txt"

Compare bot activity before vs after

Give it 24–72 hours. Then re-run your user-agent summary and compare counts.

For a single bot token:

sudo grep -Ei 'GPTBot' /var/log/nginx/access.log | wc -l

Don’t stop at totals. Check which URLs those bots request.

If they fetch /robots.txt and then keep pulling disallowed pages, assume they’re ignoring the directives.

Step 6: Add a “polite” AI policy page and link it (optional)

A short policy page won’t block anything. It gives compliant operators a place to read your rules. It also gives them a way to contact you about access or licensing.

Create /ai-policy/ with your rules.
Add it to your footer.
Keep it factual and short.

It also helps internally. When someone asks why a bot is blocked, the rationale shouldn’t live only in a config comment.

Step 7: Hardening (optional): back up robots.txt and keep change control

If the site is business-critical, treat robots.txt like any other production config file.

Back it up: keep a copy in Git or your ops repo.
Test restores: if you back up your whole web root, make sure restores don’t overwrite a newer policy.
Document owners: decide who can change robots.txt (marketing, engineering, or both).

If you also run PostgreSQL for your app stack and want reliable backups, see our pg_dump backup + restore test tutorial. Same idea: edits are easy; recovery discipline keeps incidents short.

Troubleshooting: common mistakes that make robots.txt ineffective

robots.txt returns 404 or 403
Fix the file path and permissions. Confirm with curl -I https://example.com/robots.txt.
Wrong document root on multi-site servers
You edited /var/www/html/robots.txt but your vhost serves /var/www/example.com/public. Verify with nginx -T or apache2ctl -S.
A plugin or app overwrote robots.txt
Some CMS tools generate virtual robots rules. Prefer a real file on disk, and disable “dynamic robots” if it conflicts.
Your CDN caches the old file
Purge cache for /robots.txt. Then reduce max-age for that path.
You blocked search engines by accident
Don’t add broad User-agent: * disallows unless you mean it. Keep the default section open, then block specific agents.

Going beyond robots.txt: what to do if bots ignore you

If you still see scraping from a user agent that fetched robots.txt, assume it won’t comply. At that point you need enforcement: IP blocks, rate limits, bot challenges, or authenticated access.

Apply the same thinking to internal endpoints. Put them behind a proxy and require keys.

Two practical next steps on HostMyCode:

If you’re exposing any internal apps, build a consistent reverse-proxy layer with HTTPS. Our tutorial on Caddy reverse proxy for Docker on a VPS is a clean baseline.
If you’re already self-hosting AI services, secure them like an API. See securing a vLLM API with Nginx, keys, and rate limits for the pattern.

Summary: a repeatable workflow to limit AI scraping

To block AI crawlers responsibly, use a simple loop: measure, change, verify.

Pull real user agents from access logs. Add only rules you can justify. Deploy carefully, then watch traffic for a few days to see what changed.

If you want predictable log access, straightforward rollbacks, and the option to add enforcement later, do this on infrastructure you control. A managed VPS hosting plan from HostMyCode is a solid middle ground: you keep control of site policy, and you can get help when a change touches web server config.

If you manage more than one site, a consistent robots.txt policy saves time and prevents avoidable SEO mistakes. Run your sites on a HostMyCode VPS (or choose managed VPS hosting if you want help with web server changes), then standardize your robots rules and validate them in your logs.

FAQ

Does robots.txt fully prevent AI companies from scraping my site?

No. It only controls compliant crawlers. Keep monitoring logs, and if a bot ignores robots.txt, move to server-level blocking or rate limiting.

Will blocking AI bots hurt my SEO?

Not by itself, as long as you don’t block normal search engines. The common mistake is adding broad Disallow: / under User-agent: * or blocking assets your pages need.

How do I know which AI crawlers are visiting my site?

Check your access logs and summarize user agents. In most setups, you can extract top user agents from the last 50k–200k lines and then filter for suspicious tokens.

Where should robots.txt live for WordPress?

In the WordPress document root (often public_html/ or /var/www/your-site/public_html). Confirm it’s reachable at https://yourdomain/robots.txt and isn’t being overridden by a plugin.

Should I block by user-agent or by path?

Blocking by user-agent is simplest if your aim is “no AI crawling.” Blocking by path is safer if you only want to protect specific sections, but it assumes bots comply.