Rate limiting and bots on a small site, without blocking Google

Open the access log of any site that has been online for a year and count the lines that came from a person. On a small site it is often under half. The rest is Googlebot doing its job, a dozen AI crawlers reading everything, SEO tools measuring your backlinks, scripts guessing passwords at wp-login.php, and scanners asking for /.env and /wp-config.php.bak on the off chance. This guide sorts that traffic into what to allow, what to slow and what to block, and adds the protections in order of cost, starting with the ones that are free and cannot hurt Google. Budget an hour for the CDN and server steps, plus a log check a week later.

Quick answer

• Read your access log with the log analyser first. It tells you which bots hit you, how much, and whether the "Googlebot" in your log is real.
• robots.txt is a request, not a lock. It works on Google, Bing and most AI crawlers. It does nothing against scanners and password guessers.
• Put the site behind a CDN with bot rules (Cloudflare's free plan is enough): challenge bad traffic, rate-limit wp-login.php and xmlrpc.php, and let verified search bots through untouched.
• At the web server, add a request limit on the login path and block dotfiles. On WordPress, disable XML-RPC if nothing uses it.
• Never block by the string "Googlebot", never challenge all bots, never geo-block the country your visitors or your CDN edge are in.
• A week later, re-read the log: fewer fake Googlebots, fewer 404s from scanners, a lower average response time.

Why bot traffic matters on a small site

Every request costs the server the same whether a person or a script made it. On shared hosting with one CPU share and a PHP application behind it, a scanner walking through 3,000 plugin paths in a minute uses the same capacity as 3,000 real page views, and while it runs, everyone else's Time to First Byte goes up. That is how bot traffic turns into a speed problem: the report's TTFB finding says "slow server", and the cause is a script in Vietnam guessing your admin password.

The damage has four shapes:

• CPU and response time. Login attempts and XML-RPC calls run PHP and hit the database on every request; page caches do not help because the requests are POSTs to uncached paths.
• Bandwidth. Scrapers that copy the whole site pull every image. On a metered plan that is money; on a fair-use plan it is a warning email from the host.
• Log noise. Thousands of 404s from scanners hide the 404s that matter: the ones Googlebot hits because a real page moved.
• Credential stuffing. Password guesses are cheap for the attacker and only have to work once. A brute-force run against a site with a weak password succeeds in hours.

Bot traffic is also not one thing: some you want (Googlebot, Bingbot, link preview fetchers), some you may want (AI crawlers), some is useless but harmless (SEO tools), and some is hostile. Treating it as one thing fails in both directions: blocking everything that says "bot" loses Google; allowing everything costs TTFB.

How getReport checks it

getReport does not measure bot load directly; no external tool can see your server's CPU. What it shows are the symptoms and the record. The record is your access log, which the log analyser reads in your browser (nothing is uploaded) and turns into one row per bot:

The columns that matter for this job are Hits and Share (who costs you the most), 404 (scanners generate hundreds; a real crawler generates a few), Avg ms (how slow the server was for that bot, when the log records response time) and Verified. Verification is a separate click: the addresses that claimed to be Googlebot or Bingbot are sent to our API, which does the reverse-DNS check Google documents, and the row then shows how many hits came from real, fake and unknown addresses.

One limit to know: bots are attributed by user agent. A scanner or a password-guessing script that sends a Chrome user agent is counted under "Human visits", not in the table. You find those with the 404 and 5xx tiles at the top and with a grep on the log (step 1 below).

The symptoms show up in an ordinary report:

A slow first byte on a site with a page cache and a small database is the classic sign of a server busy with something other than visitors. And a version in the Server header is an invitation: scanners pick their exploits from it. The report's robots.txt reading shows the third piece, which of the known AI crawlers you currently allow:

Step by step

1. Find out what is actually hitting you

Export a week of access logs from the hosting panel (cPanel: Metrics → Raw Access; Plesk: Logs; a VPS: /var/log/nginx/access.log or /var/log/apache2/access.log, .gz is fine) and drop the file on the log analyser. Write down four numbers: the share of bot hits, the Googlebot 404 percentage, the number of fake Googlebot addresses after verification, and the average response time for Googlebot. You will compare against them later.

Then look for the traffic the table cannot attribute. Over SSH, the two commands that find login guessing and scanners:

# Addresses with the most POSTs to the login page or XML-RPC this week
grep -E 'POST /(wp-login\.php|xmlrpc\.php)' access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20

# Addresses with the most 404s (scanners walk through hundreds of paths)
awk '$9 == 404 {print $1}' access.log | sort | uniq -c | sort -rn | head -20

The field numbers assume the common log format (address first, status ninth); adjust for JSON logs. A single address with 500 login POSTs in a day is a brute-force run. A single address with 2,000 404s, most of them under /wp-content/plugins/…/readme.txt, /.env, /.git/config or /backup.zip, is a vulnerability scanner. Note both patterns; the rules below target them.

2. robots.txt for the polite bots

robots.txt is obeyed by search engines, by the AI crawlers that identify themselves, and by most SEO tools. It is ignored by everything hostile, so it is a policy file, not a protection. Two things belong in it for this job.

First, the AI crawlers you do not want. The ai-crawlers-robots finding lists which of the known ones your file currently blocks; the decision (training bots vs search bots vs user-triggered fetches) is in AI crawlers and robots.txt. A group per crawler is all it takes:

# robots.txt at the site root
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Second, what not to put in it: Crawl-delay. Google ignores the directive (its crawl rate is adaptive and set from how fast your server answers); Bing and Yandex honour it. Adding Crawl-delay: 10 therefore slows Bing and changes nothing for the crawler that fetches you most. If Googlebot really is too fast for the server, the fix is server speed, and in the worst case a temporary 503 or 429 on overload, which Google treats as a signal to slow down.

3. The CDN layer: cheapest per request

A CDN answers the request before it reaches your server, so a bot stopped there costs you nothing. Cloudflare's free plan covers everything a small site needs:

• Bot Fight Mode (Security → Bots) challenges traffic that matches known bad-bot patterns. It leaves Cloudflare's list of verified bots alone, which includes Googlebot and Bingbot, so turning it on does not affect crawling.
• A WAF custom rule with the Managed Challenge action for the paths that only humans should reach. A managed challenge is invisible to most browsers and stops scripts cold:

(http.request.uri.path eq "/wp-login.php" and not cf.client.bot) or
(http.request.uri.path eq "/xmlrpc.php")

• A rate limiting rule (Security → WAF → Rate limiting rules) for login attempts. The free plan includes rate limiting with a fixed short counting window, which is enough for this: more than a handful of POSTs to the login page from one address in ten seconds is never a person.

Expression:  (http.request.uri.path eq "/wp-login.php" and http.request.method eq "POST")
Rate:        5 requests per 10 seconds, per IP
Action:      Block for 10 minutes

The field cf.client.bot is true for the verified bots on Cloudflare's list, which is the safety net in every rule above: as long as a rule says not cf.client.bot, Googlebot is never challenged by it. Bunny and Fastly have equivalents (Bunny Shield, Fastly's Next-Gen WAF); the principle is the same, exclude verified search bots explicitly and rate-limit the login path, not the whole site.

One trap: when the site is behind a CDN, the web server sees the CDN's addresses, not the visitor's. Any server-side rule in step 4 needs the real address restored first (real_ip_header CF-Connecting-IP on nginx, mod_remoteip on Apache), and the origin should accept connections only from the CDN's ranges, or bots that know the origin address walk around the CDN entirely.

4. Web server rate limits

For sites without a CDN, or as a second layer behind one, the web server can count requests itself. On nginx, limit_req does exactly this and costs nothing measurable:

# /etc/nginx/nginx.conf, inside the http { } block
limit_req_zone $binary_remote_addr zone=login:10m rate=10r/m;

# /etc/nginx/sites-available/example.com, inside the server { } block
location = /wp-login.php {
    limit_req zone=login burst=5 nodelay;
    limit_req_status 429;
    include fastcgi_params;
    fastcgi_pass unix:/run/php/php8.3-fpm.sock;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}

location = /xmlrpc.php {
    return 403;
}

# Never serve dotfiles (.env, .git, .htaccess)
location ~ /\. {
    deny all;
}

rate=10r/m allows ten login requests per minute per address; burst=5 lets a real person who mistypes twice through without a 429. Copy the fastcgi_pass line from the existing location ~ \.php$ block so the login page still runs PHP.

Apache has no request-rate module in its default set. mod_ratelimit, despite the name, limits bandwidth per connection, which does nothing against 500 small POSTs. The Apache answer to request rates is mod_evasive (where the host has installed it) or fail2ban below. What .htaccess can do is close the door on XML-RPC and dotfiles:

# .htaccess at the site root
<Files "xmlrpc.php">
    Require all denied
</Files>

RedirectMatch 404 /\..*$

fail2ban watches the log and bans addresses at the firewall after repeated matches, which works on any web server. The filter and jail for WordPress logins:

# /etc/fail2ban/filter.d/wordpress-login.conf
[Definition]
failregex = ^<HOST> .* "POST /(wp-login\.php|xmlrpc\.php)
ignoreregex =

# /etc/fail2ban/jail.d/wordpress-login.conf
[wordpress-login]
enabled  = true
filter   = wordpress-login
logpath  = /var/log/nginx/access.log
port     = http,https
maxretry = 10
findtime = 10m
bantime  = 1h

Ten login POSTs in ten minutes from one address earns an hour's ban. Run sudo fail2ban-client status wordpress-login a day later to see the count; if it is zero, check that logpath matches the file your server writes.

5. The application layer

Whatever reaches PHP is the most expensive kind of request, so the goal is to leave as little as possible for this layer. Three things still belong here:

• XML-RPC off if nothing uses it (Jetpack and some older publishing apps do). The server block above is cheaper, but where you cannot edit server config, the filter in a small plugin or the theme's functions.php makes WordPress refuse the calls:

add_filter('xmlrpc_enabled', '__return_false');

• A login limiter at the PHP level (Limit Login Attempts Reloaded or similar) when neither a CDN rule nor a server rule is available. It works, but every blocked attempt still boots WordPress; on a shared host that is the cost you were trying to avoid.
• A WAF plugin (Wordfence, Solid Security, and the like) inspects every request in PHP. That is a real defence against exploit attempts, and it has a price: 20–100 ms of extra TTFB on every uncached request, and more while its scanner runs. If the CDN layer is in place, run the plugin's firewall in its lightest mode or leave it to the scanner-only role. The rest of the WordPress side (updates, passwords, two-factor, DISALLOW_FILE_EDIT) is in WordPress security basics without a plugin.

6. Hide what invites scanners

Scanners choose exploits by what the server tells them. Server: Apache/2.4.29 and X-Powered-By: PHP/7.4.3 narrow the search to a handful of known holes. server_tokens off; on nginx, ServerTokens Prod on Apache and expose_php = Off in php.ini remove the version numbers; the server and X-Powered-By headers guide has the config for each platform. Combine it with the dotfile rule from step 4 so /.env and /.git/HEAD answer 404, and the two things every scanner tries first come back empty. The security headers checker confirms both in one run.

What never to do

• Block the user agent "Googlebot" without verifying the address. Most fake Googlebots are scrapers hoping you allow the name. Blocking by name stops the real one too. Verify by IP (the log analyser's Verified column, or Google's published IP ranges) and block the fakes by address.
• Challenge all bots. A JavaScript challenge or a CAPTCHA on every request locks out Googlebot, Bingbot, link preview fetchers and uptime monitors. Every rule with a challenge needs the verified-bot exclusion.
• Geo-block a whole country. Your CDN's edge servers, your uptime monitor, your own developer on holiday, and Google's crawlers (which fetch from the US) all have countries. Block addresses, not maps.
• Rate-limit so tightly that Googlebot backs off. Google slows down when it gets 429 or 5xx answers, and Search Console → Settings → Crawl stats shows it: a drop in crawl requests and a rise in "server errors" after the day you added the rule. Limits belong on the login path and XML-RPC, not on /.

Verify

• Re-run the log analyser on the following week's log. The fake Googlebot count should fall to near zero, the 404 share of unknown bots should drop, and the average response time for Googlebot should be no worse than before.
• grep -c 'POST /wp-login.php' access.log for the new week is a fraction of the old week's number.
• Run the TTFB test: the first byte should be lower or unchanged. If it got slower, a plugin-level firewall is the usual reason.
• Search Console → Settings → Crawl stats: total crawl requests steady, no new spike in 429 or 5xx responses.
• curl -sI https://example.com/xmlrpc.php answers 403 (or 404), and curl -sI https://example.com/.env answers 404.

Common mistakes

• The rule works in testing and does nothing in production. Symptom: the same address keeps guessing. The server sees the CDN's address for every visitor, so per-IP limits count everyone as one client. Restore the real address with real_ip_header or mod_remoteip.
• robots.txt blocks a scanner. Symptom: the Disallow line exists, the 404s continue. Hostile bots never read the file. Move that rule to the CDN or the firewall.
• The login page rate limit catches the team. Symptom: colleagues behind one office address get 429 on the second login of the morning. Add burst on nginx, raise the per-window count on the CDN, or exclude the office address.
• A security plugin and a CDN both challenge. Symptom: TTFB doubled after "adding protection". Two firewalls inspect every request; keep the CDN as the firewall and let the plugin do scanning only.