Mistake: matching crawler IPs as strings instead of CIDR ranges

Mistake: matching crawler IPs as strings instead of CIDR ranges

"Starts with 66.249" both misses ranges and lets spoofers through. Sources:

→ Google's googlebot.json / special-crawlers.json — IPs come as CIDR blocks; prefix-string matching is wrong at the boundaries.
Takeaway: parse the JSON and test membership with a real subnet check.

→ Python ipaddress docs — ip_network + in does correct CIDR containment in three lines.
Takeaway: never startswith() on an IP string.

★ Pick of the week — a grepcidr / mapcidr note — Stream-filter a whole log against a CIDR file at the command line.
Takeaway: grepcidr -f google-ranges.txt access.log beats any regex.