Spotting fake Googlebot: user-agent filter vs IP verification

Spotting fake Googlebot: user-agent filter vs IP verification

Filtering on the UA string is where everyone starts. Here's why it's only step one.

🔗 User-agent match — grep Googlebot is instant, but the UA is trivially spoofed; scrapers wear it daily to dodge rate limits. Fine for a rough count, dangerous for decisions.
🔗 IP verification — the only trustworthy filter. Google's docs spell out reverse-DNS or the published IP ranges. Anything claiming to be Googlebot from a residential ASN is lying.

⭐ Pick of the week: the eye-opener from log audits — on many sites, 20-40% of "Googlebot" lines fail IP verification. If you optimize crawl budget off the unverified count, you're tuning to noise.

Takeaway: UA filter to scope the data; IP verification before you trust a single conclusion.