Case studies: catching fake Googlebot

Case studies: catching fake Googlebot

Three teams who verified the bot instead of trusting the user-agent.

→ Cloudflare blog sample — reverse-DNS-checked every self-declared Googlebot in the logs; 31% failed and were scrapers spoofing the UA. Blocking them cut origin load 18%.
Takeaway: the user-agent string is a claim, not proof.

⭐ Pick of the week — Distilled verification script — combined Google's published IP ranges with reverse DNS and logged a confidence score per request. Found a 'Googlebot' hammering one URL 9,000x/hour from a residential IP.
Takeaway: real Googlebot doesn't behave like that.

→ Google's own verify-Googlebot doc — the canonical method everyone above implements.
Takeaway: bookmark it; the IP ranges change.