Playbook: verify a Googlebot hit is really Googlebot
Four sources that turn "is this real?" into a repeatable check.
→ Google Search Central docs — the canonical reverse-then-forward DNS method. Takeaway: host 66.249.66.1 must end in .googlebot.com or .google.com, then forward-resolve back to the same IP.
→ Google's published IP ranges — the googlebot.json and special-crawlers.json files. Takeaway: match against the ranges instead of DNS when you need speed at scale.
★ Pick of the week — Stack Overflow thread on awk-based rDNS — a one-pass script piping unique bot IPs through host. Takeaway: dedupe IPs first, you'll cut lookups by 90%.
→ Cloudflare bot docs — note the GoogleOther and Google-InspectionTool agents share infra but serve different products. Takeaway: don't block them assuming they're Googlebot.
Never trust the user-agent string alone — it's trivially spoofed.
Logfile Roundup
@LogfileRoundup
Playbook: verify a Googlebot hit is really Googlebot
Этот пост опубликован в Telegram-канале Logfile Roundup. Подписаться можно по ссылке: @LogfileRoundup.