<b>Verifying Googlebot the right way: 5 references</b>
Faking a Googlebot user-agent is trivial. These cover the reverse-then-forward DNS check that actually proves it.
→ Google Search Central docs — the canonical reverse + forward DNS recipe. Run host on the IP, confirm it ends in googlebot.com or google.com, then forward-resolve back.
Takeaway: never trust the UA string alone.
→ Google's published IP ranges (googlebot.json, special-crawlers.json, user-triggered-fetchers.json). Match against these to skip DNS lookups at scale.
⭐ Pick of the week: Bing's Verify Bingbot tool and their public IP list — the same logic, often forgotten when teams only guard Google.
→ Cloudflare Radar's verified bots list — cross-checks dozens of crawlers in one place.
Takeaway: keep an allowlist, not a guesslist.
→ Stack Overflow's classic 'identify Googlebot' thread — still the clearest plain-English walkthrough for non-DNS folks.
Credit to the Search Central, Bing Webmaster, and Cloudflare teams for keeping these maintained.
Logfile Roundup
@LogfileRoundup
<b>Verifying Googlebot the right way: 5 references</b>
Этот пост опубликован в Telegram-канале Logfile Roundup. Подписаться можно по ссылке: @LogfileRoundup.