Logfile Roundup
Logfile Roundup
@LogfileRoundup

<b>Spotting fake crawlers in your logs: 5 solid reads</b>

<b>Spotting fake crawlers in your logs: 5 solid reads</b>
Scrapers love wearing a Googlebot costume. Here's how the pros unmask them.

→ <b>Google's "verifying Googlebot" page</b> — rDNS is step one, always.
★ <b>Pick of the week — Cloudflare's bot-management writeup</b> — breaks down the tells: UA claims Googlebot but the IP is a residential proxy or a datacenter ASN Google doesn't own. ASN lookup beats DNS for speed.
→ <b>Detectify / ipinfo ASN guides</b> — map an IP to its owning network; "Googlebot" from Hetzner or DigitalOcean is fake, period.
→ <b>DataDome's bot-traffic reports</b> — useful baselines for what fraction of "Googlebot" hits are typically spoofed.
→ <b>AbuseIPDB</b> — cross-check noisy IPs you keep seeing.

Takeaway: real Googlebot lives in Google's ASNs (e.g. AS15169). Wrong ASN = impostor, no matter the UA.
Этот пост опубликован в Telegram-канале Logfile Roundup. Подписаться можно по ссылке: @LogfileRoundup.
start

Готовы запустить рекламу через сеть public.tg?

Новый оффер, продукт, GEO, кейс, событие или партнёрский запуск — соберём маршрут под задачу и отдадим медиаплан.

Telegram для медиаплана: @dumay. Быстрый тест: $20 за канал, $1000 за пакет по сети.