Playbook: classify every bot in your logs
Building a clean bot/human/fake-bot split, from sources that maintain the lists.
→ The ua-parser regexes (BrowserStack-maintained) — battle-tested user-agent patterns. Takeaway: don't hand-roll UA regex, this corpus already covers the long tail.
→ Matomo's bot detection list — open, frequently updated, MIT-licensed. Takeaway: a ready table of known crawler UAs to left-join against.
★ Pick of the week — Cloudflare Radar's verified-bots list — bots that publish IP ranges for verification. Takeaway: split your traffic into verified-bot / claims-to-be-bot / human, and treat the middle bucket as suspect.
→ Google's crawler overview page — distinguishes Googlebot, AdsBot, GoogleOther, inspection tools. Takeaway: tag each by purpose, not as one blob.
Three buckets beat two — the "claims-to-be" group is where the interesting story usually lives.
Logfile Roundup
@LogfileRoundup
Playbook: classify every bot in your logs
Этот пост опубликован в Telegram-канале Logfile Roundup. Подписаться можно по ссылке: @LogfileRoundup.