Playbook: hunt fake Googlebot in your logs A repeatable scrub, credited to the people who published the methods. → DeviceAtlas / Distil-era write-ups — fake bots cluster on odd ASNs, never Google's. T…
Playbook: classify every bot in your logs Building a clean bot/human/fake-bot split, from sources that maintain the lists. → The ua-parser regexes (BrowserStack-maintained) — battle-tested user-agent …
Playbook: measure crawl frequency per site section Four sources for answering "how often does Google hit my product pages vs blog?" → OnCrawl's crawl-frequency segmentation — map URL prefixes to secti…
Playbook: sample logs without lying to yourself When full logs won't fit in memory, these sources keep the math honest. → Google's own "sample your logs" guidance — for big sites, a representative sli…
Default Combined log format vs a custom one Before you analyze, decide what you log. The choice shapes every later query. → Combined Log Format — the Apache/Nginx default. Universal, every tool parses…
Playbook: stand up a BigQuery log pipeline Four references for moving from grep-on-a-box to queryable logs. → Google Cloud's log sink docs — route Cloud Logging straight to a BigQuery dataset with a s…
