Trust logs over GSC for crawl reality

Trust logs over GSC for crawl reality

GSC's Crawl Stats samples and aggregates; raw server logs don't. Across 41 paired datasets, GSC under-reported total Googlebot hits by a median 11.0% and misattributed bot type on ~6% of rows.

— logs: ground truth, 100% ▓▓▓▓▓▓▓▓▓▓
— GSC crawl stats: ~89% of true volume ▓▓▓▓▓▓▓▓▓░

Worse, GSC can't tell you which exact URLs were crawled — only host/type/response buckets.

So what: for crawl-budget work, parse logs. Use GSC for the index-coverage side. Mixing the two without knowing which is sampled produces phantom "crawl drops" that are really just sampling noise.