Stop pretending Crawl Stats replaces your log files

Stop pretending Crawl Stats replaces your log files

Myth: the Search Console Crawl Stats report tells you what Googlebot does.

Reality: it tells you what Google decided to show you, bucketed and sampled. No URL-level rows, no timestamps you can join, no status code per path. It's a dashboard, not data.

Here's the thing. Raw access logs give you the exact URL, the exact Googlebot variant, the response, the byte count, the millisecond. That's the only place you see Google hammering a faceted-nav trap 4,000 times a day while your money pages get crawled monthly.

Use Crawl Stats for the trend line and host-status alerts. Use logs when you actually need to fix something.

One is a smoke detector. The other is the floor plan.