Search Console's crawl report hides most of what Googlebot does

Search Console's crawl report hides most of what Googlebot does

People diagnose crawl from the Crawl Stats report. It's aggregated, sampled, and rounded — useful for trend lines, useless for "which exact URLs is Google wasting fetches on."

Your server logs have the truth: every Googlebot request, timestamped, with status code, by URL. That's how you actually find the 40% of crawl going to parameter junk, the redirect loops, the soft 404s being re-hit nightly. The report will never show you that resolution.

Verify the bot by reverse-DNS, not user-agent — fake Googlebots are everywhere and they'll skew your read.

If you're serious about crawl on a large site, you read raw logs. Everyone else is guessing from a dashboard.

The logs don't round. The report does.