<b>Log-file analysis: the only honest crawl-budget data</b>

<b>Log-file analysis: the only honest crawl-budget data</b>

GSC's Crawl Stats is a summary. Your server logs are the receipts. I review the logs first.

Setup: pull access logs, filter verified Googlebot by reverse-DNS, parse with Screaming Frog Log Analyser or a quick script.

What it does well:
— Shows exactly which URLs Googlebot wastes crawls on — faceted filters, params, redirects
— Reveals crawl frequency per template, so you see WHAT gets re-crawled and what's starved
— Verified-bot filtering kills the fake-Googlebot noise

Where it falls short:
— You must verify by IP/DNS — ~30% of 'Googlebot' user-agents are spoofed
— Useless on tiny sites with low crawl volume
— Log access on shared hosting is often a fight

Pros: ground truth, reveals crawl waste, no sampling
Cons: setup friction, needs scale, verification overhead

Best for: 10k+ URL sites with crawl-budget symptoms.
NOT for: a 50-page brochure site — there's no budget problem to find.