Read your log files to find where crawl budget dies

Read your log files to find where crawl budget dies
Client swore Google ignored their new section. Logs told the real story. The read:
— Grab 30 days of raw server access logs (ask hosting/devops, or pull from CDN)
— Filter to verified Googlebot hits by user-agent AND reverse-DNS — fakes are everywhere
— Aggregate bot hits by URL folder: which sections get crawled, which get zero
— Spot the waste: bot hammering ?sort= params, 404s, redirect chains
— Cross-reference: high-value folders with low bot hits = your fix target
— After fixes, re-pull logs in 30 days and compare crawl distribution
Found the bot spending 40% of hits on dead parameter URLs. Cleared them, new section finally got crawled.
Verify Googlebot by reverse DNS — half of 'Googlebot' in raw logs is scrapers faking the UA.