Playbook: audit crawl budget with server logs

Playbook: audit crawl budget with server logs

GSC tells you what got crawled; logs tell you what Googlebot wasted time on. Steps:
— Pull 30 days of raw access logs, filter to verified Googlebot by reverse-DNS, not user-agent
— Find URLs crawled often but driving no traffic — params, filters, pagination eating budget
— Block the junk in robots.txt; confirm important pages are crawled at all
— Compare crawl frequency of money pages vs thin pages and rebalance internal links

Screaming Frog Log Analyzer
👍 desktop, joins logs to crawl data, $109/yr
👎 you must fetch logs yourself, learning curve on big files

Oncrawl
👍 cloud, automated log ingestion, segment reports
👎 enterprise pricing, overkill under 50k pages

Verdict: Frog Log Analyzer for a one-time audit, Oncrawl if you run a large site continuously.

Best for: big sites where Google ignores new pages.

No affiliate link — we just use this stuff.