Playbook: a log-file analysis to find where Google wastes crawl budget

Playbook: a log-file analysis to find where Google wastes crawl budget
I ran real server logs through Screaming Frog Log Analyser and JetOctopus on a 50k-URL site. This is the audit that finds problems crawlers can't.

The workflow:
— Pull 30 days of raw access logs (filter to verified Googlebot by reverse-DNS, not just user-agent).
— Group hits by directory. If Googlebot spends 40% of crawls on /tag/ or faceted URLs, that's wasted budget.
— Find pages Google crawls often but that get zero traffic — candidates for noindex or robots block.
— Find important pages Google rarely or never crawls — internal-linking or sitemap gap.
— Compare crawl frequency vs last-modified — Google re-crawling static pages = misallocated attention.

Screaming Frog Log Analyser:
➕ Cheap one-time desktop tool, joins logs to your crawl
➖ Manual, doesn't scale to millions of lines smoothly
JetOctopus:
➕ Real-time log streaming + crawl in one cloud platform
➖ Enterprise pricing

Best for: occasional audits → Frog. Always-on big sites → JetOctopus.
Pick this if you have a large site and crawl/index lag. Skip entirely under ~10k URLs — crawl budget isn't your problem.