Case study: log-file analysis fixed a crawl-budget leak (faceted nav)

Case study: log-file analysis fixed a crawl-budget leak (faceted nav)

Setup: An ecommerce site had 200k useful URLs but Googlebot spent 64% of crawls on filtered faceted-nav permutations. I parsed 30 days of access logs in Screaming Frog Log Analyser (£99/yr) plus GSC Crawl Stats (free).

What it did well:
— Quantified the waste precisely: 1.9M Googlebot hits on parameter URLs that served zero traffic
— After disallowing those params in robots.txt, crawls on real product pages rose 38% in three weeks

Where it fell short:
— Log Analyser is clunky and chokes above ~10M lines without splitting files
— Robots disallow doesn't deindex already-indexed junk; needed noindex on a subset first

Verdict: 4.5/5. Crawl budget is real on big sites and logs are the only honest measurement.

Best for: sites over ~100k URLs. Not for: small sites — Google crawls you fine.