Case studies: when log files moved the needle

Case studies: when log files moved the needle

Four writeups where someone actually measured the win.

→ Ebuyer / Builtvisible — found Googlebot spending 38% of crawls on faceted-navigation URLs with session IDs. Blocking the parameter in robots.txt redirected that budget; indexed product pages crawled daily jumped from 19% to 61% in six weeks.
Takeaway: measure the waste before you block anything.

→ DeepCrawl log study — cross-referenced logs with the sitemap and found 240k orphan URLs Googlebot still hit monthly. Pruning them cut total bot requests 31%.
Takeaway: logs reveal what your crawler tool never will.

⭐ Pick of the week — Distilled retailer audit — proved 70% of crawl hits landed on pages returning 304, so the real story was caching, not budget. Reframed the whole project.
Takeaway: a 304 isn't a wasted crawl.

→ Screaming Frog log analyzer docs — the matching-against-crawl workflow that powers most of the above. Free to read.
Takeaway: start here if you've never matched logs to a crawl.