Log files show the crawl waste you will never see in analytics

Log files show the crawl waste you will never see in analytics

Open server logs and sort by user-agent, status code, and request path. For ecommerce sites, the first red flag is usually bots spending most of their time on filter URLs, sort parameters, internal search, and endless pagination. If those URLs create near-duplicate pages, bots can burn crawl budget without helping important category or product pages get discovered.

Look for patterns like:
— the same product-list page with many parameter combinations
— repeated hits on empty filter states
— 404, 301, and 302 chains that keep getting requested
— faceted URLs with no canonical plan
— pages crawled often but never linked from core navigation

The useful part is not volume alone, but waste per template. A page that gets 2,000 hits a month can be fine if it is a key category. The same number on ten thousand filter permutations is a trap. Compare bot hits against indexability: if a URL is crawlable, thin, and unlikely to rank, it should usually be blocked, normalized, or noindexed based on your site setup.

Use logs to confirm whether your fixes work. After pruning parameters, adding canonicals, or tightening internal links, watch for fewer hits on junk URLs and more consistent crawling of money pages. If bots keep returning to dead ends, the problem is usually internal links, not the crawl settings alone.

Start with the top 20 wasteful URL patterns, not the whole site. Clean the biggest leaks first, then check logs again.