How do I diagnose and fix wasted crawl budget on a big store?

How do I diagnose and fix wasted crawl budget on a big store?

'Q: Googlebot crawls tons of junk URLs and ignores my new products. How do I redirect its attention?'

Short answer: Read your server logs to find where the crawler wastes time, then cut those paths and strengthen links to what matters.

The longer version: Work from data.
— Pull 30 days of access logs, filter to Googlebot, and rank URLs by crawl frequency.
— Identify the waste: filter/sort params, internal search, faceted infinity, soft-404s, redirect chains.
— Kill it: robots-block true infinite spaces, fix redirect chains to single hops, and noindex thin auto-pages.
— Boost discovery of what matters: clean sitemaps split by section, fresh internal links to new products, and 'lastmod' that's actually accurate.
— Recheck logs in 30 days; crawl share should shift toward products.

Example: a marketplace found 60% of crawl hits going to sort parameters; blocking them freed the crawler to pick up new listings within days instead of weeks.

Rule of thumb: you can't optimize crawl budget you haven't measured in the logs.

Got an e-com SEO question? Drop it.