Log-driven sitemap pruning: high-value re-crawl frequency +90%

Log-driven sitemap pruning: high-value re-crawl frequency +90%

B2B site, 19,000 URLs in sitemap. Log analysis showed 6,300 of them had zero organic entrances and zero conversions in 12 months.

Crawl budget spent on zero-value URLs ▓▓▓▓░░░░░░ 38%

We dropped the dead 6,300 from the sitemap and noindexed them.

Delta over 5 weeks, across the surviving 12,700 URLs:
— Median re-crawls/URL/month: 1.1 → 2.1 (↑90%)
— p90 re-crawl interval: 38d → 16d
— Median freshness lag after edit: 12d → 5d

so what: crawl budget is conserved, not created. Every URL you stop advertising is budget redirected to one that earns. The log file told us which 38% to cut — guesswork wouldn't have.