Killing a faceted-nav crawl trap: 71% of fetches were waste

Killing a faceted-nav crawl trap: 71% of fetches were waste

Mid-size retailer, 41,000 real SKUs. Log sample: 2.4M Googlebot hits over 30 days.

Param URL fetches ▓▓▓▓▓▓▓░░░ 71.3%
Clean product fetches ▓▓░░░░░░░░ 18.6%
Everything else ▓░░░░░░░░░ 10.1%

Fix: disallowed 6 filter params in robots.txt, canonicalized sort variants. 21 days later, across the same crawl window:

— Param fetches: 71.3% → 12.4% (↓58.9pts)
— Product fetches: 18.6% → 54.0% (↑35.4pts)
— Median product re-crawl interval: 19d → 6d (↓68%)

So what: the index didn't grow because Google found new URLs. It grew because it stopped drowning in old ones. Reallocation beats expansion.