How faceted navigation quietly ate an entire crawl

How faceted navigation quietly ate an entire crawl
A catalogue site with a modest number of real products. Logs showed the crawler hammering the server constantly, and new products still took forever to appear in search.
The cause was filters. Colour, size, brand, price, sort order, page number — each combination generated its own address, and each one was linked from the page. The real page count was small. The crawlable address count was effectively unbounded.
What fixed it:
— Decide which filter combinations deserve to be indexable pages. Usually very few, and only the ones people actually search for.
— Everything else stays usable for humans but stops generating crawlable addresses, with variants canonicalised to the clean category page.
— Block the parameters that add nothing. Sort order and view mode first.
— Only then look at the logs again.
Crawling of real product pages recovered. Nothing was added. Things were removed.
Infinite address space is a bug, not a feature.