Myth: "Crawl budget doesn't matter under 10k URLs"

Myth: "Crawl budget doesn't matter under 10k URLs"

Sample: 287 audited domains, all under 10k indexable URLs.

Median wasted crawl share (non-200, redirects, faceted noise):

— Under 1k URLs: 19.4% ▓▓░░░░░░░░
— 1k-5k URLs: 31.7% ▓▓▓░░░░░░░
— 5k-10k URLs: 44.2% ▓▓▓▓░░░░░░

The advice assumes total URL count is the constraint. It isn't. The constraint is the ratio of useful fetches to total fetches. On a 4k-URL site with 60k crawlable parameter permutations, Googlebot spends p90 of 71.3% of hits on URLs you never wanted indexed.

Delta after blocking parameter URLs in robots.txt: useful-fetch share 55.8% → 88.1% (↑32.3pp), median time-to-discovery of new pages 9.1d → 3.4d.

So what: crawl budget is a waste-ratio problem, not a size threshold. Audit fetch composition, not URL count.