The crawl-budget staircase

The crawl-budget staircase

Large pSEO sites burn crawl budget on low-value URLs and starve the pages that matter. Build a staircase that points crawlers up.

Five steps:

1. Tier the URLs — Owner: SEO. Money pages, support pages, junk. Gate: every URL is tagged with one tier.

2. Log-sample weekly — Owner: dev. Pull server logs, count Googlebot hits per tier. Gate: report exists.

3. Find the waste — Tier-3 (junk: parameters, faceted noise, paginated tails) getting more than 20 percent of bot hits is the leak.

4. Block the drain — robots.txt disallow on parameter paths that produce no indexable value; remove them from sitemaps. Gate: tier-3 share drops next cycle.

5. Feed the winners — internal links and sitemap priority concentrate on tier-1. Gate: tier-1 crawl frequency rises.

— Crawl budget is reallocated, not created; you steal it from junk
— robots.txt disallow saves crawl budget but does not de-index already-indexed URLs; pair with noindex first, then block

Ship gate: don't ship a block until logs confirm where the budget actually goes.