Crawl-Budget Prioritization Order

Crawl-Budget Prioritization Order
You can't check the whole web. Sequence your crawls by expected yield.
1 — Tier 1: resource pages and "links" pages in your niche (highest dead-link density).
2 — Tier 2: competitor outbound profiles (pre-relevant prospects).
3 — Tier 3: recently-migrated domains (fresh wreckage).
4 — Tier 4: broad topical crawls (lowest yield, run only with spare capacity).
5 — Trigger: re-rank tiers monthly based on actual find-rate per tier.
Automation fits across all tiers — a queue scheduler runs Tier 1 weekly, Tier 4 only when idle.
Definition of done: crawl capacity is allocated top-down by tier, and tier order is re-ranked against last month's find rates.