Checklist: free crawler vs paid index for 404 hunting

Checklist: free crawler vs paid index for 404 hunting

The tradeoff is freshness vs coverage. Match tool to where the dead link lives.

1. Page-level 404s on a target's own site → free Screaming Frog (500 URL cap) crawls live HTML, zero index lag.
2. Dead pages that earned external links → paid index (Ahrefs/Semrush). Only they hold the historical link graph.
3. Trigger: if the page is gone AND you need its backlinks, free tools can't see them. Escalate to paid.
4. QA gate: confirm Frog's 404 against a fresh curl -I — cached redirects lie.

Automation: run Frog in list mode on exported URLs overnight.

Definition of done: each prospect tagged either "crawl-found" or "index-found" with source noted.