Framework: status-code check vs content check for soft-404 detection

Framework: status-code check vs content check for soft-404 detection

The trap: a dead page returning HTTP 200. Two-layer detection.

1. Layer one — status check (any bulk tool). Catches honest 404/410 only.
2. Layer two — content check. Crawl the body for trigger phrases: "not found", "no longer available", "page removed", thin word count under 50.
3. Decision rule: never trust a 200 alone for an old URL. Always run layer two before discarding it as live.
4. QA gate: manually open any 200 with a trigger phrase to confirm it's truly dead, not a styled error.

Automation: Frog custom extraction for trigger phrases on all 200-status crawls.

Definition of done: every 200-status URL passed through content check, soft-404s reclassified.