Crawlers miss your worst crawl leak because they read your status codes

Crawlers miss your worst crawl leak because they read your status codes

Myth: run a site crawler, it'll find the pages wasting crawl budget.

No. Soft 404s — empty results, "no products found," thin filter pages — return HTTP 200. Your crawler sees 200 and moves on, happy. Google sees 200, crawls them at full priority, then quietly classifies them as soft 404 and keeps re-crawling to check.

Here's the thing. The pages bleeding the most budget are the ones that look fine to every tool measuring status codes. You only catch them in the Search Console index report's "soft 404" bucket, or by spotting near-identical thin pages in logs by hit volume.

The tradeoff: crawlers find hard errors fast. They're blind to pages that lie with a 200.

The leak you can't see is the one returning success.