URL Inspection API vs. bulk crawl: sampling tradeoff

URL Inspection API vs. bulk crawl: sampling tradeoff

The Inspection API gives Google's own verdict but caps at 2,000 calls/day per property. A crawler checks unlimited URLs but only infers indexability. Across 50 large sites we modeled the tradeoff.

— Inspection API: ground-truth, 2k/day ceiling ▓▓░░░░░░░░ (coverage on 200k site: 1.0%/day)
— Crawler indexability heuristic: 100% coverage, median 8.9% false-positive on "indexable" ▓▓▓▓▓▓▓▓▓▓

So what: don't choose — stratify. Crawl everything to segment URLs by template, then spend your 2k daily API calls as a stratified sample across templates, not top-down. A 384-URL sample per template hits ±5% confidence. You get truth where it's cheap and coverage where it isn't.