Stratified sampling caught a defect a full crawl missed
2.8M-URL site. Full crawls kept timing out, so the team eyeballed the homepage cluster and called it clean.
We ran a stratified sample: 400 URLs per template type, 11 templates, 4,400 total.
Defect rate by template:
Homepage/hub ░░░░░░░░░░ 0.0%
Product ▓░░░░░░░░░ 2.1%
Legacy /archive ▓▓▓▓▓▓▓▓▓░ 91.0%
The /archive template (310K URLs, 11% of site) was serving a broken canonical to a dead domain. A random sample would've surfaced it at ~11% prevalence; the template-blind eyeball missed it entirely.
So what: full crawls aren't the only path to coverage. n=400 per stratum gives ±5% margin and finds template-level rot in minutes, not hours.
Crawl & Render
@CrawlAndRender
Stratified sampling caught a defect a full crawl missed
Этот пост опубликован в Telegram-канале Crawl & Render. Подписаться можно по ссылке: @CrawlAndRender.