Playbook: stand up a crawl-efficiency dashboard

Playbook: stand up a crawl-efficiency dashboard
Define crawl efficiency = unique indexable 200s crawled ÷ total crawl requests. Median = 0.62; best-in-class ≥0.80 (sample: 130 sites).

Build it:
— Step 1: nightly job parses logs → counts requests, dedupes URLs, joins indexability flag
— Step 2: plot the ratio daily. Baseline vs. now: a 0.05 ↓ drop means new waste entered the crawl
— Step 3: decompose denominator: % on params, % on 404s, % on redirects. Watch which band grew
— Step 4: alert when params-share crosses 15% ▓▓▓░░░░░░░ — usually a new faceted-nav leak
— Step 5: tie each dip to a deploy date; most efficiency drops are shipped, not organic

So what: a single efficiency number turns crawl health into one tracked metric. Correlate dips with deploys and you catch crawl leaks the day they ship.