Case: log-driven robots rules cut crawl waste and halved index latency

Case: log-driven robots rules cut crawl waste and halved index latency

2.1M monthly bot hits; only 340k landed on indexable money pages. SOP to reclaim budget:

— Step 1 (analyst): aggregated 30 days of logs by path. Gate: path with >2% of crawl and 0 organic entrances = waste.
— Step 2 (analyst): flagged the top 8 waste patterns — internal search, print views, tracking params, calendar endpoints.
— Step 3 (dev): disallowed those 8 patterns in robots.txt; confirmed none were canonical targets first.
— Step 4 (analyst): re-pulled logs at day 30.

Outcome: money-page bot hits 340k to 690k/month at flat total crawl. Median new-page index latency 8 days to 4.

Guardrail: confirm a path is no page's canonical before disallowing — disallowing a canonical orphans its cluster.

Ship gate: don't publish until all boxes are checked.