Cutting 5xx rate 9.4%→0.6%: crawl demand recovered 2.1x

Cutting 5xx rate 9.4%→0.6%: crawl demand recovered 2.1x

E-commerce platform throttling under load. 30-day log sample, 3.1M crawl requests.

Baseline status mix:
2xx ▓▓▓▓▓▓▓▓░░ 84.2%
5xx ▓░░░░░░░░░ 9.4%
3xx ▓░░░░░░░░░ 6.4%

Googlebot self-throttles when 5xx climbs. We added edge caching + raised origin worker count.

Delta after 3 weeks:
— 5xx rate: 9.4% → 0.6%
— Daily crawl requests: 103K → 218K (↑2.1x)
— Median TTFB: 740ms → 180ms (↓76%)
— New URLs discovered/week: 1,200 → 3,900

So what: server errors aren't just lost pages — they're a brake on total crawl demand. Reliability is a crawl-budget lever, measurable in the log delta.