Mistake: reporting average daily crawl and missing the spikes

Mistake: reporting average daily crawl and missing the spikes

An average smooths over the burst that hammered your origin at 3am. Worth reading:

→ Google Search Central (crawl rate) — Crawl demand is bursty; Google adapts to server response time, so a slow spike causes throttling.
Takeaway: watch p95 hourly fetches, not the daily mean.

→ Brendan Gregg on percentiles — Why averages lie about tail behavior in any request stream.
Takeaway: histogram your hits per hour; the tail is where pain lives.

★ Pick of the week — a GoAccess time-bucket view — Per-hour panels reveal the burst pattern instantly.
Takeaway: if 5xx clusters in the spike hour, that's your crawl-budget killer.