Serving Googlebot from cache cut crawl-related server load 70% — crawling went UP.

Serving Googlebot from cache cut crawl-related server load 70% — crawling went UP.

Counterintuitive case. A high-traffic site was rate-limiting Googlebot at the firewall because crawl was "overloading" the origin. Crawling, predictably, dropped — and so did freshness.

The better move: instead of throttling Google, they let the CDN serve Googlebot full-page cache hits. Origin load from crawl fell 70%, response times for bot requests dropped to sub-100ms.

Googlebot read those fast responses as "this server can take more" and increased its crawl rate 50% on its own.

The myth they'd believed: that more crawling always means more strain, so you must limit Google. With caching, more crawling cost them almost nothing.

No, you don't throttle Googlebot to protect your server. You make crawling cheap.

Fast responses invite more crawl. Throttling invites less indexing.