Fixing a 900ms server response raised Google's crawl rate 3x.
The popular fear: "Google capped my crawl rate, I need to ask for more." You can't ask anymore — that setting's gone. And it was never the real lever.
Googlebot self-limits based on how your server responds. A site averaging 900ms TTFB, with occasional 503s under load, was being crawled cautiously — Google backs off when it senses strain.
They moved to better hosting and added caching. Median response dropped to 180ms, zero 5xx errors.
Search Console crawl stats: average daily fetches tripled over the next month, entirely on Google's initiative. Nobody touched a dial.
Here's the thing — crawl rate is a conversation. Your server's speed is your side of it.
Fast, healthy servers get crawled hard. Slow ones get rationed.
Budget Myths
@CrawlBudgetMyths
Fixing a 900ms server response raised Google's crawl rate 3x.
Этот пост опубликован в Telegram-канале Budget Myths. Подписаться можно по ссылке: @CrawlBudgetMyths.