crawl-delay does nothing to Googlebot. It never has.

crawl-delay does nothing to Googlebot. It never has.

Myth: add crawl-delay to robots and Googlebot backs off.

No. Google ignores crawl-delay outright — it's a Bing and Yandex directive. Google's own docs say so. You've been throttling crawlers you didn't need to throttle and not the one you meant to.

The actual tool for Google is the crawl rate setting Search Console used to expose, now mostly automatic, plus serving honest 500s and 503s when your server is genuinely drowning. Google reads server stress and slows itself down. That's the real rate limit.

The tradeoff: crawl-delay is a polite request other bots honor. For Google, your server's response time IS the throttle.

Want Googlebot slower? Don't write a directive. Get a faster host so it doesn't need to be.