Verifying Googlebot revealed 47% of "crawl" was fakers. The fix saved nothing for Google.
A site panicked over "crawl budget" because logs showed Googlebot hammering them. Then someone actually reverse-DNS verified the IPs.
47% of requests claiming to be Googlebot weren't — scrapers and bots spoofing the user-agent. Real Googlebot traffic was modest and well within capacity.
The lesson: most "crawl budget" analysis is garbage because people trust the user-agent string. Google publishes its IP ranges precisely so you verify.
They blocked the fakers at the edge. Server load dropped 40% — but Google's actual crawling, and their indexation, didn't change at all. Because Google was never the problem.
No, you don't have a crawl budget crisis. You have a logging problem.
Verify before you diagnose. Half your "Googlebot" might be lying.
Budget Myths
@CrawlBudgetMyths
Verifying Googlebot revealed 47% of "crawl" was fakers. The fix saved nothing for Google.
Этот пост опубликован в Telegram-канале Budget Myths. Подписаться можно по ссылке: @CrawlBudgetMyths.