Block scraper bots eating your bandwidth
Ahrefs, Semrush and a swarm of no-name scrapers were 30% of my traffic. Cut them at nginx, not in robots.txt (they ignore it):
— Map them: map $http_user_agent $bad_bot { default 0; ~*(ahrefsbot|semrushbot|mj12bot|dotbot) 1; }
— In server block: if ($bad_bot) { return 444; }
— 444 closes the connection with zero response — cheapest possible rejection
— Keep real search bots (Googlebot, Bingbot) OFF the list, obviously
Watch your bandwidth graph drop the same day. Try it tonight.
Root Access Daily
@RootAccessDaily
Block scraper bots eating your bandwidth
Этот пост опубликован в Telegram-канале Root Access Daily. Подписаться можно по ссылке: @RootAccessDaily.