Budget Myths
Budget Myths
@CrawlBudgetMyths

<b>noindex does not save you a single crawl</b>

<b>noindex does not save you a single crawl</b>

The pitch: slap noindex on thin pages and Googlebot stops wasting crawls on them. No.

Google has to crawl the page to see the noindex tag. The directive lives in the HTML or the header — both require a fetch. So a noindexed URL gets crawled, repeatedly, often for years, to confirm it's still noindexed. You spent crawl, not saved it.

If the goal is to actually stop the fetching, that's robots.txt disallow — which blocks the crawl but then can't read your noindex. Different tools, different jobs.

noindex controls indexing. It was never a crawl-control lever, and pretending it is just adds fetches.

Want fewer crawls? Don't link to the URL at all.
Этот пост опубликован в Telegram-канале Budget Myths. Подписаться можно по ссылке: @CrawlBudgetMyths.
tech

Свежие посты в категории «Tech Infrastructure»

Все каналы категории →

start

Готовы запустить рекламу через сеть public.tg?

Новый оффер, продукт, GEO, кейс, событие или партнёрский запуск — соберём маршрут под задачу и отдадим медиаплан.

Telegram для медиаплана: @dumay. Быстрый тест: $20 за канал, $1000 за пакет по сети.