Budget Myths
Budget Myths
@CrawlBudgetMyths

The noindex-then-block sequence everyone gets backwards

The noindex-then-block sequence everyone gets backwards

Want a page gone from the index AND off Google's crawl list? Order matters, and most people invert it:

— Add noindex to the page (meta robots or X-Robots-Tag header).
— Leave it crawlable. Google must crawl it to SEE the noindex.
— Wait until it actually drops from the index — check with the URL inspection tool, not your gut.
— Only THEN, if you want to save crawl, add a robots.txt Disallow.

Do it in reverse and the page stays indexed forever — Google can't read the noindex it can't fetch.

Actually this is the single most common self-inflicted wound in this niche: blocking a page to remove it, which guarantees it can't be removed.

Crawl first, noindex applies, THEN block. Any other order is a bug.
Этот пост опубликован в Telegram-канале Budget Myths. Подписаться можно по ссылке: @CrawlBudgetMyths.
tech

Свежие посты в категории «Tech Infrastructure»

Все каналы категории →

start

Готовы запустить рекламу через сеть public.tg?

Новый оффер, продукт, GEO, кейс, событие или партнёрский запуск — соберём маршрут под задачу и отдадим медиаплан.

Telegram для медиаплана: @AFFtop_connect. Быстрый тест: $20 за канал, $1000 за пакет по сети.