Index or Bust
Index or Bust
@IndexOrBust

Noindex inside a robots.txt-blocked page = stuck forever

Noindex inside a robots.txt-blocked page = stuck forever

This is the classic deadlock, and people set it up by accident constantly. You add noindex to a page you want removed, then also Disallow it in robots.txt to "be safe." Result: Google can't crawl the page, so it never sees the noindex, so the URL can linger in the index as a URL-only entry indefinitely.

What the tools tell you:
— GSC flags it as "Indexed, though blocked by robots.txt"
— That message is a symptom, not a config you want

Where people get it wrong:
— Assuming Disallow removes a page (it only blocks crawling, not indexing)
— Stacking both directives for the same URL

The fix: pick one. For removal, allow crawling and keep noindex until it drops out — usually a few crawls — then optionally block later. Never block a URL you're trying to deindex.

Best for: understanding why a "removed" page won't die.
Not for: anyone who treats robots.txt as a deindex tool.
Этот пост опубликован в Telegram-канале Index or Bust. Подписаться можно по ссылке: @IndexOrBust.
tech

Свежие посты в категории «Tech Infrastructure»

Все каналы категории →

start

Готовы запустить рекламу через сеть public.tg?

Новый оффер, продукт, GEO, кейс, событие или партнёрский запуск — соберём маршрут под задачу и отдадим медиаплан.

Telegram для медиаплана: @AFFtop_connect. Быстрый тест: $20 за канал, $1000 за пакет по сети.