Decision rule: noindex or disallow, never both
This is the most common technical mistake I see, and it is a logic error rather than a config error.
Disallow stops the crawl. Noindex asks for removal from the index. If you disallow a page, the crawler cannot fetch it, which means it can never see the noindex tag. The page then sits in the index as a bare address with no description, indefinitely.
The rule:
— Want it out of the index? Let it be crawled, serve noindex, wait until it drops, then block it if you want to save the crawl.
— Want to save crawl on pages that were never indexed anyway? Disallow alone is fine.
— Page has links pointing at it and you want that value to go somewhere? Neither. Redirect it.
— Sensitive content? Neither tag is a security control. Put it behind authentication.
Order matters more than the tags. Removal first, blocking second.
Robots controls access. Meta robots controls indexing. They are not synonyms.
The Access Log
@TheAccessLog
Decision rule: noindex or disallow, never both
Этот пост опубликован в Telegram-канале The Access Log. Подписаться можно по ссылке: @TheAccessLog.