Blocking a URL in robots.txt can still get it indexed

Blocking a URL in robots.txt can still get it indexed

The assumption: robots.txt disallow = page disappears from Google. Wrong, and it backfires.

Disallow stops the crawl, not the indexing. If other pages link to that blocked URL, Google can index it from the anchor text alone — that's the dreaded "indexed, though blocked by robots.txt." It can't read the page, so it can't see your noindex, so it can't honor the very tag you'd use to remove it. You've locked yourself out.

Want it gone from the index? Allow the crawl and use noindex. Want it un-crawled? Disallow — but accept it may linger as a URL-only entry.

You can't block and noindex the same URL. Pick one.

These tools cancel each other out.